If the step of establishing a significant interaction is completed first, then investigators minimize their chances of making a type I error — a false-positive error — to 1 chance by calculating just 1 P value, instead of several. But if a researcher begins by testing 3 within-group effects, such as within each of the 3 different age groups in the study, then they end up running 3 tests. As a result, the number of chances of a type I error balloons to 3.
In their study, the investigators originally wanted to understand how age might impact angiogenesis, so they examined tumor samples from younger and older patients with melanoma. They found that while angiogenesis increased with age, the expression of VEGF decreased.
“That was totally confusing, right? Because we’re seeing all this increased angiogenesis but a loss of VEGF,” said Ashani Tanuja Weeraratna, PhD, senior study author and the E.V. McCollum Chair of Biochemistry and Molecular Biology at the Johns Hopkins School of Public Health in Baltimore, Maryland. Dr Weeraratna is also a professor in the department of oncology at Johns Hopkins School of Medicine.
This finding led the researchers to suspect that the aging-related drop in VEGF might translate into age-specific differences in response to bevacizumab. By means of post hoc analysis, the investigators then analyzed data from the AVAST-M trial, which had involved 1343 patients with resected melanoma, and broke down the results by patient age. The patients were subsequently separated into 3 age groups: those younger than 45 years, those older than 65 years, and those in between the 2 age categories. The researchers then compared disease-free survival and overall survival among the participants in these 3 groups.
But Dr Sainani outlined potential problems with the type of analysis the researchers had employed. “We don’t know how many other subgroup variables they looked at,” she said. “Every subgroup considered increases the chance of a false-positive finding.”
Despite these limitations, the researchers went through with their analysis and found an apparent confirmation of their earlier suspicions. “What we saw was that the patients under the age of 45 actually did respond to Avastin, whereas older patients did not, even though they have more angiogenesis,” Dr Weeraratna said.
Specifically, the investigators concluded that patients in this age group who received adjuvant bevacizumab had a longer disease-free interval (hazard ratio [HR], 0.71; 95% CI, 0.53–0.96), whereas this benefit was not found in patients older than 65 years (HR, 1.00; 95% CI, 0.72–1.38). The authors referred to this finding as “a significantly longer disease-free interval” in younger patients.
The authors then indicated in the paper that an analogical tendency applied to overall survival. “Similar patterns of treatment effect trended strongly for the outcome of overall survival as well,” they wrote. The researchers subsequently directed the readers to a related figure included in a supplement to the paper.
However, as stated in a press release published by American Association for Cancer Research, the organization that publishes the journal Clinical Cancer Research where the paper was published, the finding about overall survival was not statistically significant.5
Dr Sainani was not impressed by the wording the study authors had used to discuss their findings on overall survival in the paper. “‘Trended strongly’ is code for ‘we failed to meet statistical significance,’” she said.
She said that the wording was “misleading” because none of the interactions presented were significant. This means that the authors lacked proof that the effects of the drug changed depending on patient age.