National cancer registries are increasingly being used to evaluate sarcoma treatment outcomes, but new research from investigators at Vanderbilt University Medical Center suggests manuscripts resulting from these studies should be read with a cautious eye.
Herbert Schwartz, M.D., former chair of the Department of Orthopaedic Surgery and Rehabilitation at Vanderbilt, has been taking note of this increase in sarcoma registry research and has sounded an alarm over an unusual frequency of discordant findings.
“Two different research groups can ask the same database the same question over the same time frame and get a different answer,” he said.
To understand why, Schwartz and Vanderbilt colleagues, including Joshua Lawrenz, M.D., an assistant professor in the Division of Musculoskeletal Oncology, zoned in on the challenges of capturing a cancer case’s true dynamics in single-snapshot registry abstracts.
“Cancer treatment, especially when it’s multidisciplinary, is very complex,” Lawrenz said. “If you just look at it at one point in time, like a cancer coder would, you might not get the full story.”
Differing Outcomes
In their new study, presented at the 2021 Musculoskeletal Tumor Society annual meeting, the research team first quantified the uptick in sarcoma registry research, focusing on studies published between 2000 and 2020 that pulled data from the National Cancer Database or the Surveillance, Epidemiology, and End Results (SEER) program database.
“Cancer treatment, especially when it’s multidisciplinary, is very complex. If you just look at it at one point in time, like a cancer coder would, you might not get the full story.”
“In 2000, there weren’t many publications, and then they kind of stabilized between 2009 and 2010. But beginning in 2018, the number of annual publications tripled,” Schwartz said.
Of the 299 sarcoma registry studies published during the 20-year time frame, 94 were identified as sharing a similar research question. Further analysis revealed that 44 percent of the publications with similar research questions had concordant findings, 35 percent had discordant findings, and 29 percent had mixed findings – meaning their results were partially inconsistent.
“About two-thirds of the 94 studies that shared a similar study question didn’t provide the same outcome answer,” Schwartz said. “That struck us.”
As one example, three studies published between 2016 and 2018 evaluated radiation versus surgery in localized Ewing’s sarcoma using the SEER database and reported three different results, Schwartz said.
“One said radiation is not inferior to surgery. The other one said radiation is worse than surgery. And the other one said radiation has the same results as surgery.”
Abstraction Errors
To get to the root of these discrepancies, the team conducted an internal audit with Vanderbilt cancer coders – individuals that abstract information from EMRs for submission to cancer registries.
“Two different research groups can ask the same database the same question over the same timeframe and get a different answer.”
For roughly 100 cases, the cancer coders were asked to abstract data on six variables: date of birth, date of surgery, surgical margin, tumor size, cancer stage, and whether adjuvant treatment was received. The coder’s abstraction results were then compared to the consensus of Vanderbilt’s sarcoma tumor board.
“Of the six variables we tested, the date of birth was wrong 4 percent of the time,” Schwartz said. “That represents human error. Someone has to take the medical record from this screen and type it into another screen, another spreadsheet, and it’s wrong 4 percent of the time.”
For 23 percent of cases, conflicting surgery dates were copied into the abstract. Schwartz draws attention to the typically high number of procedures cancer patients undergo, including biopsies and two-stage operations.
“There’s never just one date,” Schwartz said. “It can be hard to discern which is the planned, definitive surgery.”
Other variables like tumor size and cancer stage that may vary over the course of treatment are other opportunities for abstraction error, Lawrenz adds.
“We believe that prospective, multicenter studies – such as through a consortium where data are collated by physicians in a coordinated method – are more appropriate for investigations that seek to influence treatment strategy or health policy.”
Outlook Going Forward
Lawrenz emphasized the value of the registry data for ongoing and future research.
“Despite the inherent challenges of these national cancer registries, they remain particularly invaluable in musculoskeletal cancer research for nationwide incidence estimations, risk factor and prognostic factor assessment, patient demographic and hospital-level variable assessment, and assessing patterns of care over time,” Lawrenz said.
And he added this caution: “It’s important we understand the strengths and weaknesses of each national registry when evaluating studies or using them yourself. We believe that prospective, multicenter studies – such as through a consortium where data are collated by physicians in a coordinated method – are more appropriate for investigations that seek to influence treatment strategy or health policy.”
Schwartz suggests that anyone looking to cancer-registry literature for guidance not rely on just a single study. He also encourages a thorough examination of methods to check for variations in data analysis that may lead to discordant findings.
“You have to do a deep dive into the paper and see, for example, how they handle missing data and what kind of modeling they use,” Schwartz said.
