Pitfalls Aplenty for Observational Research

May 13, 2012

By Deborah Borfitz  

May 14, 2012 | Limitations on the use of observational data in comparative effectiveness research (CER) were among the themes of a Post-Approval Summit held in early May at Harvard Medical School*. The march toward bigger and often mandatory post-approval studies needs to be rooted in “strong science,” enabled by industry partnerships with large health care delivery organizations, insurers, the Food and Drug Administration (FDA), and academia, according to keynote speaker Michael Rosenblatt, executive vice president and chief medical officer at Merck.   

The promises of “big data” culled from the real world are many, including enhanced understanding of the natural history and management of diseases, earlier detection of side effects, better targeting of drugs to patients, comparisons across countries and modalities of treatment, and advancement of regulatory science and innovation, says Rosenblatt. The FDA has already assembled 17 data partners, representing upwards of 125 million patients, to establish a safety surveillance system for marketed drugs known as the Sentinel Initiative. Ultimately, it may also be used for CER.  

By law, data sources used by the Sentinel Initiative are to include electronic health records (EHRs) by 2013. The complicating issue is that clinical data sets cut across varying practice patterns, decision-making algorithms, and coding practices, Rosenblatt says. The size of the Sentinel Initiative alone could make it difficult to tease out clinically significant risks. Old-fashioned chart reviews will be “required to provide reliable information,” he adds, noting that even heart attacks still get miscoded.  

In a Vaccine Safety Datalink project conducted by Harvard Pilgrim Health Care a few years back, only one in ten vaccine-related safety concerns picked up by rapid cycle analysis proved upon chart review to be a true association, Rosenblatt shares. An Observational Medical Outcomes Partnership study more recently found a 16% to 30% false-positive rate for active surveillance methods when looking at 53 known drug-outcome pairs, including warfarin and bleeding (true positive) and ACE inhibitors and hip fractures (true negative).  

Observational data alone is also insufficient for CER, says Rosenblatt. As recently reported in the New England Journal of Medicine, an observational study of over 1,800 patients concluded that coronary artery bypass surgery has a 1% benefit per year over percutaneous coronary intervention. But some of the confounding factors that biased treatment selection—gender as well as presence of stable angina and three-vessel disease at baseline—may also strongly correlate with mortality.  

Federal legislation as currently drafted allows health plans to use CER information to cover only one drug in a category (i.e. lipid-lowering drugs), despite “heterogeneity of response” for many classes of drugs, says Rosenblatt. Given that only 50% of chronic-disease patients end up on the medicine prescribed for them, study data will need to be adjusted for “drug adherence” lest compliant patients be deprived of an efficacious drug.  

Rosenblatt advocates a cautious approach to CER due to the heterogeneity of data sets and methodologies as well as the pitfalls of retrospective analyses (i.e. erroneously concluding hormone replacement therapy is protective against heart disease). “The data sets make it possible to get precisely the wrong answer and the ‘clinical investigation’ can be done by one person on a laptop with no hypothesis” whose questionable findings go global on the Internet.  

Best practices for CER include replication of important conclusions, access by academic groups and pharmaceutical companies to the same or comparable database used by health systems and regulators, and independent review of study findings prior to dissemination, says Rosenblatt. “Urgency should not trump accuracy,” he adds, because erroneous conclusions “encourage harmful treatments or interfere with important therapies.”  

Post-approval studies in emerging markets  

Summit sponsor Quintiles brought in two speakers to talk about challenges in designing and implementing post-approval studies of any kind in emerging markets. The key success factors—site and patient recruitment and retention—are applicable anywhere, but require a different set of tactics to achieve than in mature markets. Based on presentations by Michelle Bulliard, vice president of clinical operations and regional managing director (Europe) for Quintiles Outcome, and Ken Lee, Quintiles’ vice president and head of site services and feasibility, to triumph in emerging markets study sponsors should:  

  • Start an observational research awareness campaign, use targeted communications during pre-study visits describing scientific benefits of the trial, and foster close collaboration between site staff and project team. Don’t focus solely on key opinion leaders or “overpromise” a study to too many doctors. Help sites get needed regulatory approvals, which often mimic the pathway for clinical trials, and use site agreement templates specific to late-phase research. As much as possible, use electronic data capture (EDC) with automated email reminder features and a data download tool. Realize that some sites in Asia forbid investigator participation in late-phase observational research.  
  • Simplify informed consent documents and use no more patient questionnaires than necessary. Ask sites to submit a screening log and have a backup plan to mitigate lower-than-expected enrollment. Provide support in the local language.  
  • Promptly pay sites and send study updates and personalized messages from the study sponsor as critical milestones are reached. Incite friendly competition between EDC-using sites around meeting certain benchmarks. To counter data-entry burnout, offer EDC users statistical analysis support on their own patients’ data for investigator presentation/publication purposes. Provide laminated cards recapping what to do during patient visits.  
  • Track patients’ age to ensure minors are re-consented upon reaching adulthood. Thank patients and acknowledge their participation via small tokens of appreciation and a follow-up newsletter.  
  • Remember disease variation and treatment practice differences when designing studies. Think local also about affordability and access to medicines. In Asia, medications may be reimbursed in full (Korea, Taiwan, and Japan), on a sometimes/limited basis (Singapore, Malaysia, Thailand, and parts of China), or not at all (Vietnam, Philippines, Indonesia, and Pakistan). Patients paying a large sum of their savings to access medicine won’t necessarily feel obliged to do anything more, like sign an informed consent document. Significant differences between emerging and western markets in terms of epidemiology, etiology, disease management, and how response to therapy gets assessed should factor into study plans; different outcomes can emerge from the same protocol.  

* Post-Approval Summit, Harvard Medical School, May 1-2, 2012