PrecisionFDA to Test Accuracy of Genomic Analysis Tools

By Aaron Krol

August 27, 2015 | As the FDA prepares to take a more active role in the oversight of genetic tests, the agency is finding itself in unfamiliar territory: regulating software.

In the past, reviewing new diagnostics rarely involved stepping far outside the wet lab. Tests might involve finding a pathogen under the microscope, or detecting an analyte through a chemical reaction in a test tube: simple yes or no questions that just had to be checked to make sure they reliably gave the right answers.

In genetics, however, data has to go through a lot of checkpoints before it gets to yes or no. Can a new computational pipeline crawl through a string of DNA code to find genetic variants? Does it always call those variants correctly ― and are they really linked to the disease you want to diagnose?

These are questions that even professional bioinformaticians have sometimes struggled to answer. The trouble is, virtually all DNA data is delivered and mediated through computer programs, making it hard to know what the ground truth is. Even if workflows consistently give the same answers, it’s difficult to be sure they’re telling us what’s really in our genomes, and not just falling victim to the same common biases and errors.

With the federal government’s dismal reputation for handling IT projects, you might think the FDA is the last organization we’d want trying to tackle this complex issue, especially with the safety and accuracy of our medical testing on the line. But in fact, the FDA has a record of success and flexibility in IT. Last year, the agency formed a new Office of Health Informatics under Chief Health Informatics Officer Taha Kass-Hout, who has moved swiftly to transition to more reliance on cloud computing and more agile and open software development. His office’s first project, openFDA, was widely praised for making it easier for the public to get useful information out of FDA databases covering drug side effects, product recalls, and medical device defects.

Now, the Office of Health Informatics has embarked on a new project called precisionFDA, in which the FDA will try to better understand how accurate computational pipelines really are at interpreting genetic data. An initial research and development grant to build the platform has been awarded to DNAnexus, a company whose cloud-based service has set records for rapid genomic analysis at a huge scale.

The first goal of the precisionFDA project is to fulfill its regulatory mission. The platform will be loaded with extremely well-curated “truth sets” of DNA data, such as those produced by the Genome in a Bottle Consortium ― genomes so thoroughly studied that we can say with high confidence whether computational pipelines are getting the right answers out of them. The platform will also include “reference pipelines,” standard sets of computational tools that represent the state of the art in analysis. Combined, these materials will let users compare their custom workflows against common standards, scoring their accuracy and identifying areas where they fall short. Eventually, the FDA will be able to use this platform to gauge whether new sets of tools can be trusted to make diagnoses in real patients.

But precisionFDA also serves a second purpose. Since the FDA announced its decision to fold laboratory developed tests, including the large majority of genetic tests, into the same regulatory procedures it uses for mass market diagnostics, the companies and hospital labs who deliver this testing have worried that the agency will stifle innovation in the burgeoning field of genomic medicine. By building precisionFDA on open source software, and actively seeking input from the institutions who will be regulated, the FDA is hoping to offer transparency in its regulatory process and win some community buy-in for its standards.

“This really is something we view as a community effort,” says David Litwack, a policy advisor on personalized medicine in the FDA Office of In Vitro Diagnostics and Radiological Health. “It is an open platform, and it’s a chance for people who are developing and working in this space to really have an effect on how FDA looks at genomics.”

The experience of building openFDA has also taught the agency how an open and inclusive development process can create powerful new functions for big IT projects. “The initial intention for openFDA was to provide easier access to already publicly available data, but we’ve seen how further innovation happened from the marketplace and from users, especially the researcher community,” Kass-Hout tells Bio-IT World. “So hopefully some surprise will come from the community that we haven’t thought about. This is the beauty of making this open.”

“A Groundswell of Activity”

Omar Serang, Chief Cloud Officer of DNAnexus, became connected with the FDA Office of Health Informatics when his company attended the ribbon-cutting ceremony for openFDA. “Our jaws dropped to see a federal agency using GitHub, holding code hackathons,” he says. “It was a crazy collision of Silicon Valley culture and federal government culture.”

DNAnexus was already a party to industry discussions about collecting better reference material for testing out new genomic technologies. Serang says he’s spoken with testing companies like 23andMe and Counsyl, DNA sequencing giant Illumina, academic institutes like the Baylor College of Medicine, and a group at the Centers for Disease Control and Prevention led by Lisa Callman about what kinds of DNA samples would make the best reference material, and how to make them universally accessible. Those conversations are now informing the construction of precisionFDA, as DNAnexus has taken on the contract for that project.

“We don’t have enough ground truth out there, covering enough ethnicities, covering enough genomic makeups,” Serang says. “There’s a groundswell of community activity already taking place that precisionFDA can capitalize on.”

With help from various industry and academic groups, Serang and his team at DNAnexus are compiling a list of essential reference materials to make available in precisionFDA at launch. These will likely include the Genome in a Bottle assembly of a genome called NA12878; Illumina’s “platinum genome” assembly of the same specimen; and the “Lupski Genome” assembled at Baylor.

But this handful of resources will represent only a fraction of the material needed to accurately assess how pipelines perform across the spectrum of human genomic variation. That will be especially true of pipelines used in diagnostic tests: users will want to see whether their tools can catch the thousands of different variants involved in rare hereditary diseases, for instance, or different subtypes of cancer.

To include enough material to satisfy all these requirements, precisionFDA is taking three different approaches. First, Serang says it’s very likely the precisionFDA team and its partners will collect new DNA data for specific niche purposes. “The initial pilot is focusing on carrier screening and Mendelian disease,” he says. As the project continues, it will also become important to collect more complex sources of variation, like DNA from the immune-related HLA region, or tumor-normal pairs that will test the ability of tools to detect large structural variants involved in cancer.

Second, DNAnexus is building programs to generate in silico data of high enough quality to mimic real DNA reads. Serang says this function will have two components. The first, a “FASTQ synthesizer,” will create artificial FASTQ files ― a common format for DNA data ― that have similar characteristics to real data from a sequencer, including occasional errors. The second, a “FASTQ injector,” will be able to take real or synthetic FASTQ libraries and spike them with a few artificial reads that users want their pipelines to be able to find. For instance, the FASTQ injector might add some known cystic fibrosis mutations to an otherwise healthy genome, so that a diagnostic test for cystic fibrosis could be evaluated inside precisionFDA even in the absence of real disease samples.

Most importantly, however, users of precisionFDA will be able to add any data of their choosing to the platform. This can be done publicly, to share useful datasets with the FDA and the broader community, or in a private environment within precisionFDA purely for users to see how their tools perform.

“At this stage in the pilot, this is not about validation or approval,” says Serang. “This is all about vetting and analysis, and to be able to have the community feeling very safe working in their private data spaces.”

New Frontiers for the FDA

PrecisionFDA will not be the first project to tackle comparisons between genomic analysis tools. Resources like GCAT (which has collaborated with Genome in a Bottle) exist to score pipelines and give a rough idea of which common tools are most reliable in which situations. But Serang thinks DNAnexus can not only give these comparisons a higher profile and a role in regulation, but also make them easier to use.

“What we’re finding is that the work that’s most advanced algorithmically is also sort of behind from a user interface perspective,” he says. “Our commitment is to bringing the best call comparator to bear with a more future-friendly front end.”

To that end, DNAnexus is building precisionFDA on top of the commercial DNAnexus platform. The new precisionFDA web portal will feature open APIs for running tasks and getting back comparisons with other pipelines, and will eventually be free to access for any users who want to assess their tools’ reliability. The FDA is aiming to make a beta release available this December.

“PrecisionFDA can at least offer people the necessary resources to validate their tests, and that itself we believe to be a huge way of bringing innovation to this marketplace,” says Kass-Hout.

It’s a marketplace that is changing quickly, giving the FDA new and fluid responsibilities. Past forms of FDA review have generally focused on a single use case for each test, but genomic devices could easily give results across hundreds of different diseases areas. That makes it much more important to check their accuracy upstream in the testing process, rather than solely in terms of clinical results.

“We recognize that the way tests are being used, and almost certainly will be used in the future, is almost use-independent, or use-blind,” says Litwack. “That’s a challenge we’re responding to with this technology.”

Although it will probably be a while before precisionFDA can formally be used to evaluate tests during new product reviews, even beta users will be playing an important role in defining future regulations. Right now, the standards for genomic software are often unclear; only a handful of genomic tests or testing platforms have been cleared by the FDA for clinical use, and their reviews had to be heavily negotiated. By testing pipelines within precisionFDA, the agency and the organizations it regulates will be able to reach some consensus on what aspects of genomic testing give computational tools the most trouble, and what metrics can best evaluate those pain points.

“There will be contributions from the community around reference materials, or adding their own pipelines or workflows to replicate results,” says Kass-Hout. “OpenFDA used a very similar model, this agile, dynamic way of implementing a platform where people were able to build applications or websites on top… For us at FDA, we’re breaking new frontiers in ways we’re adopting this for science.”