precisionFDA Announces Winners of Consistency Challenge

By Jacque-Remy Orvis

June 3, 2016 | The U.S. Food and Drug Administration (FDA) commissioner Dr. Robert Califf announced the winners of the first precisionFDA community challenge last month. Awards for the Consistency Challenge were given to Rafael Aldana and his team from Sentieon for Best Performance and Highest Reproducibility, and to Deepak Grover from Sanofi-Genzyme for Highest Accuracy.

The Consistency Challenge is the first in a series of nine community challenges FDA is hosting within precisionFDA, the platform built to assess genomic analysis pipelines that recently garnered its own accolades as a Best Practices winner at the Bio-IT World Conference & Expo.

The Consistency Challenge served as an opportunity to compare mapping and variant calling pipelines to a “truth set” provided by the Genome in a Bottle consortium. Participants processed a batch of raw sequencing data from a well-characterized human cell line, and compared their pipelines’ results to a gold standard VCF file provided by the Genome in a Bottle project.

“We are excited and honored about the [Consistency Challenge] award,” says Jun Ye, CEO of Sentieon, a small Mountain View-based bioinformatics software company new to the genomics industry. The Sentieon team tested the robustness and validity of their “dual-mapping pipeline,” and was honored for best performance and highest reproducibility.

“It’s very important to have these kinds of challenges. [They will] help harmonize the industry,” continues Ye. “Before precisionFDA, there wasn’t a standard, commonly accepted way to benchmark bioinformatics tools; everyone used different criteria and different data, so there was no common standard to say which one is best. This is a great first step for the community.”

The Consistency Challenge closed on April 25, 2016, with 21 total entries from 17 submitting parties. Entries were graded on reproducibility, recall, precision, and accuracy.

“We could write rules, but that would hinder innovation,” says Taha Kass-Hout, M.D., M.S., FDA’s Chief Health Informatics Officer (CHIO) and Director of FDA’s Office of Health Informatics. “[This community input] will help us regulate this industry in the future, but in a more rapid, more growing way, rather than just one regulation set it stone.”

DNAnexus was contracted to build the precisionFDA platform, and they also act as contributing members to the precisionFDA community. Mike Lin, VP of an R&D group at DNAnexus and participant in the challenges, called precisionFDA as a whole a “tremendous opportunity” to learn from other groups. Lin didn’t use a proprietary pipeline in the Consistency Challenge; his team entered Fermikit, an open source pipeline by Heng Li. “We took widely used, in some cases underappreciated, open-source pipelines that we know our customers are interested in but are unsure of.”

FDA’s Kass-Hout says precisionFDA was designed as a hub for genomics, software, and other communities to collaborate on solving the regulation gap in next-generation sequencing. Providing developer tools, reference materials, and computational power to facilitate growth in existing pipeline developments as well as new ones, precisionFDA is also a social interface for the community to confer about improving regulations and security in genetic testing as the FDA gears up for Obama’s Precision Medicine Initiative announced in January 2015.

The FDA asked the genomics community how to enable it to improve data analysis tools and keep up with the maturing genomics industry. “The community is working with us to figure out new ways to evaluate [pipelines], “says Zivana Tezak, associate director for Science and Technology at FDA’s Office of In Vitro Diagnostics and Radiological Health, Center for Devices and Radiological Health. “It’s not just the FDA making a decision. Everyone gets to contribute.”

“It’s a fun experience for the community,” adds Kass-Hout, who led the precisionFDA initiative. He calls precisionFDA a neutral playground or sandbox. “Software is always improving,” he says. “We want to provide a forum for people to be able to innovate and have pride in what they have done, whether it is open source or something they have a trade secret behind, but still be able to come meet in an area to compare results.”

In return, the genomics community will help the FDA single out regulatory gaps in software.

The opportunity to contribute is self-serving, says winner Deepak Grover of Sanofi-Genzyme. “[Companies] have to be in tune with what’s going on in the outside world,” he says. By participating in the challenges, “you can share and contribute and also learn. It’s a two-way street in terms of methodologies. And you can use those methodologies to enhance the general portfolio of the company.”

As the winners of the Consistency Challenge were announced, precisionFDA wrapped up the final days of the second research test, the Truth Challenge, for genomics community members to identify genetic variants in one known and one unknown sample dataset using their choice of genetic pipeline.

“With the first challenge, we were looking for an honest assessment of your pipeline,” says Elaine Johanson, precisionFDA project manager and deputy director of FDA’s Office of Health Informatics. “It was an opportunity for people to get their feet wet.”

For the second challenge, precisionFDA withheld the truth data. It was not until after the challenge closed that the Genome in the Bottle consortium released the truth dataset.

“It’s going to be interesting now to calculate everyone’s results,” says Johanson. “And whether they tuned their pipelines in the sample in the first challenge.”