Intel, Penn Medicine Launch Federated Learning Model For Brain Tumors

By Allison Proffitt

May 28, 2020 | The University of Pennsylvania and Intel have built a federation of 30 institutions to use federated learning to train artificial intelligence (AI) models to identify boundaries of brain tumors.

Led by Spyridon Bakas at the Center for Biomedical Image Computing and Analytics (CBICA) at the Perelman School of Medicine at the University of Pennsylvania, the federation is the next step forward in a years-long effort to gather data that would empower AI in brain image analysis.

“During my last few years of research, I’ve been focusing on learning predictive models of brain tumor segmentation, predicting genomics, predicting overall survival of patients from baseline scans…, evaluating the propensity of the tumor to be more aggressive by various means, toward improving the clinical outcome of patients,” Bakas told Bio-IT World.

Brain tumors have multiple, sensitive sub-compartments that show up in different scans in different ways: active parts of the tumor that are really vascularized, necrotic parts of the tumor where tissue has died, and parts of tumors that are infiltrating healthy tissue.

“An abnormality in the brain is fairly easy for someone to spot,” Bakas said. Applying AI to the problem, “is not to detect whether there is a tumor or not; it’s to detect the boundaries of the tumor.”

Bakas has been leading the International Brain Tumor Segmentation Challenge (BraTS) since 2017. The annual challenges, which began in 2012, evaluates state-of-the-art methods for the segmentation of brain tumors in multimodal magnetic resonance imaging (MRI) scans. Along the way, Bakas explained, he became very familiar with the hurdles of putting together a centralized dataset to serve as the community benchmark dataset for BraTS. Participants were concerned about privacy both from institutional and cultural perspectives. Organizations were only willing to share MRI images on a few dozen patients.

After nearly ten years of the BraTS challenge, the group has assembled a cumulative dataset from 650 adult patients consisting of four images from each patient.

But each member institution has at least this much data, Bakas explained. “If there were no privacy or data ownership concerns, we should have instead 650 x 21 [patients]—a full dataset from each of the institutions involved in BraTS.”

Surely there was a solution.

Bakas, a computational scientist by training, began thinking of machine learning approaches. At a conference in early 2018 he met researchers from Intel AI and Intel Labs. “We came up with this idea: utilizing federated machine learning in medicine for medical images.”

Enter Intel

Federated learning was first introduced by Google in 2017 and used to train autocorrect models for texting. Data owners train a model architecture locally using only their own data. Each model is then shared with an aggregation server, which creates a consensus model of all the accumulated knowledge from the data owners, even though the raw data never leave their institutions.

Jason Martin, principal engineer at Intel Labs and a privacy researcher, had been exploring ways to apply federated learning, particularly in privacy-sensitive domains.

“As we looked at [federated learning], we realized we could contribute some security and privacy technology and domain knowledge to it,” Martin said. “We began looking at what important use cases would benefit from this sort of privacy-preserving federated learning.”

Medical images made sense, so Bakas, Martin, and their colleagues started with a feasibility study: creating a faux federated learning system using the BraTS data already in hand. Could a federated learning model train an algorithm to the same accuracy as pooled data?

The team split the donated BraTS data roughly in half. Data from nine institutions was used for validation and training of the pooled model; data from 10 institutions was split by home institution and trained in a federated way—as if it had never been shared at all.

“We trained an AI model on the centralized data that was all shared. Then we obtained performance X,” Bakas explained. “Then we took the data, distributed it as if it was not shared, and applied the federated approach. We trained the model on each institution’s data and we aggregated it in a way to create a consensus model.

The team found that the two groups performed nearly identically. The performance of the federated semantic segmentation models on multimodal brain scans had a Dice Coefficient of 0.852 while the models trained by sharing data had a Dice Coefficient of 0.862. (A Dice Coefficient of 1 would be perfect alignment between the model and reality.)

“The federated model was performing 99% to the performance of the model trained on centralized data,” Bakas said.

This feasibility study was delivered in 2018 at the International MICCAI Brainlesion Workshop (DOI: 10.1007/978-3-030-11723-8_9). Bakas and Martin are both authors.

“We were the first to demonstrate that if you split that data back up to the original 10 medical institutions that donated it, and treated them as a federation instead of a central donation, that you could get almost the same segmentation performance out of the machine learning model,” Martin said. “That was very exciting!”

And it’s on that foundation that the new federation will be built.

Bakas was awarded a grant by the Informatics Technology for Cancer Research (ITCR) program of the National Cancer Institute (NCI) of the National Institutes of Health (NIH) to explore the idea further. The three-year, $1.2 million grant funds Penn Medicine’s work developing the Federated Tumor Segmentation (FeTS) platform, open-source toolkit, and user-friendly GUI. Penn Medicine brings the medical imaging and machine learning competence, Bakas explained. Intel Labs is the technology provider and brings all the security and privacy expertise.

“We’ve been working with Spyros to create a real federated learning platform—the actual software and infrastructure components that are necessary for multiple institutions to collaborate across the public internet,” Martin said.

New Federation, New Questions

Although the feasibility study was very promising for the accuracy of federated learning, there are still questions to be answered. How would such a federation behave in the real world? What are the privacy risks? How much work will federated learning be for the data-hosting institutions?

“We know some of the answers to those questions, and frankly some of them I expect we’re going to learn as we go,” Martin said.

The original BraTS group included 19 institutions; the current federation numbers 30 groups. All of the federation members are research institutions and have committed research effort, not just data, to the project.

There will be significant work for data host institutions, at least at first. The federated learning model is a supervised learning model. The data must be annotated by hand. In this case, a radiologist at each member organization will label tumors in the MRI images according to a protocol established through BraTS. That is a costly step. In the future—at a much larger scale—a federated learning model could shift to unsupervised learning, lessening some of the load for participating institutions.

Other big questions for data hosts will focus on privacy and bandwidth. Martin expects the earliest federation members will likely move the data they have committed to use for model training to a separate server for greater privacy. In theory, of course, those data won’t leave the institution, but the model architecture will come and go via the internet. “It will take some time, I think, to gain comfort with something that has internet connectivity being connected to, say, the healthcare systems themselves,” he said.

Some of the challenges in a federated learning model are simply social and relational: building trust within the group. It’s very important, Martin argues, that all of the participants trust that others are playing by the same rules. Hardware can help.

“A lot of our focus here at Intel has been on ensuring that, when [the federation is] deployed, the parties participating can trust each other to contribute fairly, and that the rules of engagement—if you will—are enforced by the hardware platforms and the software that we’re developing,” Martin explained.

But the whole model still requires interpersonal trust.

“The platform we’re creating still requires that you come to the table—or virtual table—and you decide on your experiments or your goals,” Martin said. “Once you agree on them, then those are the rules that we want to enforce through the mechanisms in the system architecture. What you agree to at the table can’t be compromised either intentionally or unintentionally by malware or bad actors that come into the system.”

Open Model

The Federation itself is just launching. Martin’s team is tackling, “what you might consider some of the boring parts of the federation,” he explained: “getting the number of connections working and such. We’re hardening that software to the point where it can be deployed to the remainder of the federation.”

Bakas and his lab are, “in the process of actually serving the algorithm for phase two to the outside institutions.” The software will be made available open source, he said. “We want this framework to be publicly available. We don’t want to create a proprietary software that someone needs to pay a license or use to build upon.” Bakas believes the framework could be easily applied to other disease areas or even data types.

Once the software is in place, each “federated round”—one cycle of training the model architecture at the data hosts and then sending it back to the aggregator—will take “on the scale of days to weeks,” Martin predicts.

This won’t be a one-and-done circuit. Returning the consensus model back to the member institutions will deliver clinical impact, Bakas says, and it will only improve as more diverse data are included. The model will improve as well. Federated learning enables researchers to explore new model architectures—hyperparameters—and tune the model. Bakas and his team at Penn Medicine will be able to refine the machine learning models with the additional data federated learning makes available.