African Centers of Excellence in Bioinformatics and Data-Intensive Sciences: Building The Technology Foundations For Global Growth

By Allison Proffitt 

February 23, 2021 | Shortly after his appointment as the CIO for the National Institute of Allergy and Infectious Diseases (NIAID) in 2003, Mike Tartakovsky renamed the NIAID Office of Technology and Information Systems. Rechristened the NIAID Office of Cyberinfrastructure and Computational Biology (OCICB), the name change reflects the realization that information technology has become a fundamental research tool for the biomedical research community. As he put it in a conversation with Bio-IT World earlier this year: "There are no non-computational disciplines left." 

NIAID was doing its part to share that message by training collaborators and sponsored researchers in Africa, India, and around the globe in advanced bioinformatics. But, Tartakovsky remembered, the institute was missing the mark.  

“The general sentiment from the collaborators was, ‘NIAID provided an excellent training. We know how to use the tools, but we do not have access to these tools. We don’t have the computational infrastructure to apply our knowledge. What can we do?’” Tartakovsky said. Such feedback prompted NIAID to move beyond training and launch the African Centers of Excellence (ACE) in Bioinformatics (now African Centers of Excellence in Bioinformatics and Data-Intensive Science).  

The vision, Tartakovsky explained, was to set up centers of excellence funded through public-private partnerships that will eventually serve beyond their host universities. Two ACE programs have been launched so far. In Mali, the first ACE partnered with the University of Sciences, Techniques, and Technologies of Bamako (USTTB). Such an undertaking has been large and has required input from many players both public and private. The Foundation for the NIH (a non-profit chartered by Congress to support these collaborations and donations for the NIH) coordinates donations. 

ACE Mali launched in 2015, timed to coincide with the launch of USTTB’s graduate degree program in bioinformatics. As the host institution, USTTB “donates” the facilities, electricity, covers cleaning costs, staff, faculty, and more. At its launch, Intel, HPE, EMC, and BioTeam made significant hardware, services, and software donations to ACE Mali. 

Hot, Dry, and Dusty

The University of Bamako has been home to NIAID research initiatives since 1990 when NIAID set up a Malaria Research Laboratory there looking at the mosquito vectors of the malaria parasite. The program grew to include other diseases endemic to the region as well as malaria and Ebola vaccine development work and clinical trials. In 2002, Mali became home to one of the International Centers for Excellence in Research (ICER), an NIH program designed to develop and sustain research programs in disease-endemic countries through partnerships with local scientists. The NIAID built a team of Malian IT staff and the cyberinfrastructure to support this extensive research program. 

So, in many ways, the University of Bamako was a natural location for the first of NIAID’s African Centers of Excellence for Bioinformatics and Data Intensive Sciences. But in myriad other ways, Bamako, Mali, is particularly unsuited to hosting a data center.

Mali is a low-income, land-locked country in West Africa lying almost entirely in the Sahel, the transitional region between the Sahara Desert to the north and the savannah to the south. It’s hot, dusty, and dry. Internet is extremely expensive and unreliable.

And here Chris Whalen and the BioTeam set up a data center.

Whalen is the program lead of the International Biomedical Research Support Program for Research Data and Communication Technologies Corp. in support of both ACE programs. The BioTeam is a scientific computing consultancy dedicated to empowering scientists through advanced technologies and data strategies. In 2015, when then-General Manager of Government Services (now CEO) Ari Berman heard about the ACE vision, he agreed to donate BioTeam’s time and expertise. 

Whalen knew what he was getting into; he has been setting up networks and data centers in Mali since 1998. But for Berman, this was new territory.

Deeply Constrained

Many of the challenges were ones Whalen had tackled before. The dust in the air is inescapable, so he added washable air filters to both the primary and back up air conditioners. The staff wash the filters once a month. 

In the hot and dry season, temperatures can soar to 120 degrees and relative humidity drops to 5%—dangerously low for IT equipment. “One spark, and all of the sudden, you have a fire in your data center!” Whalen said. The water table also drops, the cooling system can’t draw water, and it shuts down. He added a backup cooling system and a water tank on the roof of the data center to maintain humidity. When the UPS (uninterruptible power supply) proved insufficient, he added solar panels and solar batteries. “We can actually run the entire classroom [on solar power for] about 48 hours now.” 

Power is a particularly limiting factor. “Power has turned out to be one of the most difficult challenges that we face operating high-performance computing infrastructure in Africa,” Whalen said. Even though power in Mali is free for the university, it’s painfully unreliable.

Whalen and Berman brainstormed their plan for the Mali installation. “I remember when Ari and I were first talking about it, he had some really grand ideas,” Whalen said. But then he explained the realities of the power supply. “I don’t remember exactly the limits I set on him, but he felt deeply constrained.”

Internet capacity was another constraint that caught Berman by surprise. “One of the things that Ari had hoped to do was [ship the system to Mali and] configure it in place,” Whalen remembers. “When I told him that we were only talking about a 10-megabit internet connection, he was flabbergasted.”

But Whalen had been problem solving in this tough environment for more than 22 years, and Berman and the BioTeam—though usually far more resourced—relish a challenge. “We designed it and installed everything on it, optimized all the software, and set it up so that when it shipped out there it could be stood up and used by them,” Berman said. 

It was a success. With Intel-donated hardware and BioTeam donating configuration and planning, they set up 60 processors with Hyper-threading, 120 cores, a terabyte of memory, and a half a petabyte of storage. “It was probably the most powerful computer in West Africa at that point,” Whalen says. 

That powerful computer powered the ACE Total Learning Center, an optimized classroom to support the University of Bamako’s fledgling bioinformatics program as well as the science faculty. The Total Learning Center hosts 18 stations so that the entire classroom can be run off one UPS. Each station is an HP thin client that connects via microwave connection to the data center on the other side of the Niger River. A large screen at the front is optimized for tele-learning (when the internet connection cooperates).

Take Two: Uganda 

After ACE Mali was up and running, NIAID turned its attention to ACE Uganda. Uganda has, in many ways, the opposite challenges of Mali. While internet connectivity in Mali is a major limitation, power in Uganda is the prohibitively expense. Whalen and Berman are again working together to tackle the challenges. 

“Power in Uganda is amazingly expensive—something like $0.23/kWh,” Berman said. “Now we’re trying to develop a more sustainable, energy conscious type of system that involves things like immersion cooling, or self-contained cooling, or solar power. There are a number of different options that we’re looking at to try to make this a more sustainable unit with reduced cost of energy.”

ACE Uganda also kicked off with new partners. The Infectious Disease Institute at Makerere University in Kampala is the host institution, and the Resource and Education Network of Uganda (RENU) has also greatly supported the ACE. The Texas Advanced Computing Center joined the ACE Uganda project, donating decommissioned racks from Stampede One—one of their supercomputers—and sharing expertise on running high performance computers on solar power.

Virtual Uganda

In addition to a teaching lab, ACE Uganda has a Visualization Lab. The Viz Lab is the brainchild of Darrell Hurt, Chief of the Bioinformatics and Computational Biosciences Branch at NIAID and Meghan McCarthy, Project Lead, NIH 3D Print Exchange within BCBB. They had created a virtual reality visualization lab at NIAID and were enamored of the possibilities for education, research, and collaboration.

“When it came time to launch the ACE Center in Uganda, we said, ‘Well, you know what? We created a Viz Lab here. Much of what ACE is about is creating a service center for bioinformatics in Africa. That’s what we are; we’re a service center for bioinformatics at NIAID,” Hurt said.

Hurt, McCarthy, and their group set out to replicate NIAID’s Viz Lab in Kampala.

“We had this amazing opportunity to actually start from scratch with dedicated space, getting all the best hardware and tech,” McCarthy said. “Actually, what went into the viz lab in Uganda is more advanced and newer hardware than we were using at the time at NIAID.” 

The Viz Lab features custom-built workstations with a modular configuration, each with a small form-factor PC that is connected to an external NVIDIA GPU. “It’s not a behemoth, a giant, huge desktop PC with all this hardware in it… If something fails, you just swap out,” McCarthy said. “It’s not a full system failure.”

The Viz Lab is also equipped with head-mounted displays, screens for interactive virtual learning, software for VR, and—most recently—masks that fit under the VR rig. Hurt and McCarthy are working with Tom Goddard and others from the RBVI at the University of California, San Francisco. Their ChimeraX software is a next-generation molecular visualization and analysis program.

From the US, the team is using the Viz Lab to better understand the kind of infrastructure needed for a seamless, shared VR experience. “We’re trying to understand things like bandwidth, latency, and network,” Hurt explained. “If we should be leveraging Internet2 and other research and education networks, or if we should be doing other kinds of network connectivity.” 

And in Kampala, staff from the University’s Infectious Disease Institute are also using the Viz Lab and ChimeraX’s capacity to view and interact with proteins, communicate and train groups virtually, and more.

“I think, in some ways, that the Viz Lab is one of the most used parts now,” said Hurt. Although NIAID staff have not been able to travel to ACE Uganda for a year, he said the lab gets near-daily use for demonstrations, research, or communications. “It’s really satisfying to see that even though it’s something brand new for the people there, it is being embraced.”

On The Back Side Of A Pandemic 

While ACE Mali and ACE Uganda have both seen significant progress, moving toward sustainability is still a challenge.

This is a long-time collaboration, Tartakovsky admits, outlining a three-phase maturation plan for each ACE that he expects to take seven to 10 years. “The first stage is an education, and that’s why we have a partnership with the university. It cannot be done without the sustainable pipeline of young people being educated and understanding what the modern science is, and how to apply computational components to the modern science,” he explained. 

The next step is post-educational training, after students graduate and begin work on real projects. “Phase three is the most ambitious phase, where we want to move the Center for Excellence: where they will become self-sustainable service centers for the regional research,” Tartakovsky said. “We strategically put one in West Africa, in Mali. Another is in east-central if you will, in Uganda,” Tartakovsky said. These centers—and potentially others—are positioned to support regional research.  

Tartakovsky is extremely encouraged by the progress he’s already seen. ACE Mali has started its fourth master’s cohort and has graduated about 30 students in bioinformatics already. Several graduates are pursuing PhDs. “That’s a quite impressive  progress, they are also now enrolling people into that program from other countries in West Africa!” Tartakovsky exclaims.  

Whalen agrees. “Seeing the investigators there publishing in high-impact journals and getting competitive grants from the premier sponsors of research in the world; seeing the work that they’re doing in collaboration with the American scientists—that, to me, is one of the most rewarding things. I can’t think of anything that would be more valuable than that.”

Of course, the program needs are still high and self-sustainability is a long way away. Tartakovsky hopes that the international community will continue to support these new graduates and the Centers that facilitated their training. “At the end of the day, we are dealing with countries that, despite all their commitment to modernization, are sometimes limited on the resources they can invest in this area,” he said.

Private companies can continue to partner with the effort by making in-kind donations through the Foundation for the NIH. Academic groups can partner to help facilitate local staff training so they have the skills to support the infrastructure.  

And, Tartakovsky pointed out, centers can be supported toward self-sustainability by collaborative partnerships with academic and research groups from the US, EU, and around the world.  

“Rather than exporting data from African countries to do the analysis someplace else, please leverage the existing capabilities!” Tartakovsky pleaded. “You will be amazed at how much the research capacity in Africa has grown. Innovation can’t happen without research and research can’t happen without scientists. Supporting African scientists creates jobs and matures the bioinformatics landscape of the continent.” 

And there’s payoff at home as well, Whalen pointed out. “If you think about it from the perspective of response to global pathogens or emerging or reemerging pathogens, having partners who are on the ground in places all over the world, who are ready and able to be collaborators in response, is really increasing our defense posture against these kinds of pathogens.”