Ten Year Trends, Covid’s Impact, Where Dagdigian Was Wrong

By Allison Proffitt

May 12, 2022 | At Bio-IT World’s 20^th anniversary event last week, Chris Dagdigian and friends from BioTeam, once again closed out programming with a rapid-fire look at the bio-IT landscape and an IT trends assessment—calling out what is working, what’s not, and how it’s all evolving.

Repeating a format he introduced last fall, Dagdigian led a panel of speakers, each commenting on their own experience with bio-IT trends. This year all four speakers were experienced BioTeam consultants, but Dagdigian also flagged Twitter friends who can “speak freely” including James Cuff (@DrCuff, hear his Trends from the Trenches podcast here), Chris Dwan (@fdmnts), @hpcguru, and Corey Quinn (@QuinnyPig).

From his personal vantage point—having given an IT trends talk at Bio-IT World since 2009—Dagdigian began by outlining the trends that have held firm over the past decade. He still starts every year, Dagdigian said, with the existential dread. Science has always changed more rapidly than IT can keep up, and most certainly faster than your IT budget renews. This remains a problem, Dagdigian said, and there is a real risk when IT builds a wrong solution for the scientist.

Cloud, he repeated, remains a capability play, not a cost savings strategy. Capability and flexibility still justify cloud adoption, they do not, however, justify a multi-cloud approach. A multi-cloud strategy is “definitely dumb”, Dagdigian said, while a hybrid cloud approach is “absolutely fine.” Multi-cloud requires developers to devolve applications to the lowest common API denominator. It’s a degraded experience, he said, unless you were “all in” on Kubernetes, which can reasonably port between AWS, Google Cloud, and Microsoft Azure. In his trademark bluntness, Dagdigian said any company with a multi-cloud strategy is a “red flag for poor senior leadership.”

As in years past, moving and managing data is a pain, Dagdigian said, and he again threatened to call out scientists who build careers and publications lists on “data intensive science” but refuse to take responsibility for their own data.

“It’s a shared responsibility model. My job as an IT person is to provide you with safe, durable storage options that are fit for purpose and aligned with what you’re trying to do. The combo between science and IT is to provide end users with tools to manage, govern, pull actionable insights, understand what we’re actually storing. But finally end users have to take some responsibility. That’s the sort of missing piece of the equation. It is wildly inappropriate for IT to make a lot of storage and data management decisions,” he said.

Dagdigian deemed many of the problems that we’ve struggled with in years past “solved problems” including compute, networking, and storage. He called compute “mostly a financial planning exercise” and flagged Internet2 and PetaGene as solid networking options that are no longer hard, risky, or exotic.

He pointed to many vendors in the Bio-IT space that can help with storage that all have strong track records and referenceable customers. He advised starting with object storage or scale-out NAS—only exploring something else if business or scientific needs require.

So Smug, So Wrong

But one of the great attractions to Dagdigian’s annual insights is his willingness—even delight—in point out his past errors. He flagged his own storage failed prediction with zeal: “The future of scientific data at rest is object storage,” he recounted on a slide, and attributed the quote to “some jerk.”

It sounded good! Object storage can be deployed on premises and in the cloud. Metadata tagging is fantastic for scientific data management and search. And object storage is purpose built for a FAIR future in which humans are not the dominant data consumers.

“I am completely, utterly, totally wrong on this,” he said. “We’re still using POSIX Windows or Linux flavored storage.”

It turns out, Dagdigian conceded, scientists do still assume that humans are doing most of the folder browsing, and neither commercial code nor open-source code is object-aware. Scientists who just need to transform data in R or Python don’t have the bandwidth to learn object storage.

In fact, he flagged a death of tiered storage. Machine learning and AI have messed up long-held storage design patterns in the past three to four years, he said.

“The concept of having an archive tier or a nearline tier or a slow tier doesn’t make a lot of sense. If you’re talking about machine learning or AI, you’re really churning through all of your data—old and new—all the time. You’re constantly reevaluating, retraining, pulling out different training sets,” Dagidian said. “I no longer can get away with tiers of different speed and capacity. If I need to satisfy the ML and AI people, you pretty much need one single tier of performant storage.”

The vendor landscape on this new storage structure is crowded, he said, but he highlighted VAST Data, Weka, Hammerspace, and DellEMC.

COVID-Era Trends

Next, Dagdigian turned his attention to the trends arising the past year or two, starting with one of his “single biggest obnoxious problems”: the scarcity of GPUs on the cloud, particularly in Amazon’s US-East-1 availability code. One Boston-based client is building their first AWS footprint in US-East-2, “simply because we cannot get the GPUs that we need, particularly for computational chemistry workloads.”

An increasingly attractive alternative, Dagdigian said, is launching clusters in co-location spaces. He highlighted Markley Group as New England’s best-connected co-location facility and gave a quick outline of the solution he’s placing there: 1.5 Tb of RAM, 40 CPU cores, four Nvidia Tesla V100 GPUs for about $70,000. As part of a hybrid cloud solution, a co-lo cluster creates a hedge against the cloud GPU scarcity or rising cost. He recommends using such solutions to “soak up” computational chemistry, simulations, and other persistent workloads.

From his current personal BioTeam workload, Dagdigian is integrating AWS Parallelcluster with Schrodinger computational chemistry tools and SLURM’s license-aware job scheduling. It may be a niche use case, he conceded, but “an autoscaling HPC grid that understands that you cannot run a job unless a particular license is available” is a “magic nirvana” for us.”

Finally, he zipped through a few “mini-trends” that seem to be at reality inflection points.

Zero-trust, SD-WAN, and SASE networking are “starting to become a little bit real,” he said. One client is using Cato Networks to create a software defined network to stitch together laptops, office buildings, and multiple cloud regions, “with crazy security.” Perimeter81 is another group he’s had some experience with, calling the space in general, “full of innovation.”
A mini-trend causing some concern, he said, is the zero-infrastructure trend, relying on Egnyte, Dropbox, Box.com, or others. Startups leasing space are proud of the fact that they are growing fast and have no on-premises storage, no CTO, no local IT staff, and no IT director. “That’s a little bit of a problem!” Dagdigian said. For starters: Egnyte has a max file size, and it doesn’t support Linux.
Data science has diffused everywhere, and Dagdigian is starting to see IT looking closely at the “operational support, security, and scalability side of data science.” Prompted, in part, by newly remote workforces, companies are working to get everyone on the same tools. Platform-as-a-Service has gained traction, then, to centralize and support a consistent model.