Big Plans At The Broad

By Allison Proffitt

September 15, 2016 | Bill Mayo had been working at the Broad Institute for less than a week when someone asked him: Hey, Bill, what are you going to do? “I said I’d be an idiot if I had an answer already,” Mayo told me last month, sitting in his corner office in the Broad Institute’s Ames Street building.

Now—seven months later—the newest Chief Information Officer of the Broad Institute has some answers.

Dressed in jeans, leaned back in “Dick Cavett chairs”, Mayo sells his vision with a marketer’s acumen. He plans to do more than tend to the Broad Institute’s “15,000-ish core” high performance computing cluster and manage the on-going migration to the cloud. Mayo wants to shift the technical identity of the 11-year-old institution.

The Broad didn’t, “need another HPC guy,” Mayo said. “We’re world class HPC... We need[ed] someone who knows how to really build a world class IT organization for the world class institute that we have become.”

And so Mayo finds himself leading a team of some of the best in the business—Chris Dwan, Director, IT Architecture and Strategy; Katie Shakun, Associate Director, IT Operations; Ian Poynter, Associate Director, Information Security; and Robert Damian, Director, Enterprise Systems—none of whom he knew before arriving at the Broad. With the Broad’s HPC experts available, Mayo finds himself with space to set some big goals and address some larger questions.

Big Questions

With his team, he’s asking professional development questions of the Broad’s many analysts, service professionals, HPC Professionals, and scientific liaisons on his team: “How do we cultivate talent within the team? How do we help people figure out what matters to them? How do we help them get the right training? How do we have an org structure that doesn’t get in the way of people broadening their skill set?”

Mayo rattles off a range of big challenges: Changing sharing models to give researchers access to data instead of copying the data and shipping it to them - something that speeds research, better secures data and lowers the cost of working with it. He wants to find a solution for maintaining data sets well after a grant wraps up and the program behind it moves on—a common problem in academic research that he calls a tragedy. He wonders about the role of Machine Learning expertise in speeding access to data while ensuring only scientifically valid, medically appropriate, properly consented questions are asked of the data.

Another challenge: how to share data across collaborators using multiple on-premises and cloud partners—an idea he refers to as building a "data Switzerland". He is exploring partnerships to think about the role of blockchain technology in building an identity management structure with no central authority.

And on it goes. "One of my favorite things about The Broad is our belief that anything can be done, we just need to find the right partners and commit to it," he wrote in a follow up email after our initial conversation.

He also wants to tackle an organizational challenge, one that impacts many research institutions, where there are a mix of employees and collaborators, who need to tap into systems seamlessly. “Two-thirds of my user population — two-thirds of all Broadies—are employed by one of our partner institutions. Fully two-thirds,” Mayo says, so he’s working on the best ways to connect those users, and their home institutions, and make their data and systems accessible.

The team is also working on better aligning policies with these partners, and building strong external relationships for the Broad with other groups.

For example, “We had a process where we bill for storage based on how much [storage capacity] is used on a given day of the month. It encourages people to purge by having it not an average of the month but by having it on a fixed day,” Mayo explained. At a partner institution —which shares several investigators with the Broad Institute—the CIO instituted a similar policy, but the billing date is different.

“We can watch the network and we can watch all kinds of data move from us to them just before our billing and then move back just before his billing. That’s wasted researcher time,” Mayo laments. “Let’s just agree on the same day of the month.”

Goal Number Two

Mayo and his team worked together to set their goals: four major efforts each with a handful of bullets underneath. In building the list, Mayo and his team scrawled flow charts and taped spreadsheets and lists to a wall papered in melamine across from a row of cubicles. The thermostat falls between the operating budget and metrics. There are as many question marks as answers, and feedback loops for partnerships and strategic choices.

His expected job functions—“implement this system; upgrade that”—all fall under “sub-bullet three of goal number two,” Mayo laughs.

Balancing the Broad’s on-premises high performance computing cluster and its cloud real estate fits there. He believes the Broad is one of the largest consumers of both types of capacity. “Moving to the cloud in a major way for scientific HPC is difficult. I think we predicted we were going to be done in three months or six months or nine months or whatever and we’re still in the middle of it.”

The Broad is using between about 30 and 40 petabytes of storage now. Between 20 and 30 petabytes are local, and Mayo says cloud-based compute has grown from zero to 12 petabytes in the past year and a half. “That has clearly blunted what would’ve had to happen on prem. But… on prem grew by two petabytes last year,” he said. “We do predict the on prem curve turning downward but we made very aggressive assumptions about how fast people would move to cloud and it’s still only a gentle downward flow. It’s not like it falls off a cliff and goes away.”

Part of Mayo’s first priorities, then, has been hardware and software refresh on a still-growing on premises cluster.

“We still need as much as we always had… We flattened the curve a bunch, but it’s still actually climbing up and we got a little behind in maintenance. So this year we’ve got a big maintenance refresh program in place and we’ve already executed on a lot of that.

Like everything else on the planning wall, capacity across the Broad’s platforms, programs, and areas of focus are tracked on a matrix taped between the technology label and a projections list. Red dots reflect areas where, “lack of resources is currently causing missed opportunities.” Both the compute and data storage rows have red dots.

It’s all terribly transparent. “Everybody has appropriate access to everything to ask valid questions,” Mayo said.

Being able to ask valid questions is an underlying principle to Mayo’s theme for the team: “thoroughly enjoying your job.” It’s the answer he had already committed to for himself the first week on the job when the team member asked him what he planned to do.

To Mayo, thoroughly enjoying the job means more than Friday night pints at Meadhall, a craft brew and mead pub across the street from the Institute. “Thoroughly enjoying yourself is being good at what you do because it’s no fun if you’re not good at what you’re doing. Be part of something big; it’s really no fun to do boring stuff. Be part of a team, know who you’re working with, connect, and all that kind of stuff. And be on the right side of history.”

Boston Guy at the Broad

It’s a mantra Mayo only recently adopted. For almost 20 years, Mayo worked his way up through the ranks at Gillette. He directed portions of the company’s worldwide information systems, and managed the IT underpinning the North American value chain. He said he’d likely still be there “if Proctor & Gamble didn’t come buy us [in 2005] and say, ‘Hey, why don’t you move to Cincinnati?’ I’m like, ‘Hey, I’m kind of a Boston guy.’”

Mayo decided to stay in New England, and ended up teaching leadership and communications courses at Northeastern University, running global applications at ModusLink, and directing supply chain and commercial operations IT at Biogen.

The majority of his experience still comes from Gillette, and Mayo believes his time at the 100-year old company is well-applied at the Broad.

“[Gillette] was an amazing training ground. We knew what good looked like. We were addressing all the core operational issues, which are the same things that apply today:… How do I manage what’s going on? How do I build insight into my finances? How do I have good understanding of my operations? How do I make smart decisions? How do I be consistent? Etc. All those things existed. We just didn’t know about Docker packets and Puppet and Chef and cloud and whatever. But they’re all there.”

Now Mayo is applying that operations expertise to new questions. On one of the outer branches of the idea tree growing down the hallway, there’s a graph Mayo and Chris Dwan drew together.

“There’s possible and impossible; and there’s conceivable and inconceivable. Draw the line between the inconceivable and the impossible and we think our job is to bring new things over that line. That’s the way we’re telling the story here nowadays.”

Editor’s note: Bill Mayo will be joining the Plenary CIO panel at the 2017 Bio-IT World Conference & Expo, May 23-25, Boston.