Successful Data Migration Projects Take Planning

Contributed Commentary By Dan Wheeler

August 10, 2020 | I used to hate migration projects. Early in my career, I once vowed to a customer that I would never work on another migration project again. And while that project was considered a success, it didn’t feel like a success to me. It was frustrating, stressful and I still have scars to show for it. However, as an IT consultant working in the Life Sciences industry, avoiding migration projects wasn’t realistic.

Given all the merger and divestiture activity and companies not investing in their solutions and allowing them to go obsolete (a topic for another day), there would always be a need to move data from one system to another. Realizing that avoidance was not the answer, I reflected on the challenges that we faced on that initial migration project and looked for ways to make things less painful in the future. Today, when I look back on the dozens of migration projects I’ve worked on over the last 20 years, the most successful projects have incorporated all, or most, of the elements described below.

It starts with planning. This may sound obvious, but I’ve seen too many times where either a Data Migration Plan didn’t exist or it was written at too high a level to be useful. For example, I worked on one project where we were months into the project before the topic of migration testing was discussed. The migration team assumed this would be covered in the overall Test Plan and be performed by the Test Team, while the Test Team said its scope was limited to testing the application, not the migrated data. So a migration test strategy had to be designed on the fly and required additional resources (cost) to fill the gap. Have the conversations regarding scope, timing, testing approach, what documentation is required and who is approving it, at the beginning of the project to avoid unpleasant surprises downstream.

The second element of a good migration project is ensuring alignment with the application workstream. Many times, data migrations are done in conjunction with a new system implementation, which requires its own configuration and alignment with business processes. The migration team cannot work in a vacuum and ignore the outcome of the application workstream. Some of this is straightforward—a migration team can’t define migration mapping rules if the target system data model is not solidified.

However, the migration team needs to know more than just what are the target system data elements; they need to know why they are there and how they will be used by the business. For example, on one project we found a field “Department” was configured in the target system. The source system didn’t have a similar value, so the migration team suggested to either leave it blank or set some default value. However, as the team dug into the target system configuration, it was found this field helped determine the security applied to the migrated document. This required the migration team to take a closer look at defining migration rules or allowing for data enrichment to set this field properly so that the right security was applied.

Another must for a successful migration project is conducting migration prototypes and getting user feedback. Defining migration rules usually involves many conversations and are often documented in ways that are not very readable to an end user. The best way for a user to see the migration requirement in action is to see a subset of the data migrated into the target system. For sure, running these migration prototypes is helpful to the technical team to ensure the migration can be run end-to-end with no issues like connectivity or performance. However, these migration prototypes are most helpful to the business to ensure the requirements provided on paper make sense in an actual system.

The final element of a successful migration project is defining a strategy for data cleansing and enrichment. I was never on a project where source system data didn’t require some type of manipulation in order to make it fit into the target system. Some of this can be attributed to “dirty” data, such as having a free text “Product” field where you have 15 permutations of “Aspirin,” but some are due to data standard changes that have evolved over the years. Regardless of the data’s state, the approach for how to align it with the target must be discussed. Does the business have the ability to change source system data from the user interface, or can it have IT make back-end changes? Sometimes neither will work and data enrichment spreadsheets need to be introduced that will allow users to enter the values. Introducing data enrichment spreadsheets, while sometimes necessary, introduces their own challenges in terms of managing the deltas. This approach should also be documented in the Data Migration Plan.

Make no mistake, migration projects are challenging even when following all of the elements described above. But a methodical approach to these projects certainly improves outcomes and lessens the pain experienced by all. Or perhaps my tolerance for pain has just increased over the years!

Dan Wheeler is founder & CEO of Daelight Solutions. With more than 20 years of experience as an entrepreneur, senior executive and management consultant, Dan enjoys tackling enterprise information management solutions for the life science industry. He is passionate about earning the trust and confidence of Daelight Solutions’ customers, while providing unparalleled technology strategy and delivery services. Prior to starting Daelight Solutions, Dan was director of consulting services at EMC and managing partner for Sitrof Technologies, Inc. He earned a bachelor’s degree in computer science from The College of New Jersey and an MBA from Rider University. He can be reached at dwheeler@daelightsolutions.com.