What is moving where? Where is the data going to be stored? Is this going to cause any problems or disruptions? Who is doing what during this whole process?
Planning for a data migration is a process that tends to require meticulous planning and clear communication to ensure everything goes smoothly. To reduce risk and save time, it’s critical to have a solid understanding of the data structures involved.
This blog explores key considerations during the planning stage of a data migration process. Before preparing the data and executing the actual migration, here are some important factors to ensure a smooth transition.
Let the planning begin!
When making key decisions, such as addressing the questions mentioned at the beginning of this article, the active involvement of the client—or whoever’s data you are working with—is crucial. They often possess the deepest knowledge of their existing systems, and their insights and expertise can significantly facilitate a smoother transition and effectively address potential challenges.
While it is critical for us as consultants to offer the most effective recommendations for our clients, the final decisions ultimately rest with them, especially when a significant amount of their own data is involved.
Setting the Scope
Before actually diving into the nitty-gritty of a data migration, it’s necessary to set a clear and detailed scope. Like we mentioned, this involves working closely with the client to define precisely what data, objects, and tables will be moved from one system to another. All this information will be compiled into a detailed Statement of Work (SOW), which the client must approve in writing. While other companies might use different documents, for simplicity, we’ll refer to the scope statement as the SOW in this article.
A data migration exercise isn’t just about moving data from one place to another; it’s also an opportunity to address any existing process or tech debt the client might have. Instead of merely shifting data to the new system, this is the perfect time to evaluate and improve the underlying systems and processes currently in play. From our experience, clients often take this opportunity to cleanse their data or update their data model to avoid carrying over old issues into the new system. Any requested improvements should be captured in the SOW or any kind of change order document if they are proposed after the initial review, decreasing the likelihood of misunderstandings down the road.
Sometimes, the data needs to be transformed before it can be imported into the new system. Core system architectures can differ, and there may be required fields in the target system that didn’t exist in the old one. Work closely with the system owners to determine the best approach for filling in these required fields. We’ll dive deeper into defining responsibilities and task ownership shortly!
When setting the scope, try to evaluate the potential value the client will gain from different levels of effort and resources. For instance, you’ll need to determine the cost-benefit ratio of migrating 100,000 records versus 1 million records—there’s a difference! There might be a point of diminishing returns where the effort outweighs the benefits. Why does the client want to move to a new system? Who is this benefiting and when will they start to reap these benefits? Knowing the real value helps clarify what data should be moved and keeps everyone motivated and on track during the process.
Understanding the relationships between different data objects is crucial, as it affects both deployment and the amount of data you ultimately end up moving. Building an Entity Relationship Diagram (ERD) can help visualize these relationships, making it easier for the client to understand how their data is interconnected. An ERD can also illustrate why certain data must be transferred, even if it initially seems outside the scope of the migration, and highlight the potential consequences of not moving this data.
Once you’ve established the scope of the migration, you can start making detailed decisions about how to implement it. With a clear scope, a smoother migration process is surely more likely to happen, which sets the stage for a successful project.
Who’s Doing What? Defining Responsibilities
A key step in getting ready for a data migration is clearly laying out everyone’s responsibilities. At Lane Four, we make sure to create a comprehensive document outlining what both our team and the client will be responsible for during the migration process.
For instance, our clients are often responsible for:
- Providing timely access to both the source system (where data currently lives) and [new] target system (where we’re transferring the data);
- Ensuring data is backed up and stored in the correct place;
- Describing any customizations or automations that we need to know about—helping us better understand the business logic behind the target system;
- Making decisions on required core data fields and any other information that needs to change before migrating;
- Conducting user acceptance testing (UAT) on the data we migrate to ensure everything works as expected
On our end, Lane Four takes on several key responsibilities to ensure a smooth migration, like:
- Managing the actual data extraction, transformation, and loading (ETL) process, ensuring data integrity and accuracy throughout;
- Handling the creation of any necessary tools and documentation to facilitate the migration and provide ongoing support to address any issues that arise;
- Planning and executing rollback procedures in case any issues arise during the migration, ensuring a fallback option is available;
- Overseeing the execution of the migration and any related cutover activities to switch from the old system to the new one seamlessly;
- Implementing layers of testing and confirmation beyond UAT to verify data accuracy and system functionality at multiple stages of the migration;
- Like we mentioned earlier, working closely with the client to review and refine the SOW, ensuring all details are captured and agreed upon
The document outlining all responsibilities can be included in the detailed scope statement or SOW contents, unless the client has already provided separate written approval for this. It’s imperative that both parties understand and sign-off on the responsibilities outlined.
Choosing the right tool(s)
Picking the correct tools for a data migration is essential for a successful process. However, there’s a chance you’ll need to carefully select appropriate third-party software tools on a project-by-project basis, tailoring your approach to the specific needs of each migration.
For smaller or simpler migrations, basic spreadsheet programs can suffice for sorting and storing the data. These tools, coupled with Data Loader and Import Wizard, provide a solid approach for handling simple data transfers. However, larger or more complex migrations may require the use of more robust third-party tools like MuleSoft. These powerful platforms offer ETL capabilities that are better equipped to manage large volumes of data and complex transformations.
There are also many other tools available that cater to projects falling somewhere between these two options, offering varying levels of functionality and providing flexibility to meet the specific demands of different migrations. It’s important to thoroughly evaluate the scale and complexity of the migration to determine the best tool for the job. Using the right tool not only simplifies the process but also helps ensure data integrity and accuracy throughout the migration.
Understanding the Target System
Before moving any data, you need to thoroughly understand the configuration of the client’s systems. This involves looking into everything from data formats to existing automation. Because the system is tailored to- and designed to mirror the client’s business processes, their input is invaluable for grasping the nuances of any automation in place. You’ll often rely on their insights to understand the specific automations and any customizations. This collaboration helps you navigate the intricacies of their system and ensures you can handle the data appropriately.
Depending on how the system is configured, it might be necessary to temporarily disable some automations during the migration. This precaution prevents the deployment from triggering unintended actions and errors. During what you wished would be a smooth and controlled migration process, you may find yourself swarmed with error email upon error email! Trust us, we’ve been there… So, we highly recommend that you take the time to fully understand the target system before starting the migration. This step not only helps prevent issues during the data transfer but also ensures that the new setup functions seamlessly with the existing business processes.
Designing a Test Plan
Be sure to prioritize devising a test strategy that accurately measures your level of success while minimizing disruptions to the client’s systems. Once you start to deploy to the target production system, you will need to verify that the data has arrived complete and intact. This includes making sure that all automation, data relationships, and reports remain unchanged. To achieve this, you may implement various tests to thoroughly evaluate your migration setup and address any failures promptly.
Here are some effective tests that we regularly include in our test plans at Lane Four. We recommend incorporating these tests into your test plan to thoroughly assess the success of your data migration. Flip the cards to learn more!
Sample Set
Sum Check
Spot Checks
Reports Test
Count Check
Integrated Systems Check
Getting ready for a data migration means planning carefully, communicating clearly, and really understanding your client’s systems. From setting a detailed scope to defining responsibilities and picking the right tools, each step is key before even starting to execute on moving data from one place to another. Want more confidence that your data migration will go smoothly? Let’s chat and get you on the right track!
Let's chat!
