Planning Data Migration the Right Way
What is data migration, really?
Let’s start with a couple basics. (I’ll be brief and get to the good stuff fast.) Data migration is the process of transferring information from one system to a new system. This can include transferring data between different file formats and different storage types. The term “data migration” is actually quite broad and covers a whole gamut of types of data, activities, and systems. Examples include:
- Migrating a file system or storage repository from on-premises to a company data center
- Migrating to the cloud
- Server upgrades
- Server consolidations
- Migrating to a document management system (DMS) or Enterprise Content Management (ECM)
- Implementing SharePoint (or increasing the use thereof)
- Implementing OpenText
- Transitioning from Novell Operating System to Windows
- Implementing Network Attached Storage (NAS) or a Storage Area Network (SAN)
- Remapping file paths into archives
- Folder restructuring
- Implementing Distributed File System (DFS).
What triggers a data migration?
• Routine upgrades of hardware and/or software.
• The legacy system does not have enough storage capacity.
• The business has been acquired by another company.
• The business or government agency has merged with another entity that uses different software and/or different file naming conventions.
• The organization splits into two or more organizations or units, each with a new name and requiring new file paths and filenames.
• Someone orders all data storage moved off premises.
The difficulty level of a data migration varies widely. The amount of data, the formats it’s stored in and the number and types of systems affected can all play a role.
Whether you’re transferring a single database or undertaking a large-scale data migration project, it’s not just “important” to be prepared. Serious problems including downtime, lost income, extra work, and upset end-users can result from failing to adequately plan (not to mention lost weekends). The purpose of this checklist is to help you avoid some of these perils.
Part 1: Planning Your Data Migration – First Steps
Before you begin transferring data, take some time to determine exactly what data to transfer and what systems it affects. (Okay, this is kinda obvious. But, hey, it wouldn’t be a checklist, if it didn’t start from the beginning. We’ll get to some lesser-known actions [and “gotchas”], before we are done.) Figure out what type of data you need to transfer, which systems use this data and what other files the data links to.
Note: That last phrase of that last sentence (“and what other files the data links to”), on first glance may appear to be just innocuous advice. But in the real world, this has actually been one of the “gotchas” that many IT pros have failed to account for, sometimes to their embarrassment. It’s easy to forget just how connected data is to other data. Users all across your organization are connecting (linking) data into their files all the time. They think nothing of it. And they don’t ever mention it to anyone in IT. If you change one thing, say a folder name for example, you could be affecting other data somewhere else your system in ways you hadn’t thought about yet. More on this will follow.
Have you checked your network for any obstacles to data transfer?
If you’re transferring data over a network, it’s important to have the most bandwidth you can. Check for firewalls that might delay data transfer. Consider transferring data when there are fewer users — or even better, no users — online. Schedule data migration for overnight if possible. If you must share a network during data migration, consider throttling your usage to prevent slowing down the network for other users.
Have you developed a map of what data needs to be migrated?
Before beginning your data migration project, develop a map of what data needs to be migrated. Your map should include where your files are and what they’re connected to. To maintain link integrity during data migration, include parent/child files in your map. This will help you make sure that once the transfer is complete, employees aren’t stuck dealing with broken links.
If you are not yet 100% familiar with what is meant by “parent file” and “child file”, click here to get a brief definition.
Bonus Tip: Print out your map before you begin migrating your data. Yes, it’s old-school, but it’s worth having a paper backup before you begin the process. Plus, sometimes you can get more of a broad view when you lay something out on paper on your desk. This can help you catch errors.
Have you done a landscape analysis to determine what systems the migration will affect?
Data migration usually focuses on the core and target systems. However, these two systems are rarely the only ones affected. Prevent broken links during migration by doing a landscape analysis. This will help you map out what other systems use the data you’ll be migrating, and help you see how the data migration could influence them.
This is where LinkFixer Advanced comes into play. It has a feature called “Inoculate” that automatically creates associations (sort of like a secondary means of linking) between each parent file and all its child files before they’re migrated. After migration, the “Cure” function acts as a batch re-link operation. A tool that automatically handles links during your data migration can save you thousands of hours.
Have you identified software necessary to complete the migration?
Once you’ve identified the systems, amount and type of data you’ll be migrating, you’ll need to choose software to help you move the data to the target system. There is a range of different data migration software available. However, it’s a good idea to pick something specific to the type of data you’ll be migrating.
Imagine the “bad” things
I wrote the following in an article in 2015: “Most of your failed data migration projects (heck, most failed IT projects of any kind) fail due to lack of imagination in the planning stage. Notice that I didn’t say ‘lack of planning’. (I wouldn’t insult you like that.) But there is an insufficient amount of imagining what the various bad outcomes of various actions might be, so that you can prevent the bad things from happening, or at least be ready when they do.” This is still true today. Take a moment and do this exercise. It could save you untold heartache.
Part 2: Figuring Out How to Transfer Data
Your next step in the planning process is to figure out how to transfer data. Test your hypotheses early and often during this stage. Corruption and loss of data are two of the biggest risks in any migration project. Your data transfer method should help to minimize data loss. Good test cases can also cut down on the need for time-consuming manual validation.
Have you accessed your data?
If you don’t already have the full access you need, get access to your data early in the data migration process. This will save you a lot of grief down the road. Check out individual files. See what types of security permissions they have. Will you need to change these permissions during the transfer? What types of security permissions will they need in the new databases? Are there are any quirks in how the database is used that you’ll need to account for? Actually looking at your data will let you test out these hypotheses.
Do you have a plan for validating the data transfer?
Validating the data transfer is a crucial step of a data migration; after all, corrupt or lost data is often a costly problem. Planning for data validation from the get-go is the way to go. There are three types of data validation possible:
- Validating random samples
- Validating subsets of data
- Validating the complete data set
In an ideal world, it would be possible to validate the complete data set. But in the real world… do you know any IT professionals with the time and resources to validate a complete data set? Consider validating random samples or subsets of the data to save time and sanity.
Do you have a recovery plan for data that didn’t transfer properly?
Despite your planning and testing, some of the data might not transfer properly. This can include minimal issues like using the wrong delimiter, or the core and target columns using different units of measurement. But it can also include major issues like data loss and corruption.
Two issues you should consider now include:
- Semantic data and file naming conventions: It’s common for departments to develop their own file naming conventions, or to use database fields in unusual ways. This usually isn’t a problem. But when you’re transferring data to a new system, it’s important to understand your users’ idioms. If you’re transferring data used by a particular department, consult with them now to make sure that you understand any unusual ways that they input information. Consider getting employees from these departments to visually validate samples of their data.
- Data loss and corruption: During the planning process, you should also consider what to do if data doesn’t transfer. This is a situation where prevention is the best cure. Will you be able to access a backup of any corrupted data? What can you do to prevent data corruption from occurring? Consider things like cleaning and deduplicating data before starting the transfer. You can also use tools like LinkFixer Advanced to inoculate (protect) links before the transfer.
Part 3: Beginning the Data Transfer: Preparing Data for a Migration
Whether it’s duplicates, bad links or outdated formats, most data has at least a few issues. Clean up your data before you transfer it. This will cut down on the amount of data you’ll need to transfer, as well as the amount that you’ll need to validate after it’s done.
Have you cleaned the data you’ll be transferring?
Cleaning data before migrating it is significantly easier than cleaning it after. It can also cut down on problems with corruption and data loss in the target system. Extract, clean and deduplicate data before you transfer it. Fix any broken links and file type transformations in this stage.
Have you checked to see whether any data should be archived instead of transferred?
Beginning a data migration process is a good time to review your data archiving procedures. You may not want to transfer all of your data. Some of it may be more suitable for archiving. This has the benefit of cutting down on work for you. Review or spot-check some of the data involved with the applicable department heads who use the particular files in question. If your organization has an Information Management (IM) department or group, you definitely should check with them as well. They will be very interested in what you are about to do. I can assure you of that.
Be careful not to overreact, though. Although it’s tempting to archive data instead of transferring it, be realistic about what types of data need to be accessed regularly.
Tip: Survey some of your key end users and your IM section to see what data they would be okay with archiving. They may save you a lot of time from having to restore something they regularly need. Conversely, they may also reveal some additional data that can be archived that you hadn’t considered.
Have you tested the new hardware?
One of the biggest mistakes in data migration is not testing the new hardware before you transfer data to it. Since most people transfer data from a smaller, outdated system to a newer, bigger system, they don’t bother to test the hardware. Unfortunately, this means that people complete a data migration, only to have the hardware break shortly after.
Don’t be one of those people. Test your new hardware thoroughly before transferring data onto it.
Part 4: Data Security Concerns
Protecting data is a growing concern for executives, financial officers, information managers, and public relations people. When you take on a data migration project, expect that many stakeholders will have concerns about security.
Addressing data security concerns early in the process can get your project critical support from these groups.
Have you identified relevant stakeholders from data security or privacy?
There aren’t many ways to stop a data migration project quicker than ignoring security. Find out now who’s concerned with data security and data privacy issues at your business, and figure out how to address their concerns.
Along with the IT department, consider the following groups:
- Human Resources departments
- Accounting departments
- Legal departments
- Chief Financial Officers (CFO)
- Chief Information Officers (CIO)
- Third party software contractors
- Information Management (IM) personnel
Identify their specific concerns and the data affected by these issues.
Have you vetted the software and personnel involved in the data migration?
Many of the data privacy risks that a company faces come from careless, uninformed or just plain disgruntled employees. That’s why it’s important to vet any personnel who will play a role in the migration.
This is also a good time to review who has what types of permissions to access confidential data. It’s far too common to have accounts that are either not in use (or belong to ex-employees) retain access to restricted files. Remove these permissions before beginning a data transfer to cut down on the risk of data breaches.
By the same token, it’s important to vet the software you use for a data transfer. Don’t just pick up any open-source file transfer software! Make sure the software you use is legitimate and well regarded.
Have you reviewed industry statutes to ensure that you’re conforming to best practices?
If you’re undertaking a data migration project in a specialized industry, you may have to adhere to specific requirements governing how data is protected during transfer. This is particularly true of healthcare and financial organizations. Check to make sure that any software and migration methods you use meet these standards. Industry best practices are also a good way to get management on board with the data migration.
Part 5: Getting Leaders to Buy-in to Your Data Migration
Getting business leaders to buy into a data migration project isn’t always easy. Neither executives nor users always understand the benefits of migrating data to a new format or system. More than a few IT professionals have found themselves struggling to get funding and personnel for data migration.
Have you set realistic deadlines and deliverables?
More than half of all data migration projects go over the deadline. Therefore, it’s important to be upfront about realistic deadlines for your project.
It can help to group the data by functional areas. Your data map can help you to identify these areas and schedule realistic deadlines. Be sure to allot more time for resource-intensive areas. If you’ve identified areas where there are variations in how the system is used, consider allotting extra time for testing and transferring data.
One way to increase buy-in from executives and users is to schedule a few wins early in the data migration project. When users can see the benefits of the project, they’re more likely to support it. If you’ve identified an area that could see immediate benefits, consider pushing it to the front of the data transfer project. Look for low-hanging fruit to use as your “convincer”. For example, if your organization has an engineering department that has frequently complained of slow processing speed, and you estimate that after the move to those two super-fast servers these engineers are going to see a major speed and performance increase on their Computer-Aided-Design (CAD) files, consider moving them first and using the results as your “show car”.
Do you have an adequate budget for this project?
We know this doesn’t always happen, but it’s good to know how much a data migration project will cost up front! It can help to get leadership buy-in if you align the data migration with other company goals, like increased productivity or cost cutting. To get people to support the needs of the data migration, focus on how the new data format will help the company. If you’ll be saving money by turning off an old database or server, let people know.
Do you have the right personnel for this project?
Think beyond the IT department when you consider personnel. Try to get subject experts who can help to confirm whether data has transferred over properly. It can also be helpful to find real users to test the migrated data. These real users are your best gauge of whether the new database or system is working properly. If you’re in a large office, consider delegating this to department heads.
Do you have company buy-in?
For many business owners or organization leaders, data migration isn’t the most exciting topic (though they will suddenly get real “excited” if something ends up missing or messed up). Particularly for longer migrations or more technical topics, it’s important to get prior buy-in from leaders. One of the best ways to do this is to focus on the most important outcome: They will lose access to data that isn’t migrated. (This should get their attention.) Making sure that they still have access to their data is often enough to get users on board.
To repeat, one way to get the leaders on board with migrating is to build-in some short-term wins up front in your project. These will let leaders see the benefits of the migration early on, and help keep them a bit more patient throughout the slower portions of the project.
Part 6: Planning for After the Deed is Done
Have you planned for decommissioning the old server or database?
Decommissioning the old hardware can get executives and users on board in two ways. First, phasing out the old program/hardware can be a cost-saver. Most executives are easy to get on board when there is a savings involved.
Second, the fact that the old hardware will be turned off means users will no longer have access to data on the old hardware. This can help users understand that they will need to access data in a new format or via a new program. This can also make it easier to get users on board with the process of transferring data.
Do you have a maintenance plan in place?
The work’s not done simply because the data’s been migrated. You’ll also need to make sure that the data is maintained regularly. Some people identify employees in the relevant departments to maintain data, while others charge someone within the IT department with the responsibility of maintaining it.
We’ve designed LinkFixer Advanced for both data migration and maintenance. You can even use it to repair links, in batch, that were broken during an earlier data transfer. Getting started with LinkFixer Advanced is easy. Download your free trial today!