The biggest cloud database migration challenges for the year ahead

Businesses are facing a turbulent economic climate, and the threat of recession looms over the year ahead - so the argument for migrating data to the cloud in terms of both costs and efficiency has never been stronger. With the economy becoming ever more digital, more businesses are taking the decision to migrate their on-premise or self-managed warehouse to the cloud. But this is not always an easy journey. Many businesses fail to migrate effectively, or end up going wildly over budget. This is why planning and prepping strategically is more important than ever to any cloud database migration.

Even before the pandemic, research showed that up to 90% of CIOs reported data migration projects falling short of their objectives, and just a quarter (26%) achieved data migration within their expected timeframe. According to research by Gartner, the cost is an ongoing problem, with 60% of infrastructure and operations leaders facing public cloud cost overruns. There are common issues which can derail cloud database migration projects, taking a more holistic approach to technology and skills can help business leaders avoid these pitfalls.

The skills gap

Perhaps the most common issue which can hinder projects is not having the right skillset: Deloitte identifies a skills gap as the number one problem in cloud migration. Many existing IT people simply don’t have the necessary cloud-specific knowledge and experience. People make the mistake of assuming, to take one example, that if they have a Microsoft SQL Server database administrator (DBA) that they’ll be able to instantly transfer their knowledge and skills to a cloud data warehouse.

Not so. The two are as different as driving a car and driving a motorcycle. Cloud migration projects tend to be extremely complex, so it is usually wise to have a dedicated project manager working alongside the DBA to handle everything from risk management to communication to ensure the project adheres to its timeline. If you’ve never migrated a database to the cloud before, it is worthwhile to hire in expertise in the form of an outside consulting firm or individual to offer guidance.

Failing to plan

Poor planning is a key point where cloud database migrations can stumble or fall. By their nature, cloud database migrations tend to have lots of moving parts. This is why going over budget, ‘mission creep’ and missed deadlines are so common with cloud database migrations.

Having a clear plan before starting ensures it is distributed to every stakeholder who might have an interest and can jump on to assist at any point in time. A simple and single channel of communication such as an internal Slack channel for project updates helps keep everyone in sync.

Key to this plan is deciding which sort of cloud migration is right for your company. There is no one-size-fits-all solution. Different strategies have their own upsides and downsides. So-called ‘Big Bang’ database migrations, where there’s a hard cut-off and one system is

switched off mean that if something goes wrong, it can be extremely difficult to roll back the migration.

Piecemeal migrations, where the overall project is broken down into smaller parts which can be migrated individually one by one, remove this risk, but can add a new layer of complexity. With some systems using the old database and others using the new cloud database, it can leave businesses operating without all their data in one place. Migrating bit by bit does have benefits though, allowing business leaders to react to problems as they come up without too much damage to the business as a whole. Taking it slow helps to avoid serious problems which might stop your business in its tracks.

Parallel live migrations, also known as zero-downtime migrations, are possibly the best choice, where both old and new databases are live at the same time with production data. This offers peace of mind when something goes wrong, which it almost certainly will. Parallel live migrations provide an easy, reliable way to roll back if needed, and an ongoing validation that the data in both systems matches.

It also pays to think about your business goals, and how these might change in the future. You need to choose a data integration solution that lasts far beyond one single project and is able to grow as your business does. Again, communication is vital here: get input from stakeholders, from the CTO, CIO and CEO to understand what might be coming over the horizon. You’ll need to understand what challenges your business might face, and how the data operation will respond to these, plus how it will scale with the business. Thinking about this at an early stage can help you choose the right solution to deliver long-term value.

Adopting the wrong technology

Choosing the wrong technology can hurt your cloud database migration project. The so-called ‘lift and shift’ approach, where applications and data are moved to the cloud without redesigning the app, can be tempting due to the speed it offers, but can lead to problems down the line.

Choosing a more streamlined, automated approach can help business leaders deal with challenges such as synchronisation, data integration and potential rollbacks in the fairly likely event that there are problems. For example, change data capture automatically syncs a database with a cloud data warehouse, offering smoother flows and increased reliability. You’ll also need to think carefully about data integration technology: ETL (extract, transform, and load) takes raw data from sources, then transforms that data on a secondary processing server, before loading the data. The newer ELT (extract, load, and transform) provides flexibility and control by loading the raw data directly into a data warehouse, where it is transformed into enriched and structured data. Choosing ELT tends to speed up the data migration process measurably, and allows you to work with complete data. That said, ETL can have its place, particularly in systems with legacy architectures.

It’s also worth considering solutions which offer more automation. Many data projects get bogged down with technical details such as developing data pipelines for individual database tables. Such tasks can mean that your team’s time and resources are consumed by technical grunt work, leading to delays and increased costs. Choose the right tools and you’ll get to the finish line on time, and start reaping the benefits.

Reaping the benefits

Setting out a plan, which includes what technology to use, your timeline and how your business will continue operating during the process, will help you avoid delays which will hit your bottom line. Thinking intelligently ahead of time and ensuring you have both the right staff, the right knowledge, and most importantly, the right technological tools will ensure you avoid a rushed migration which may result in unforeseen disruptions to the entire data migration journey. The benefits of cloud database migration are clear and measurable. Planning properly ensures you’ll arrive there on time and on budget.