When Does It Make Sense to Migrate?

Migrating your infrastructure is an expensive and time-consuming operation, and ideally, you would never have to do it. Unfortunately in the real world, as a company grows and new technologies come onto the market, it is likely that the original infrastructure no longer best fulfils your product’s needs.

In this article, we discuss examples of companies that migrated and how they came to their decisions.

After losing ~$3 million following severe on-site database corruption, Netflix realised it urgently needed to move to a more reliable and secure data storage system¹². Choosing when to migrate (and what to migrate to) can be a million-dollar decision. Part of this decision involves understanding the options available: public clouds are managed by third-party providers and shared across organisations; private clouds are dedicated to a single customer³; and hybrid models combine private and public clouds, allowing data and applications to be shared between them.

From On-Premise to Cloud: Spotify

A decade after its inception, Spotify decided to switch from on-premise to a cloud infrastructure. Over the years, the Swedish music streaming service grew rapidly across the globe. Despite its success, the company’s growth raised internal concerns regarding the substantial investments required to build out their data centre infrastructure⁴. It was challenging and time-consuming to provision and maintain the enormous in-house data centres required for their services⁵. To manage these systems, around 100 teams were assembled globally. Fortunately, Spotify took a step back and realised that these operational efforts weren’t directly contributing to the company’s vision: “being the best music service in the world”⁶. After all, Spotify is fundamentally in the music business, not in building data centres⁴.

After a year of extensive planning and research, Spotify smoothly migrated around 2,000 services, 20,000 daily pipeline runs, and over 100 Petabytes of data to the cloud⁷. This migration was supported by the teams at Google Cloud Platform (GCP)⁴. For a customer as large and complex as Spotify, several cloud platform features had to be developed by GCP engineers to accommodate their specific needs. As a result, countless operational complexities were removed from their ecosystem, allowing Spotify to better focus on innovation and customer needs⁸.

From Cloud to Hybrid: Dropbox

Dropbox chose to move away from solely using a public cloud in favour of a hybrid model. Founded on Amazon Web Services (AWS), they decided 8 years later to build a new storage system from scratch. By building their own infrastructure, Dropbox gained full control over the underlying tech stack, enabling features such as ‘Project Infinite’, providing desktop users with “universal compatibility and unlimited real-time data access”⁵.

Taking a hybrid approach meant Dropbox could cheaply store infrequently accessed data (such as analytics) in AWS⁹, while keeping their core storage on their own infrastructure. They also chose to only migrate their U.S. data, since AWS remained cost-effective for them in Europe and Asia. Dropbox used quantitative methods to guide the decision: looking at growth forecasts, anticipating capacity and costs, and building a model for data centre capacity planning¹⁰. Dropbox migrated about 90% of their ~600 Petabytes of customer data in less than a year, a move that saved them $75 million in operating expenses¹¹.

Conclusion

Hybrid models are rapidly growing in popularity as they offer flexibility for companies to choose which services to keep on-premise. There are genuine benefits to both approaches. Public cloud lets companies avoid reinventing the wheel and focus their engineering effort on the product itself. On-premise is preferable when you need full control over data storage, or performance capabilities that cloud providers can’t match at an affordable cost.

Deciding when to migrate (if at all) comes down to forecasting, planning, and understanding what the product actually needs. Spotify’s approach was more qualitative: they noticed too much engineering effort was going towards maintaining infrastructure rather than building the product. Dropbox’s was more quantitative: they modelled growth and capacity, then planned accordingly. Both case studies also illustrate the importance of company goals. Spotify is a music platform and wants to stay focused on that; Dropbox is a cloud data storage company, where owning your own infrastructure becomes a strategic asset.

Written by me together with two of my Imperial College London classmates, Guji and Amelia