Data warehouse modernization: Teradata to Snowflake migration
We successfully executed a complex migration project for a top US grocery chain, transitioning them from an aging Teradata Enterprise data warehouse to a modern Snowflake-based cloud solution hosted on Azure. This endeavor involved managing a vast volume of data, numerous tables, and the conversion of legacy code. Our solution encompassed meticulous data analysis, schema design, data transformation, ETL development, automated script generation, and performance optimization.
About the Client
Our client is a prominent player in the US grocery industry, ranking among the top 5 grocery chains. With an extensive network of over 1500 stores, they have annual revenues exceeding $60 billion. They have 30+ million active customers and manage more 100,000+ items. The size of their data warehouse is 180TB, with an annual growth rate of 10TB, encompassing both customer and product information.
Client’s need and Problem statement
Our client needed to modernize their outdated Teradata Enterprise data warehouse by migrating it to a cloud-based Snowflake solution hosted on Azure. The client sought a solution that would improve enhance performance, reduce costs, and enable data-driven decision-making while ensuring scalability and data quality. They were looking for an engineering partner with strong data engineering capabilities in Snowflake who could support a multi-year platform migration program.
Tech Stack
Python
SQL
SnowFlake
Database
Kafka
DataLake
Our solution and outcomes
- We partnered with the client’s technology and engineering teams to migrate all of their data from Teradata to Snowflake. The project involved 10K+ man-house of development spanning 24 months.
- Our client realized 30% cost savings compared to Teradata by adopting Snowflake pay-as-you-go model. We were able to leverage Snowflake’s elasticity to efficiently scale up or down. Dedicated warehouses were assigned for critical batch workload increasing query performance by 40% in some cases and helping meet critical SLAs.
- The project involved re-designing their database schemas, ETL development, automated script generation to expedite data migration and rigorous testing and validation of the migrated data
