ADA helped a global retailer create a modern unified data platform on Databricks to reduce operational costs and streamline data workflows

The customer is a leading global sporting goods retailer with a footprint in 56 countries and selling products from over 20 brands.

The customer built their existing data platform on disparate cloud services (Amazon Redshift for data storage, Python/SQL for data processing, and Jenkins for workflow automation). The existing platform introduced many integration challenges that led to increased costs and reduced operational efficiencies.

ADA helped design, implement, and migrate their existing data to a new unified data platform on Databricks. We successfully resolved many technical challenges including handling and migrating diverse data sources, overcoming limited support for converting Python code to PySpark using innovative solutions, and ensuring data alignment between the old platform and the new Databricks platform.

The project was executed seamlessly, preserving data integrity and security while avoiding disruptions to ongoing operations.

Our goal is to establish a gold layer within the Databricks system, utilizing data from the silver layer for business intelligence reporting. Our approach to achieve this involves:

• Data Source Analysis: Understand current data sources and associated codes for a clear migration foundation.
• Code Transformation with Optimization and Quality Checks: Convert Python/SQL to PySpark/Spark SQL, adding data quality checks for reliable data integrity and optimizing it for space and speed.
• Data Validation and Comparison: Verify Databricks data against Redshift data, ensuring seamless alignment.
• Data Orchestration: Schedule jobs using Databricks workflows, implementing monitoring and alerts for a reliable workflow.
• Documentation and Knowledge Transfer: Document the migration process for reference and conduct knowledge transfer sessions for enhanced team adoption.