Renowned Global Developer of Automation Testing & Measurement Systems Migrates to Snowflake from Exadata and Qubole
Customer's Challenge
A renowned global producer of automated test equipment & virtual instrumentation software was on a tight deadline to migrate a significant number of data flows off its legacy technology stacks (Exadata and Qubole) onto Snowflake.
phData's Solution
phData leveraged Bladebridge, Fivetran High-Volume Replicator (formerly LDP), and the phData Toolkit to accelerate the migration and allow the client to meet its immediate deadline of getting off Exadata. Additionally, the adoption of dbt and HVR also laid the foundation for Phase 2 of the migration from Qubole to Snowflake.
Results
The phData team successfully migrated the client’s legacy Informatica data pipelines from Exadata to Snowflake. This included migrating the ingestion processing for 12 data sources, that were using a combination of Informatica PowerExchange and custom ingestion routines, to Fivetran High-Volume Replicator (HVR).
Data transformation processing was migrated from a combination of Informatica PowerCenter and custom Oracle stored procedures to dbt.
In the end, over 3,000 dbt models were created to manage data acquisition, cleansing, and curated transformations.
The Full Story
A world-renowned provider of software-defined platforms, automated test equipment, and virtual instrumentation products for engineers and scientists was facing a combination of end-of-life issues and performance problems with a number of its legacy platforms and technologies.
As an innovative organization, the client realized it needed to leverage modern technologies like the Snowflake Data Cloud to continue delivering cutting-edge solutions for its customers. The first step in the journey to data modernization was to move data off its legacy Exadata platform and onto Snowflake. Â
As they adopted Snowflake as their future data platform, they also explored modern technologies for data replication and transformation to streamline the migration and optimize subsequent processes.
There were just two factors that complicated the client’s path forward.
First, they needed to complete the migration off of Exadata prior to their next contract renewal, which resulted in a very tight timeframe.Â
Second, they lacked both the resources and technical expertise to complete this transformation in an accelerated timeframe. Â
Why phData?
The client found phData’s global presence and deep investment in automation via the phData Toolkit to be key differentiators that would help accelerate and de-risk the SQL Translation and data validation requirements for the migration as a whole.
Additionally, phData’s proven migration factory methodology and experience with best-of-breed SaaS technologies from an ELT perspective (including Fivetran/HVR and dbt Cloud) helped secure their choice as phData.Â
Establishing an Information Architecture
phData first established an Information Architecture to provide a consistent basis for building the necessary Snowflake assets required for the migration. For the migration itself, the biggest challenge was transforming over 600 PowerCenter mappings to dbt models.Â
The phData team adopted the Bladebridge conversion tool to perform the initial conversion operation before reviewing, revising, and validating the resulting models. The team also made heavy use of the phData Toolkit to convert legacy view definitions into dbt models.Â
Finally, the Data Source tool within the phData Toolkit was also used to help automate the validation process.
Take the next step
with phData.
Learn how phData can help solve your most challenging data analytics and machine learning problems.
