Marketing Company Migrates Hadoop to Snowflake With Snowpark

A major marketing company was looking to modernize its existing data products from Hadoop to Snowflake. At the time, the client was using either an on-premise Hadoop cluster or an AWS-hosted one. Multiple technologies were used to power this data product including HBase, Hive, and Map-Reduce. 

The existing process had multiple ways to run, with the normal operation being a batch process run via MapReduce. But there was also a streaming process that would run micro-batches. Finally, the scaling of this project had to run from a few thousand records on the low end, to over 100 million records on the high end. 

phData stepped in based on recommendations from Snowflake and reviewed their existing Java code base and architecture. phData recommended a new architecture using Snowpark as the foundation for their data application in Snowflake. 

Snowpark would allow them to reuse their existing in-house Java libraries and classes, which would significantly reduce the time to market. phData was one of the first companies outside of Snowflake to start working with Snowpark, and the existing experience helped to guide the recommendation and the development.

phData transformed the code from the Map-Reduce code and created a Snowpark Stored Procedure that can be called for any volume of data. This Store Procedure is automatically tested and deployed via automation. This set up the client to have a template for future work that follows current best practices. 

Because the Snowpark Stored Procedure could be called by a task, phData was able to set up a Snowflake stream and task to deal with not only the client’s batch data needs but also the streaming/micro-batch needs with the same deployment/code base. This significantly streamlined and standardized the processes.