Customer's Challenge
A global, Fortune 500 insurance company with nearly $77 billion in assets needed to modernize their on-prem architecture — both for cost savings and as a move toward digital transformation. With the clock ticking before their licenses were up for renewal, the client needed a partner who would find a viable alternative for their current setup (with Hadoop and Spark) and be able to meet their data lake needs.
phData's Solution
We stepped in with both the solution expertise and established the process for moving to the cloud quickly. After evaluating the existing infrastructure, we leveraged Terraform and our standardized CI/CD process to deploy and configure AWS EMR and the client’s existing Spark application. Finally, we helped the client navigate their new cloud environment with specific recommendations for cost and process optimization.
Results
Here’s the upshot: the client can now make an informed migration plan that will save them over a million dollars per year in licensing costs and seamlessly handle the existing processes to maintain their data lake.
While the move to the cloud was an eventuality for such a large company, phData helped our client get there faster and more efficiently with our proof of concept project. By distilling dozens of complications and dependencies in an action plan, our expert team helped the client fully understand how to move to the cloud.Â
Now, the client is set up for long-term cost savings and more accessible data for everyone from actuaries to the C-suite.Â
The Full Story
Long story, short: you can’t take your Hadoop, on-prem environment and simply move it to the cloud with the expectation that you’ll immediately save time and money.Â
If you don’t configure everything correctly for a cloud environment, you can often end up spending more money. From the beginning, it was clear that we’d need to come alongside as a partner, instead of simply a vendor in a lift and shift project.Â
In addition to our core services for moving to the cloud, we helped the client answer questions like:Â
- How do we provision in the cloud​
- How do we set up cloud security​
- How do we take advantage of the cloud-native features ​
Why phData?
phData had already been working with the client for a while, helping them manage their existing Hadoop cluster. When the client decided that it was time to move off Cloudera and into an AWS environment, they knew where to turn.
With the wellspring of trust already in place, and our expertise in Spark, AWS EMR, and AWS S3, they realized that working with phData was the fastest and most promising path forward into the cloud.Â
Helping the client get past lift and shift​
While this seemed like a straightforward cloud migration at first, the client faced a handful of interrelated problems that we would help them solve over the course of the project.Â
First, the client had a clear motivation for moving to the cloud: with the on-prem set up, they just couldn’t get the Hadoop environment to work well for their actuaries, let alone in a cost-efficient way. The main question became: how do we get this Spark application, which originally ran in Cloudera, to run efficiently in AWS EMR?Â
But the client had a secondary problem, too: the application hadn’t been set up in Spark optimally to utilize all resources. (This is just one example of how the engineering team uncovered previously unrecognized issues in their architecture.)Â
Finally, we identified opportunities to take a more cloud-native approach, allowing the client to leverage the scalability of the cloud while minimizing costs. With the complex enterprise architecture, our team quickly realized that we couldn’t simply move the client to the cloud and expect a positive outcome. After testing and validating the new solution, we documented a path forward by offering advice on cost and performance optimization.Â
Our expert team came together to address all three interrelated issues, setting the client up for long-term success with its new cloud environment.Â
Driving a solution that will work for everybody
Since the client placed an emphasis on cost savings, we recommended using AWS EMR — it will work as an on-demand resource for the client instead of an always-on platform.Â
Based on our previous work with the client, we already had Talend set up for orchestration. (Originally, it was set up to let any dev team replicate data in the cloud.) With Talend already in place, we recommended using its integration with EMR to allow for a better pricing strategy.
From there, the majority of the project focused on getting Terraform and CI/CD set up correctly to work with AWS. Our team defined the cluster and the framework to go with it in Cloud Foundation, ensuring the right configurations and effective deployment in the cluster. Finally, our team set up Livy for managing security access.Â
With dozens of ‘consumers’ for the cloud data—projects and products—we needed to set up the architecture in a way that would allow for simultaneous pulls and storage in multiple S3 buckets. This is what the architecture looks like now:Â
Take the next step
with phData.
Learn how phData can help solve your most challenging cloud infrastructure problems.
