The winter solstice is finally upon us, and we’ve hit the magical time of year when everybody is spending time with their families and friends. For our Minneapolis friends, we managed to set a record high temperature on Christmas Eve (54F!). We hope at least some of you had a “white Christmas” full of snow angels, sledding, and making snowmen.
Here at phData, our engineers have been hard at work to bring new Christmas gifts in the form of new features and bug fixes! Let’s take a look at the latest and greatest.
Data Source Tool
Our toolkit is made up of a number of different tools for your Snowflake journey. The data source tool was specifically designed to automate data platform processes at scale with ease. This tool provides functionality in a number of different ways based on its metadata and profiling capabilities. This includes things such as:
Platform migration validation
Platform migration automation
Metadata collection and visualization
Tracking platform changes over time
Data profiling and quality at scale
Data pipeline generation and automation
dbt project generation
By programmatically generating metadata about your sources, targets, and projects, you can leverage that metadata to programmatically accomplish tasks. While this may seem like a trivial thing in concept, it’s actually incredibly powerful.
Imagine you wanted to build a dbt project for your existing source data warehouse in your migration to Snowflake. You could leverage the data source tool to profile your source, apply a template against the generated metadata, and automatically create a dbt project with models for each table! This serves as a great starting point for your overall dbt project.
Over the course of the last month, we’ve introduced some quality-of-life features and fixes. One of the coolest features we’ve introduced is the ability for the data source tool to generate an Entity Relationship Diagram (ERD) from a scan of your data source.
This allows you to better understand the existing structures that are in place and more accurately perform your migration (or generate documentation, everybody’s favorite!).
The data source tool can also directly generate the Data Definition Language (DDL) for these tables as well if you decide not to use dbt!
One of the major fixes we’ve brought to the platform is removing special characters in the output file path generated when running a command. In situations where the commands have outputs, we’ll save this metadata to a directory. In some situations, the tool would end up putting a colon in the file path, which is not supported by Windows.
Advisor Tool Updates
phData, Partner of the Year with Snowflake 3 years in a row, has spent a lot of time guiding our customers through platform migrations and best practices, and we have experienced a number of different approaches to running a data platform on Snowflake.
As a result of all this experience, we have established a set of best practices and recommendations when it comes to running Snowflake. This led to us building the Advisor Tool to programmatically and automatically make recommendations to our customers based on the Snowflake setup.
With the Advisor tool, you can quickly and easily identify opportunities to improve the configuration, security, performance, and efficiency of your Snowflake environment.
One of the coolest things about the phData Advisor Tool is that it’s available on Snowflake as a native application! Check out our docs on that here.
Over the course of the last month, we’ve added a few additional checks to the Advisor Tool based on some of our customer’s usage of the tool:
Added a check to calculate data transfer amounts
Added checks that list the queries run by admin roles
ACCOUNTADMIN,SYSADMIN,USERADMIN, andSECURITYADMINAdded a check showing total storage usage ‘Database storage usage rule’
We’ve also increased the performance of the Advisor Tool as well. The tool now runs on 8 threads as opposed to the original single thread!
We highly recommend that you use the phData Advisor Tool within your Snowflake environment. It’s free to use and a single click on the Snowflake Marketplace!
Wrapping Up
As we embrace the warmth of the winter solstice and the joyous festivities with our loved ones, the phData team has been unwavering in its commitment to delivering valuable gifts to enhance your Snowflake journey.
Our engineers have diligently crafted updates and features for both the Data Source Tool and the Advisor Tool, providing you with a more streamlined and efficient experience.
Whether you’re embarking on a Snowflake migration or optimizing an existing setup, we strongly recommend integrating the phData Toolkit into your environment. It’s not just a tool but a guide to best practices, efficiency, and security—all wrapped up with the season’s spirit.
Don’t miss out—click once and explore the possibilities on the Snowflake Marketplace. Happy holidays!
phData Toolkit
If you haven’t already explored the phData Toolkit, we highly recommend checking it out!
Please spend a few minutes browsing the apps and tools available in the phData Toolkit today to set yourself up for success in 2023.
Be sure to follow this series for more updates on the phData Toolkit tools and features.
