Matillion has a GIT integration for Matillion ETL with GIT repository providers, which your company can use to leverage your development across teams and establish a more reliable environment.
In this blog, you will learn how to set up your Matillion ETL to be integrated with GIT and used as a GIT repository for your development.
What is Matillion ETL?
Matillion ETL is a platform designed to help you speed up your data pipeline development by connecting it to many different data sources, enabling teams to rapidly integrate and build sophisticated data transformations in a cloud environment with a very intuitive low-code/no-code GUI.
Why Does it Matter?
Connecting your Matillion ETL to GIT allows you to empower your solutions with all of the best GIT development practices. Connecting both allows your team to work in multiple versions simultaneously and collaborate easily, making your developments more reliable and allowing you to work at a more granular level with agile methodologies.
GIT Integration Basics
Before integrating GitHub with Matillion ETL, a few concepts are essential to establish.
- Matillion ETL’s GIT Support allows storage of versionable items. These include Orchestration Jobs, Transformation Jobs, job variables, and grid variables.
- Matillion ETL’s GIT Support does not support environmental items, which include Matillion Environments, Environment Variables, and Matillion Project Configuration features. Additionally, the GIT repository will not save API Query and Extract profiles.
Matillion Project Hierarchy
Understanding how Matillion ETL integrates with GIT, it’s helpful to know how the Matillion Project structure works.
- Project – This is the top-level structure. Each project is isolated, and users receive granted access on a project-to-project basis.
- Version – Within projects are versions, which are independent working areas. Each version points to a GIT branch in the local repository.
- Commit – A point-in-time copy of a Matillion ETL version. This version contains orchestration and transformation jobs, their settings, and orders.
- Branch – A collection of one or more commits in GIT. A GIT project typically has a master branch with other branches created to perform development or testing.
Why Use GitHub?
GitHub is an online software development platform. This platform stores, tracks and collaborates on software projects. It is a free service used by millions of developers around the world. It utilizes GIT, the software development’s most widely used version control system. Additionally, GitHub integrates well with many platforms, including Matillion ETL.
How to Integrate Github With Matillion ETL?
To begin the integration, you must first confirm that you have the proper GitHub permissions granted by your organization to allow your user to connect applications.
If you are not the GitHub admin in your organization, you should contact the team managing your environment for more information on obtaining this permission.
Configuring GitHub
Let’s say your company doesn’t have a GitHub repository, and you need to create a brand new account to store the storage. After creating your new account, you will be directed to a page like the one below.
Click the “New” button in the top left corner to establish a new repository on that page. You’ll see a page similar to the one below. Here, you can configure all the desired options for your repo, initialize it with a README file, and create the repository.
After creating your repository, you must clone your remote repository to the local one. You’ll need a URL to link the local storage you created in Matillion to the remote repository you created in Matillion. To retrieve this, hit “Code,” You can copy an HTTPS URL or use an SSH private key.
For this example, we’ll use HTTPS.
Finally, we must create a personal access token allowing Matillion to push changes to our remote repository. To do this, we navigate to the profile icon on the top right of GitHub, then navigate to “Settings”
From there, we then navigate to “Developer Settings” at the bottom of the left-hand menu.
Then we navigate to “Personal Access Tokens” and select “Generate New Token” to create our token. For our example, we’ll use the classic token.
From here, we can create a name for the token, an expiration period, and the desired scope of the token. A hash-valued token will then be generated. It’s crucial that this is copied and saved somewhere you can find it, as you will need the token to complete the Matillion integration, and it will not be displayed again. If you lose it, you’ll then have to regenerate the token.
Configuring Matillion ETL
It is now time to go to Matillion ETL and begin integrating it with the GitHub repository that you have created. The connection you created must be made per project, so navigate to the project to which you want to connect.
Find the “GIT” option on this project page by clicking the “Project” button at the top left. This step will open a new window where you can enable GIT in your project by cloning or initiating a local repository. “Init Local Repository” should be selected.
The next window will perform the initial commit, creating your local repository. There is no need to change anything in the “Commit information” field, but you must repeat your email address in the field below to click the “Confirm” button.
That will create your local GIT repository, which you will then push to your GitHub repository. Navigate to the “engine” icon in the “GIT Integration” window.
This next step will open a new window asking for the remote URI. That is where you will paste the repository URL you copied from GitHub. Simply confirm it after you’ve pasted it.
Returning to the GIT integration page, it is time to log in to the remote repository using Matillion. This time, click the “Lock” icon to configure the credentials used in the connection. Another window for connection configuration will appear.
Use the username and personal access token you created earlier in the GitHub steps. The username is the one you created when you created your account, and the token is the personal access token you generated and kept handy. The encryption type should be changed from “KMS” to “Encoded,” and then you click “Ok” to confirm.
Your GIT integration is now complete. Let’s use it for the first time to push your initial commit to the remote repository in Github to confirm it. To do so, click the “arrow up” icon in the bottom right corner, which will open a new window asking what type of push you want to perform.
Select “Thin Push” and press the “OK” button.
You should see a new window that says “Branch master: OK,” indicating that your integration was successful because you could push your commits to the remote repository. If you do not receive this message, please review the previous steps to ensure everything was done correctly. Otherwise, you must confirm that you have all the permissions required to access GIT in Matillion and Github to use “apps” with other applications.
Closing
With this blog, we could see how to connect your GitHub to Matillion ETL to build your GIT repository. This gives you a more reliable development environment, ensuring that your promotion steps from development to production are more efficient, secure, and organized.
Are you looking for more Matillion assistance? phData excels at assisting organizations in achieving success with Matillion.
Contact us today for help, advice, and actionable strategies.
