Matillion has a Git integration for Matillion ETL with Git repository providers, which your company can use to leverage your development across teams and establish a more reliable environment.
In this blog, you will learn how to set up your Matillion ETL to be integrated with Azure DevOps and used as a Git repository for your developments.
What is Matillion ETL?
Matillion ETL is a platform designed to help you speed up your data pipeline development by connecting it to many different data sources. It enables teams to rapidly integrate and build sophisticated data transformations in a cloud environment with a very intuitive low-code/no-code GUI.
Why Does it Matter?
Connecting your Matillion ETL to Azure DevOps allows you to empower your solutions with all the best Git development practices. Connecting both allows your team to work on multiple versions at the same time and collaborate easily, making your developments more reliable and allowing you to work at a more granular level with agile methodologies.
Why use Azure DevOps?
Azure DevOps is an all-in-one development platform that simplifies the software development process. It offers tools for version control, build and release management, automated testing, project management, and more.Â
By using Azure DevOps, companies can improve productivity, collaboration, and efficiency while ensuring security and regulatory compliance.
How to Integrate Azure DevOps with Matillion ETL
To begin the integration, you must first confirm that you have the proper Azure DevOps permissions granted by your organization in order to allow your user to connect applications.Â
Suppose you are different from the Azure DevOps admin in your organization. In that case, you should contact the team managing your environment for more information on obtaining this permission.
Configuring Azure DevOps
Let’s say your company doesn’t have an Azure DevOps repository, and you need to create a brand new account to store the repository. After creating your new account, you will be asked to create a new project. That project will be where you will find many of the platform’s great features, such as the repository we will use.
Pick a name that you want for that project and confirm that you want to start it. Once you have created it, you will land on a page equal to or similar to this one:
As you can see on that page, you have an option on the left-hand side called Repos, which is your repository for that project. Unlike a few other platforms, Azure DevOps creates your repository automatically without needing any extra action.
Click on the Repos button to access information about your repository. Due to that being a new repository, your landing page will be like the one below:
In case you’re using a pre-existing repository, clicking on Repos will direct you to the repository. You will need to click on the Clone button at the top right corner of that page to see that information.
Now, from that initial Repos page, you will need to copy the URL of the first box, Clone to your computer, keeping the HTTPSÂ option enabled. That will be the URL Matillion will use to access that repository.
You will need to capture a few pieces of information to use later in Matillion ETL, so I recommend you open a notepad. Save that URL for the repository there.Â
The next step will be to click on the button Generate Git Credentials right below the URL. That option will create a user and password for you to use when connecting to the repository. Pick the credentials created by Azure DevOps and add them to your notepad.
When you create credentials in this way, Azure DevOps automatically creates the token with read and write permissions for codes in the repository. Suppose you manage the permission level for that token or even create another customized token. In that case, you must go to User Settings in the top right corner (close to the icon with your initials or profile image) and pick Personal access tokens.
Now that you have the repository and credentials created, it is time to go to Matillion ETL.
Configuring Matillion ETL
In Matillion ETL, you will start integrating it with the Azure DevOps repository you created. That connection will need to be made per project, so navigate to the project you want to connect it to.
On this project page, find the Git option by clicking on the Project button at the top left. This will open a new window where you can enable Git in your project by cloning or initiating a local repository. Init Local Repository should be selected.
The following window will perform the initial commit, creating your local repository. There is no need to change anything in the Commit information field, but you must repeat your email address in the field below in order to click the Confirm button.
That will create your local Git repository, which you will then push to your Azure DevOps repository. To start, we will need to navigate to the engine icon in the Git Integration window.
This will open a new window asking for the remote URI. That is where you will paste the repository URL you copied from the Repos space in Azure DevOps. Simply confirm it after you’ve pasted it.
Returning to the Git integration page, it is now time to log in to the remote repository using Matillion. This time, click the Lock icon to configure the credentials that will be used in the connection. Another window for connection configuration will appear.
When getting the repository URL, use the username and password you generated in Azure DevOps. The encryption type should be changed from KMSÂ to Encoded, and then you click OKÂ to confirm.
Your Git integration is now complete. Let’s use it for the first time to push your initial commit to the remote repository in Azure DevOps to confirm it.Â
To do so, click the arrow up icon in the bottom right corner, which will open a new window asking what type of push you want to perform. Select Thin Push and press the OK button.
You should see a new window that says Branch master: OK, indicating that your integration was successful because you were able to push your commits to the remote repository.Â
If you do not receive this message, please review the previous steps to ensure everything was done correctly; otherwise, you should confirm that you have all the permissions required to access Git in Matillion and Azure DevOps to use tokens with other applications.
Closing
This blog shows how to connect your Azure DevOps to Matillion ETL to build your Git repository. This gives you a more reliable development environment, ensuring that your promotion steps from development to production are more efficient, secure, and organized.
Looking for more Matillion assistance?
phData excels at assisting organizations in achieving success with Matillion. Contact us today for help, advice, and actionable strategies.
