Data Engineering Role
We are seeking a highly skilled Data Engineer to join our team.
This role requires a strong understanding of data engineering concepts, experience with Azure Databricks, Azure Data Factory, SQL Server and Python.
The ideal candidate will have 5+ years of hands-on expertise in these technologies and a proven track record of designing, implementing and optimizing large-scale data pipelines.
The successful candidate will be responsible for:
* Developing data pipelines using Azure Databricks and Azure Data Factory
* Creating data flows and transformations for seamless data extraction, transformation and loading
* Developing Databricks Python notebooks for tasks such as joining, filtering and pre-aggregation
* Optimizing database performance through SQL query tuning, index optimization and code improvements
* Maintaining and enhancing SSIS package design and deployment for legacy workloads and contributing to migration and modernization into cloud-native pipelines
* Collaborating with cross-functional teams using Git for version control and Azure DevOps pipelines for deployment
* Partnering with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking and role-based security
* Implementing REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability
* Automating ETL processes and database maintenance tasks using SQL Agent Jobs, ensuring data integrity and operational reliability