Job Title: Data Engineer
We are seeking a highly skilled professional to lead the design, implementation and optimization of enterprise-grade data pipelines. This role will leverage Azure Databricks, Azure Data Factory, SQL Server, and Python to enable scalable, governed, and performant data solutions.
Key Responsibilities:
* Design, build and optimize ETL/ELT pipelines using Azure Databricks (PySpark, Delta Lake) and Azure Data Factory (ADF)
* Develop pipelines, data flows and complex transformations with ADF, PySpark and T-SQL for seamless data extraction, transformation and loading
* Develop Databricks Python notebooks for tasks such as joining, filtering and pre-aggregation
* Optimize database performance through SQL query tuning, index optimization and code improvements to ensure efficient data retrieval and manipulation
* Maintain and enhance SSIS package design and deployment for legacy workloads; contribute to migration and modernization into cloud-native pipelines
* Work with cross-functional teams using Git for version control and Azure DevOps pipelines for deployment
* Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking and role-based security
* Implement REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability
* Automate ETL processes and database maintenance tasks using SQL Agent Jobs, ensuring data integrity and operational reliability
* Craft and optimize complex T-SQL queries to support efficient data processing and analytical workloads
Required Skills & Qualifications:
* 5+ years of hands-on expertise with Azure Databricks, Python, PySpark and Delta Lake
* 5+ years of proven experience with Azure Data Factory for orchestrating and monitoring pipelines
* Strong SQL Server / T-SQL experience with a focus on query optimization, indexing strategies and coding best practices
* Demonstrated experience in SSIS package design, deployment and performance tuning
* Hands-on knowledge of Unity Catalog for governance
* Experience with Git and CI/CD practices in data engineering projects
Nice to Have:
* Exposure to Change Data Capture, Change Data Feed and Temporal Tables
* Experience with Microsoft Purview, Power BI and Azure-native integrations
* Familiarity with Profisee Master Data Management (MDM)
* Working in Agile/Scrum environments
Preferred Qualifications:
* Microsoft Certified: Azure Data Engineer Associate (DP-203)
* Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification
* Databricks Certified Data Engineer Associate or Professional
* Additional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise