Job Overview
We are seeking an experienced Data Engineering professional to join our team and help us design, build, and maintain large-scale data pipelines using Azure Databricks, Python, PySpark, and Delta Lake.
Key Responsibilities:
* Design and develop efficient ETL/ELT pipelines for data integration and transformation.
* Develop and implement complex data flows and transformations using ADF, PySpark, and T-SQL.
* Optimize database performance through SQL query tuning, index optimization, and code improvements.
* Maintain and enhance SSIS package design and deployment for legacy workloads.
* Collaborate with cross-functional teams using Git for version control and Azure DevOps pipelines for deployment.
* Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking, and role-based security.
* Implement REST APIs to retrieve analytics data from diverse external data feeds.
* Automate ETL processes and database maintenance tasks using SQL Agent Jobs.
* Stay up-to-date with the latest technologies and best practices in data engineering.
Requirements:
* 5+ years of experience with Azure Databricks, Python, PySpark, and Delta Lake.
* Proven experience with Azure Data Factory for orchestrating and monitoring pipelines.
* Strong SQL Server/T-SQL experience with a focus on query optimization, indexing strategies, and coding best practices.
* Hands-on knowledge of Unity Catalog for governance.
* Experience with Git and CI/CD practices in data engineering projects.
Nice to Have:
* Exposure to Change Data Capture (CDC), Change Data Feed (CDF), and Temporal Tables.
* Experience with Microsoft Purview, Power BI, and Azure-native integrations.
* Familiarity with Profisee Master Data Management (MDM).
* Working in Agile/Scrum environments.
Preferred Qualifications:
* Microsoft Certified: Azure Data Engineer Associate (DP-203) or equivalent advanced Azure certification.
* Databricks Certified Data Engineer Associate or Professional.
* Additional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise.