Optimizing Data Infrastructure as a Senior Data Engineer. We are seeking an expert in designing and implementing large-scale data solutions to improve data efficiency, scalability, and reliability. Your responsibilities will include leading the development of data pipelines, collaborating with cross-functional teams on cloud-native integrations, and ensuring compliance with data governance regulations. To be successful, you will need hands-on experience with Azure Databricks, Python, PySpark, and Delta Lake, along with a strong understanding of database architecture, query optimization, and version control systems like Git.
Key Responsibilities:
• Designing and building scalable data architectures using Azure Databricks, Azure Data Factory, SQL Server, and Python.
• Developing complex ETL/ELT pipelines and data flows for seamless data extraction, transformation, and loading.
• Collaborating with data scientists and analysts to ensure efficient data processing and analytics workloads.
• Implementing data governance and security measures through Microsoft Purview and Unity Catalog integration.
• Automating data maintenance tasks and optimizing database performance through SQL query tuning and code improvements.
Required Qualifications:
• 5+ years of hands-on experience with Azure Databricks, Python, PySpark, and Delta Lake.
• Strong expertise in Azure Data Factory for orchestrating and monitoring pipelines.
• Proven experience in SSIS package design, deployment, and performance tuning.
• Hands-on knowledge of Unity Catalog for governance and data lineage tracking.
• Experience with Git (Azure DevOps Repos) and CI/CD practices in data engineering projects.
Nice to Have:
• Exposure to Change Data Capture (CDC), Change Data Feed (CDF), and Temporal Tables.
• Familiarity with Profisee Master Data Management (MDM).
• Working in Agile/Scrum environments.
Preferred Qualifications:
Microsoft Certified: Azure Data Engineer Associate (DP-203)
Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification
Databricks Certified Data Engineer Associate or Professional
Additional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise