Job Description: We are seeking an expert Data Solutions Architect to design, implement, and optimize large-scale data pipelines using Azure Databricks and Azure Data Factory. This role will involve leveraging Python, PySpark, and Delta Lake to create scalable, governed, and performant data solutions.
In this position, you will play a key role in modernizing our data platform on the Azure Cloud, ensuring reliability, efficiency, and compliance across the full data lifecycle. You will be responsible for designing, building, and optimizing ETL/ELT pipelines using Azure Databricks (PySpark, Delta Lake) and Azure Data Factory (ADF).
You will also develop pipelines, data flows, and complex transformations with ADF, PySpark, and T-SQL for seamless data extraction, transformation, and loading. Additionally, you will develop Databricks Python notebooks for tasks such as joining, filtering, and pre-aggregation.
The ideal candidate will have hands-on experience with Azure Databricks, Python, PySpark, and Delta Lake, as well as strong SQL Server/T-SQL experience with a focus on query optimization, indexing strategies, and coding best practices.
Required Skills:
* 5+ years of hands-on expertise with Azure Databricks, Python, PySpark, and Delta Lake
* 5+ years of proven experience with Azure Data Factory for orchestrating and monitoring pipelines
* Strong SQL Server/T-SQL experience with a focus on query optimization, indexing strategies, and coding best practices
* Demonstrated experience in SSIS package design, deployment, and performance tuning
* Hands-on knowledge of Unity Catalog for governance
* Experience with Git (Azure DevOps Repos) and CI/CD practices in data engineering projects
Nice to Have:
* Exposure to Change Data Capture (CDC), Change Data Feed (CDF), and Temporal Tables
* Experience with Microsoft Purview, Power BI, and Azure-native integrations
* Familiarity with Profisee Master Data Management (MDM)
* Working in Agile/Scrum environments
Preferred Qualifications:
* Microsoft Certified: Azure Data Engineer Associate (DP-203)
* Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification
* Databricks Certified Data Engineer Associate or Professional
* Additional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise