Job Title: Senior Data Architect
We are seeking a highly skilled Data Architect to design, implement and optimize enterprise-grade data pipelines. In this role, you will leverage Azure Databricks, Azure Data Factory, SQL Server and Python to enable scalable, governed and performant data solutions. You will play a key role in modernizing our data platform on the Azure Cloud ensuring reliability efficiency and compliance across the full data lifecycle.
Key Responsibilities:
* Design, build and optimize ETL/ELT pipelines using Azure Databricks (PySpark, Delta Lake) and Azure Data Factory (ADF).
* Develop pipelines, data flows and complex transformations with ADF, PySpark and T-SQL for seamless data extraction transformation and loading.
* Develop Databricks Python notebooks for tasks such as joining filtering and pre-aggregation.
* Optimize database performance through SQL query tuning index optimization and code improvements to ensure efficient data retrieval and manipulation.
* Maintain and enhance SSIS package design and deployment for legacy workloads contribute to migration and modernization into cloud-native pipelines.
* Work with cross-functional teams using Git (Azure Repos) for version control and Azure DevOps pipelines (CI/CD) for deployment.
* Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging lineage tracking and role-based security.
* Implement REST APIs to retrieve analytics data from diverse external data feeds enhancing accessibility and interoperability.
* Automate ETL processes and database maintenance tasks using SQL Agent Jobs ensuring data integrity and operational reliability.
Required Skills and Qualifications:
* 5+ years of hands-on expertise with Azure Databricks Python PySpark and Delta Lake.
* 5+ years of proven experience with Azure Data Factory for orchestrating and monitoring pipelines.
* Strong SQL Server / T-SQL experience with a focus on query optimization indexing strategies and coding best practices.
* Demonstrated experience in SSIS package design deployment and performance tuning.
* Hands-on knowledge of Unity Catalog for governance.
* Experience with Git (Azure DevOps Repos) and CI/CD practices in data engineering projects.
Nice to Have:
* Exposure to Change Data Capture (CDC) Change Data Feed (CDF) and Temporal Tables.
* Experience with Microsoft Purview Power BI and Azure-native integrations.
* Familiarity with Profisee Master Data Management (MDM).
* Working in Agile/Scrum environments.
Preferred Qualifications:
* Microsoft Certified: Azure Data Engineer Associate (DP-203)
* Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification
* Databricks Certified Data Engineer Associate or Professional
* Additional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise