Job Title: Cloud Data Architect.
Job Summary: We are seeking an experienced Cloud Data Architect to lead the design, implementation and optimization of our enterprise-grade data pipelines. This is a key role in modernizing our data platform on the cloud, ensuring scalability, reliability, efficiency, and compliance across the full data lifecycle.
Responsibilities: - Design, build, and optimize ETL/ELT pipelines using Azure Databricks, PySpark, Delta Lake, and Azure Data Factory.
- Develop pipelines, data flows, and complex transformations with ADF, PySpark, and T-SQL for seamless data extraction, transformation, and loading.
- Develop Databricks Python notebooks for tasks such as joining, filtering, and pre-aggregation.
- Optimize database performance through SQL query tuning, index optimization, and code improvements to ensure efficient data retrieval and manipulation.
- Maintain and enhance SSIS package design and deployment for legacy workloads; contribute to migration and modernization into cloud-native pipelines.
- Work with cross-functional teams using Git for version control and Azure DevOps pipelines (CI/CD) for deployment.
- Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking, and role-based security.
- Implement REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability.
Requirements: - 5+ years of hands-on expertise with Azure Databricks, Python, PySpark, and Delta Lake.
- 5+ years of proven experience with Azure Data Factory for orchestrating and monitoring pipelines.
- Strong SQL Server / T-SQL experience with a focus on query optimization, indexing strategies, and coding best practices.
- Demonstrated experience in SSIS package design, deployment, and performance tuning.
- Hands-on knowledge of Unity Catalog for governance.
- Experience with Git and CI/CD practices in data engineering projects.
Nice to Have: - Exposure to Change Data Capture (CDC), Change Data Feed (CDF), and Temporal Tables.
- Experience with Microsoft Purview, Power BI, and Azure-native integrations.
- Familiarity with Profisee Master Data Management (MDM).
- Working in Agile/Scrum environments.
Preferred Qualifications: Microsoft Certified: Azure Data Engineer Associate (DP-203), Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification, Databricks Certified Data Engineer Associate or Professional, Additional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise.