Key Responsibilities:
• Data Pipeline Development: Design, build, and optimize ETL/ELT pipelines using Azure Databricks (PySpark, Delta Lake) and Azure Data Factory (ADF).
• Data Flows & Transformations: Develop pipelines, data flows, and complex transformations with ADF, PySpark, and T-SQL for seamless data extraction, transformation, and loading.
• Data Processing: Develop Databricks Python notebooks for tasks such as joining, filtering, and pre-aggregation.
• Database & Query Optimization: Optimize database performance through SQL query tuning, index optimization, and code improvements to ensure efficient data retrieval and manipulation.
• SSIS & Migration Support: Maintain and enhance SSIS package design and deployment for legacy workloads; contribute to migration and modernization into cloud-native pipelines.
• Collaboration & DevOps: Work with cross-functional teams using Git (Azure Repos) for version control and Azure DevOps pipelines (CI/CD) for deployment.
• Data Governance & Security : Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking, and role-based security.
• API & External Integration: Implement REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability.
• Automation: Automate ETL processes and database maintenance tasks using SQL Agent Jobs, ensuring data integrity and operational reliability.
Required Skills:
* Azure Databricks
* Python
* PySpark
* Delta Lake
* Azure Data Factory
* SQL Server / T-SQL
* Git
* Azure DevOps
* Microsoft Purview
* Unity Catalog
Nice To Have:
* Change Data Capture (CDC)
* Change Data Feed (CDF)
* Temporal Tables
* Microsoft Power BI
* Azure-native Integrations
* Profisee Master Data Management (MDM)
* Agile/Scrum Environment
Certifications:
* Microsoft Certified: Azure Data Engineer Associate (DP-203)
* Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification
* Databricks Certified Data Engineer Associate or Professional