Cloud Data Engineer
Job Summary:
We are seeking an experienced Cloud Data Engineer to join our team. In this role, you will design and implement scalable data pipelines using Azure Databricks, Azure Data Factory, SQL Server, and Python.
Key Responsibilities:
* Design, build, and optimize ETL/ELT pipelines using Azure Databricks (PySpark, Delta Lake) and Azure Data Factory (ADF).
* Develop pipelines, data flows, and complex transformations with ADF, PySpark, and T-SQL for seamless data extraction, transformation, and loading.
* Develop Databricks Python notebooks for tasks such as joining, filtering, and pre-aggregation.
* Optimize database performance through SQL query tuning, index optimization, and code improvements to ensure efficient data retrieval and manipulation.
* Maintain and enhance SSIS package design and deployment for legacy workloads; contribute to migration and modernization into cloud-native pipelines.
* Work with cross-functional teams using Git (Azure Repos) for version control and Azure DevOps pipelines (CI/CD) for deployment.
* Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking, and role-based security.
* Implement REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability.
* Automate ETL processes and database maintenance tasks using SQL Agent Jobs, ensuring data integrity and operational reliability.
Required Skills and Qualifications:
* 5+ years of hands-on expertise with Azure Databricks, Python, PySpark, and Delta Lake.
* 5+ years of proven experience with Azure Data Factory for orchestrating and monitoring pipelines.
* Strong SQL Server / T-SQL experience with a focus on query optimization, indexing strategies, and coding best practices.
* Demonstrated experience in SSIS package design, deployment, and performance tuning.
* Hands-on knowledge of Unity Catalog for governance.
* Experience with Git (Azure DevOps Repos) and CI/CD practices in data engineering projects.
Benefits:
* Opportunity to work with cutting-edge technologies and tools.
* Collaborative and dynamic work environment.
* Professional growth and development opportunities.