We're looking for a highly skilled Data Engineer to join our team and drive innovation in enterprise-grade data pipelines.
Job Description
Azure Databricks is the platform of choice, and we need someone who can leverage its power to design, implement, and optimize scalable, governed, and performant data solutions.
This role involves working with cross-functional teams to modernize our data platform on the Azure Cloud, ensuring reliability, efficiency, and compliance across the full data lifecycle.
* Data Pipeline Development: Design, build, and optimize ETL/ELT pipelines using Azure Databricks and Azure Data Factory.
* Data Flows & Transformations: Develop pipelines, data flows, and complex transformations with ADF and PySpark for seamless data extraction, transformation, and loading.
* Data Processing: Develop Databricks Python notebooks for tasks such as joining, filtering, and pre-aggregation.
* Database & Query Optimization: Optimize database performance through SQL query tuning, index optimization, and code improvements to ensure efficient data retrieval and manipulation.
* SSIS & Migration Support: Maintain and enhance SSIS package design and deployment for legacy workloads; contribute to migration and modernization into cloud-native pipelines.
* Collaboration & DevOps: Work with cross-functional teams using Git for version control and Azure DevOps pipelines for deployment.
* Data Governance & Security: Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking, and role-based security.
* API & External Integration: Implement REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability.
* Automation: Automate ETL processes and database maintenance tasks using SQL Agent Jobs, ensuring data integrity and operational reliability.
Required Skills and Qualifications:
* 5+ years of hands-on expertise with Azure Databricks, Python, PySpark, and Delta Lake.
* 5+ years of proven experience with Azure Data Factory for orchestrating and monitoring pipelines.
* Strong SQL Server / T-SQL experience with a focus on query optimization, indexing strategies, and coding best practices.
* Demonstrated experience in SSIS package design, deployment, and performance tuning.