We are seeking a skilled Data Engineer to join our team, responsible for designing and developing data pipelines using Azure Data Factory (ADF) and Azure Synapse Analytics.
Key Responsibilities:
* Design, develop, and maintain data pipelines using Azure Data Factory (ADF).
* Implement and manage data storage solutions using Azure Data Lake Storage (ADLS) Gen 2.
* Develop and maintain data pipelines using Azure Synapse Analytics.
* Utilize Azure Stream Analytics for real-time data processing and analytics.
* Ensure data quality and integrity through effective data governance practices.
* Implement CI/CD pipelines using Azure DevOps for automated build and deployment.
* Collaborate with data analysts and other stakeholders to understand data requirements and deliver solutions that meet business needs.
* Monitor and troubleshoot data pipelines to ensure optimal performance and reliability.
Requirements:
* Experience with Azure Functions for serverless computing.
* Knowledge of Azure Logic Apps for workflow automation.
* Experience with Azure Stream Analytics for real-time data processing.
* Familiarity with Azure API Management (APIM) for managing APIs.
* Understanding of Microsoft Purview for data governance and compliance.
* Proficiency in PySpark for data processing and transformation over either Spark.
* Prior experience with Azure data bricks.
* Strong experience with Azure Data Factory (ADF) for data integration and ETL processes.
* Expertise in Azure Synapse Analytics for ETL processes.
* Experience with CI/CD practices and tools, particularly Azure DevOps.
About the Role:
This role requires strong problem-solving skills and attention to detail, as well as excellent communication and collaboration skills. The successful candidate will be able to stay up-to-date with the latest industry trends and best practices in data engineering and cloud technologies.
What We Offer:
A dynamic and challenging work environment, opportunities for professional growth and development, and a competitive compensation package.