Imagine working with a cutting-edge cloud-based technology stack, designing and developing innovative data pipelines that drive business growth.
About the Role
We are seeking an experienced Azure Data Engineer to join our team. As a key member of our data engineering group, you will be responsible for architecting, building, and maintaining large-scale data pipelines using Azure Data Factory (ADF), Azure Synapse Analytics, and other related technologies.
You will work closely with cross-functional teams, including data analysts and engineers, to design and implement data solutions that meet business needs. Your expertise in data integration, ETL processes, and data governance will be instrumental in ensuring data quality and integrity.
In this role, you will have the opportunity to work on diverse projects, from data warehousing and business intelligence to real-time analytics and machine learning. You will also be responsible for staying up-to-date with the latest industry trends and best practices in data engineering and cloud technologies.
Key Responsibilities
* Data Pipeline Development: Design, develop, and maintain data pipelines using Azure Data Factory (ADF) and Azure Synapse Analytics.
* Data Storage Solutions: Implement and manage data storage solutions using Azure Data Lake Storage (ADLS) Gen 2.
* Real-Time Data Processing: Utilize Azure Stream Analytics for real-time data processing and analytics.
* Data Quality and Governance: Ensure data quality and integrity through effective data governance practices.
* CI/CD Pipelines: Implement CI/CD pipelines using Azure DevOps for automated build and deployment.
Requirements
* Experience with Azure Functions: Experience with Azure Functions for serverless computing.
* Azure Logic Apps: Knowledge of Azure Logic Apps for workflow automation.
* Azure Stream Analytics: Experience with Azure Stream Analytics for real-time data processing.
* Azure API Management: Familiarity with Azure API Management (APIM) for managing APIs.
* Microsoft Purview: Understanding of Microsoft Purview for data governance and compliance.
* PySpark: Proficiency in PySpark for data processing and transformation.
* Azure Data Bricks: Pool or using Azure data bricks.
* Azure Data Factory: Strong experience with Azure Data Factory (ADF) for data integration and ETL processes.
* Azure Synapse Analytics: Strong experience of Azure Synapse Analytics for ETL processes.
* Azure Data Lake Storage: Expertise in Azure Data Lake Storage (ADLS) Gen 2 for scalable data storage.
* CICD Practices: Experience with CI/CD practices and tools, particularly Azure DevOps.
* Problem-Solving Skills: Strong problem-solving skills and attention to detail.
* Communication Skills: Excellent communication and collaboration skills.
This is an exciting opportunity to join our team and contribute to the success of our organization. If you are passionate about data engineering and cloud technologies, we encourage you to apply for this role.