Azure Data Engineer Role
As an Azure Data Engineer, you will be responsible for designing and developing scalable ETL/ELT pipelines using Azure Data Services. Your expertise in Apache Spark (PySpark) on Azure Databricks will enable you to build high-performance data processing solutions.
Key Responsibilities:
1. Design and Develop Scalable ETL/ELT Pipelines: Use Azure Data Services to create efficient and reliable ETL/ELT pipelines that cater to the needs of your organization.
2. Build High-Performance Data Processing Solutions: Leverage Apache Spark (PySpark) on Azure Databricks to build scalable and fast data processing solutions.
3. Collaborate with Stakeholders: Work closely with data scientists, analysts, and business stakeholders to understand their data requirements and deliver clean, reliable datasets.
4. Optimize Data Workflows: Analyze and optimize data workflows/pipelines for performance and cost efficiency in a cloud environment.
5. Implement Best Practices: Ensure best practices around data security, governance, and compliance are implemented throughout the data engineering lifecycle.
6. Develop CI/CD Pipelines: Design and develop Continuous Integration and Continuous Deployment (CI/CD) pipelines for data engineering workflows.
7. Maintain Existing Solutions: Monitor, troubleshoot, and enhance existing data solutions for reliability and performance.
8. Document Design Patterns: Document design patterns, best practices, and operational procedures to ensure knowledge sharing and consistency.