Key Data Engineer Roles and Responsibilities
* We are seeking a skilled Data Engineer to design, develop, and maintain scalable data processing pipelines using Azure Data Services.
* The ideal candidate will have expertise in building high-performance data processing solutions with Apache Spark (PySpark) on Azure Databricks.
* This position involves collaborating with cross-functional teams to understand data requirements and deliver clean, reliable datasets.
* Data Engineers in our organization optimize data workflows/pipelines for performance and cost efficiency in the cloud environment.
* They implement best practices around data security, governance, and compliance.
* Developing CI/CD pipelines for data engineering workflows is also a key responsibility.
* Maintaining and enhancing existing data solutions for reliability and performance is crucial in this role.
Required Skills and Qualifications
* Design and development of scalable ETL/ELT pipelines using Azure Data Services.
* Building high-performance data processing solutions with Apache Spark (PySpark) on Azure Databricks.
* Collaboration with data scientists, analysts, and business stakeholders to understand data requirements.
* Optimization of data workflows/pipelines for performance and cost efficiency.
* Implementation of best practices around data security, governance, and compliance.
* Development of CI/CD pipelines for data engineering workflows.
* Maintenance and enhancement of existing data solutions for reliability and performance.
Benefits
- The opportunity to work with cutting-edge technology and tools in the field of data engineering.
- Collaborative and dynamic working environment.
Others
- We offer competitive salary packages and comprehensive benefits.
- Professional growth and development opportunities.