Job Title: Data Engineer - Cloud Computing
About the Role:
* Data Engineer with experience in cloud computing, looking to leverage expertise to design and develop scalable data pipelines using Azure Data Services.
* Develop high-performance data processing solutions with Apache Spark on Azure Databricks.
* Collaborate with data scientists, analysts, and stakeholders to deliver clean and reliable datasets.
* Optimize data workflows for performance and cost efficiency in a cloud environment.
Responsibilities:
1. Pipeline Development: Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Services.
2. Data Processing: Build high-performance data processing solutions with Apache Spark on Azure Databricks.
3. Collaboration: Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver clean, reliable datasets.
4. Optimization: Optimize data workflows/pipelines for performance and cost efficiency in a cloud environment.
Requirements:
* Experience with columnar and table formats.
* Knowledge of data quality tools.
* Familiarity with Step Functions and EventBridge.
We're Looking for Someone Who:
* Has a strong understanding of cloud-based data processing technologies.
* Is experienced in designing and developing data pipelines.
* Can collaborate effectively with cross-functional teams.