Job Title: Data Engineering Specialist
Overview:
* We are seeking an experienced Data Engineering Specialist to join our team. The ideal candidate will have a strong background in designing, developing, and maintaining scalable data pipelines using Azure Data Services.
About the Role:
* This role involves building high-performance data processing solutions with Apache Spark (PySpark) on Azure Databricks and collaborating with data scientists, analysts, and business stakeholders to understand data requirements and deliver clean, reliable datasets.
* The successful candidate will also be responsible for optimizing data workflows/pipelines for performance and cost efficiency in a cloud environment.
* In addition, they will implement best practices around data security, governance, and compliance.
* This role requires experience with columnar and table formats (Parquet, Delta, Hudi, Iceberg), as well as the use of Data Quality tools (Great Expectations, Soda).
* Familiarity with Step Functions, EventBridge, or Kinesis is also essential.
Responsibilities:
* Design and develop scalable ETL/ELT pipelines using Azure Data Services.
* Build high-performance data processing solutions with Apache Spark (PySpark) on Azure Databricks.
* Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver clean, reliable datasets.
* Optimize data workflows/pipelines for performance and cost efficiency in a cloud environment.
* Implement best practices around data security, governance, and compliance.
* Develop CI/CD pipelines for data engineering workflows.
* Monitor, troubleshoot, and enhance existing data solutions for reliability and performance.
Requirements:
* Experience with columnar and table formats (Parquet, Delta, Hudi, Iceberg).
* Use of Data Quality tools (Great Expectations, Soda).
* Familiarity with Step Functions, EventBridge, or Kinesis.
* Best practices for API security (Cognito, WAF, IAM Policies).
Soft Skills:
* Good communication and teamwork.
* Proactivity in problem-solving.
* Ability to handle agile environments and rapid changes.