Azure Data Engineer Position Overview
As a key member of our team, the Azure Data Engineer will play a pivotal role in designing and implementing scalable data pipelines using Azure Data Services.
This position requires an individual with experience in ETL/ELT pipeline development, proficient in columnar and table formats such as Parquet, Delta, Hudi, and Iceberg.
The ideal candidate will have expertise in Data Quality tools like Great Expectations and Soda, along with knowledge of Step Functions, EventBridge, or Kinesis. Familiarity with best practices for API security, including Cognito, WAF, and IAM Policies, is also essential.
In addition to technical skills, the successful candidate should possess strong soft skills, including effective communication, teamwork, problem-solving, and adaptability in agile environments.
Key responsibilities include collaborating with data scientists, analysts, and business stakeholders to deliver clean, reliable datasets and high-performance data processing solutions using Apache Spark (PySpark) on Azure Databricks.