Azure Data Engineer at Tata Consultancy Services
This role involves designing, developing, and maintaining scalable ETL/ELT pipelines using Azure Data Services, including Azure Data Factory, Azure Databricks, and Azure Synapse. You will build high-performance data processing solutions with Apache Spark (PySpark) on Azure Databricks and collaborate with data scientists, analysts, and business stakeholders to understand data requirements and deliver clean, reliable datasets.
The ideal candidate will have experience with columnar and table formats (Parquet, Delta, Hudi, Iceberg), use of Data Quality tools (Great Expectations, Soda), knowledge of Step Functions, EventBridge, or Kinesis, and best practices for API security (Cognito, WAF, IAM Policies). Soft skills include good communication and teamwork, proactivity in problem-solving, ability to handle agile environments and rapid changes, and design patterns, best practices, and operational procedures.