Job Overview:
We are seeking a seasoned Data Engineering expert with AWS expertise to join our team. As a Data Engineer, you will play a pivotal role in designing, developing, and maintaining APIs using AWS API Gateway.
You will be responsible for implementing integrations with Lambda or Fargate for data ingestion, developing processing pipelines in PySpark (AWS Glue or EMR), and managing data in S3 and Glue Data Catalog.
Your tasks will include ensuring data quality and integrity, optimizing pipeline costs and performance, and creating and maintaining technical documentation and development standards.