Data Engineer Job Description
Job Details
This position involves designing, developing, and maintaining APIs using AWS API Gateway. The ideal candidate should have a solid understanding of AWS services and be able to work in an agile environment.
Required Skills and Qualifications
* Proven experience with AWS services (API Gateway, Lambda/Fargate, S3, Glue, CloudWatch)
* Strong knowledge of PySpark (partitioning, joins, optimization)
* Experience with Glue Data Catalog and querying via Athena
* Knowledge of infrastructure as code (Terraform or CloudFormation)
* Good practices in version control (Git) and CI/CD
- Strong analytical and problem-solving skills
Benefits
* Design and develop scalable data pipelines
* Work with a talented team of engineers
* Opportunities for career growth and professional development
Others
Responsibilities include ensuring data quality and integrity, optimizing pipeline costs and performance, and creating and maintaining technical documentation and development standards.