AWS Data Engineer Job Description
As an AWS Data Engineer, you will be responsible for designing, developing, and maintaining APIs using AWS API Gateway, implementing integrations with Lambda or Fargate for data ingestion, and developing processing pipelines in PySpark (AWS Glue or EMR).
* Data Ingestion: Implement integrations with Lambda or Fargate to ingest data from various sources.
* Pipeline Development: Develop processing pipelines in PySpark (AWS Glue or EMR) to process large datasets.
* Data Management: Manage data in S3 and Glue Data Catalog to ensure data quality and integrity.
* Optimization: Optimize pipeline costs and performance to meet business requirements.
Required Skills and Qualifications
To succeed as an AWS Data Engineer, you should have:
* Proven experience with AWS and strong knowledge of PySpark.
* Experience with integration and manipulation of data in REST APIs.
* Good practices in version control (Git) and CI/CD.
* Knowledge of infrastructure as code (Terraform or CloudFormation).
* Experience with columnar and table formats (Parquet, Delta, Hudi, Iceberg) and use of Data Quality tools (Great Expectations, Soda).
Benefits
At TATA Consultancy Services, we offer a collaborative, diverse, and innovative environment that encourages teamwork and professional development. As a data engineer, you will have opportunities to work outside Brazil, participate in training programs, and enjoy a range of benefits including:
* Health insurance.
* Dental plan.
* Life insurance.
* Transportation vouchers.
* Meal voucher.
* Childcare assistance.
* Gympass.
* Reimbursement of certifications.
Working Environment
We are a dynamic team that values innovation and collaboration. As a data engineer, you will have the opportunity to work on various projects and collaborate with a diverse team of experts.