Data Engineering Role Overview
As a Data Engineer, you will design and implement scalable data processing systems to efficiently handle large volumes of data. Your primary goal is to ensure high-performance access to the data while maintaining its accuracy and integrity.
You will be responsible for developing data ingestion pipelines and integration processes to smoothly transfer data from various sources into the data lake. Additionally, you will implement data transformation and enrichment processes using serverless technologies to guarantee data quality and consistency.
Collaboration with data scientists and analysts is essential in this role, as you will need to understand their data requirements and design appropriate data models and schemas in the data lake.
To achieve these objectives, you will leverage AWS services such as S3, Lambda, Glue, Athena, Redshift, and DynamoDB. You will also optimize data storage and retrieval mechanisms to provide high-performance access to the data.
Monitoring and troubleshooting the data lake infrastructure are critical aspects of this position. You will identify and resolve performance bottlenecks, data processing errors, and other issues to ensure the smooth operation of the system.
A key aspect of this role is staying up-to-date with new AWS services and technologies. This includes evaluating new services and technologies to enhance the data lake architecture, improve data processing efficiency, and drive innovation.
Key Responsibilities:
* Design and implement scalable data processing systems using AWS technologies.
* Develop data ingestion pipelines and integration processes.
* Implement data transformation and enrichment processes using serverless technologies.
* Collaborate with data scientists and analysts to understand their data requirements.
* Leverage AWS services such as S3, Lambda, Glue, Athena, Redshift, and DynamoDB.
* Optimize data storage and retrieval mechanisms.
* Monitor and troubleshoot the data lake infrastructure.
* Evaluate new AWS services and technologies to enhance the data lake architecture.
Required Skills and Qualifications:
- Extensive Experience: 5+ years working as a Data Engineer with a strong focus on AWS technologies and serverless architectures
- Familiarity with AWS Services: S3, Lambda, Glue, Athena, Redshift, and DynamoDB
- Programming Skills: Python, Java, or Scala, along with experience using SQL for data manipulation and querying
- Data Integration and ETL Tools: AWS Glue or Apache Spark
- Data Modeling Techniques: Star and snowflake schemas
- English Language Proficiency
Benefits:
- Professional Development: Opportunities for constant evolution of your skills, always in line with your interests
- Cross-Functional Team Environment: Collaborative, diverse, and innovative environment that encourages teamwork and professional growth
- Brazil-Based Benefits: Health insurance, dental plan, life insurance, transportation vouchers, meal voucher, childcare assistance, Gympass, TCS Cares, partnership with SESC, reimbursement of certifications, free TCS Learning Portal, international experience opportunity, discount partnership with universities and language schools, bring your buddy program, TCS Gems, and Xcelerate career platform