Expand your career as a Data Engineer in a collaborative and innovative environment.
Data Engineering is a crucial role that involves designing, building, deploying, and maintaining the complex systems that store, process, and retrieve large amounts of data. As a Data Engineer, you will be responsible for architecting and implementing scalable data processing systems using AWS technologies and serverless architectures.
About this role
* You will have extensive experience (5+ years) working as a Data Engineer, with a strong focus on AWS technologies and serverless architectures;
* You will also have experience working as a Data Engineer, with focus on Azure is valuable;
* You will have in-depth knowledge of AWS services such as S3, Lambda, Glue, Athena, Redshift, and DynamoDB, and their capabilities for building scalable data processing systems;
* You will have proven expertise in designing and implementing AWS serverless architectures for large-scale data processing and storage;
* You will have strong programming skills in languages like Python, Java, or Scala, along with experience using SQL for data manipulation and querying;
* You will have hands-on experience with data integration and ETL tools, such as AWS Glue or Apache Spark, for transforming and processing data;
* You will have English proficiency;
Responsibilities
1. Design and implement an AWS Serverless DataLake architecture to efficiently handle large volumes of data and support various data processing workflows;
2. Develop data ingestion pipelines and data integration processes, ensuring the smooth and reliable transfer of data from various sources into the DataLake;
3. Implement data transformation and data enrichment processes using AWS Lambda, Glue, or similar serverless technologies to ensure data quality and consistency;
4. Collaborate with data scientists and analysts to understand their data requirements and design appropriate data models and schemas in the DataLake;
5. Optimize data storage and retrieval mechanisms, leveraging AWS services such as S3, Athena, Redshift, or DynamoDB, to provide high-performance access to the data;
6. Monitor and troubleshoot the DataLake infrastructure, identifying and resolving performance bottlenecks, data processing errors, and other issues;
7. Continuously evaluate new AWS services and technologies to enhance the DataLake architecture, improve data processing efficiency, and drive innovation;
8. Mentor and provide technical guidance to junior data engineers, fostering their growth and ensuring adherence to best practices;
9. Collaborate with cross-functional teams to understand business requirements, prioritize tasks, and deliver high-quality solutions within defined timelines;
Requirements
* AWS services: S3, Lambda, Glue, Athena, Redshift, and DynamoDB;
* Programming skills: Python, Java, or Scala, along with experience using SQL;
* Data integration and ETL tools: AWS Glue or Apache Spark;
* English proficiency;
Benefits
* Professional development and constant evolution of your skills;
* Opportunities to work outside Brazil;
* A collaborative, diverse, and innovative environment;
* TCS benefits - health insurance, dental plan, life insurance, transportation vouchers, meal/food voucher, childcare assistance, Gympass, TCS Cares, partnership with SESC, reimbursement of certifications, free TCS Learning Portal, international experience opportunity, discount partnership with universities and language schools, Bring Your Buddy, TCS Gems, Xcelerate;
About us
We promote an inclusive culture, we always work for equity. This applies to Gender, People with Disabilities, LGBTQIA+, Religion, Race, Ethnicity. All our opportunities are based on these principles. We think of different actions of inclusion and social responsibility, in order to build a TCS that respects individuality.
Data Lake, AWS, Spark, Python