Job Overview
We are seeking a skilled professional to fill the role of Data Engineer in our team. As a key member, you will be responsible for designing, developing, and maintaining scalable data pipelines using Azure Services.
About the Role
This position involves working with complex data systems to build high-performance data processing solutions using Apache Spark on Azure Databricks. You will collaborate with data scientists, analysts, and stakeholders to understand data requirements and deliver clean, reliable datasets.
The ideal candidate will have experience with columnar and table formats such as Parquet, Delta, Hudi, and Iceberg. Proficiency in Data Quality tools like Great Expectations and Soda is also required. Additionally, knowledge of Step Functions, EventBridge, or Kinesis is necessary. Familiarity with API security best practices, including Cognito, WAF, and IAM Policies, is also desired.
Responsibilities
1. Data Engineering: Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Services.
2. Data Processing: Build high-performance data processing solutions with Apache Spark (PySpark) on Azure Databricks.
3. Collaboration: Work with data scientists, analysts, and business stakeholders to understand data requirements and deliver clean, reliable datasets.
4. Optimization: Optimize data workflows/pipelines for performance and cost efficiency in a cloud environment.
Benefits
* Professional development and constant evolution of skills
* Opportunities to work outside Brazil
* A collaborative, diverse, and innovative environment that encourages teamwork
* TCS Benefits – Brazil: Health insurance, Dental Plan, Life insurance, Transportation vouchers, Meal/Food Voucher, Childcare assistance, Gympass
Other Opportunities
* TCS Cares – free psychological assistance (24 hrs/day), legal, social, and financial assistance
* Partnership with SESC
* Reimbursement of Certifications
* Free TCS Learning Portal – Online courses and live training
This is an exciting opportunity to grow your career and contribute to a dynamic team. If you have the skills and passion for data engineering, we encourage you to apply.