Nível Acadêmico: Ensino Superior Completo Turno/Horas -- Áreas de Atuação Profissional: TI - Projetos Descrição: Project Description We are looking for an experienced AWS data engineer, who will join the team responsible for the second phase of data warehouse cloud migration project.
The initial 'lift-and-shift' of an on-premise MS SQL data warehouse to AWS RDS has been completed.
This next phase focuses on refactoring the legacy system into a modern, serverless data architecture.
The primary goal is to migrate all data transformation logic from MS SQL to AWS Glue and prepare the data for consumption by business intelligence tools like AWS QuickSight.
Key Responsibilities Analyze existing MS SQL stored procedures and data structures within the PMM data warehouse to understand the current business and transformation logic.
Design, develop, and deploy scalable and efficient ETL/ELT pipelines using AWS Glue and PySpark to replace the legacy SQL processes.
Ingest and process data from various sources currently landing in Amazon S3, including SAP extracts, operational system feeds, and flat files.
Implement robust data validation, error handling, and logging mechanisms within Glue jobs.
Collaborate with architects to define data models and schemas suitable for the new data lakehouse architecture.
Optimize Glue jobs for performance and cost-efficiency.
Document the new data pipelines and processes.
Skills Must-Haves: oHigh proficiency in AWS Glue, including hands-on experience writing and deploying PySpark scripts.
oStrong experience with Python and the Spark framework (PySpark), particularly for data manipulation and transformation.
oKnowledge of MS SQL Server, including the ability to read, understand, and reverse-engineer T-SQL, stored procedures, and functions.
oHands-on experience with Amazon S3 as a data source and target.
oGood understanding of IAM roles and policies for managing secure access to AWS resources.
oStrong proficiency in English Habilidades: Highly Desirable: oExperience with AWS data orchestration services like AWS Step Functions and Lambda.
oFamiliarity with data warehousing concepts and dimensional modeling.
oKnowledge of monitoring and logging using Amazon CloudWatch.
oExperience working with data from SAP systems is a significant plus.
Formação Acadêmica:
Não informado
Salário:
A combinar
Cargo:
Engenheiro de Testes
Empresa:
Decision group
Atua com consultoria de mão de obra de serviços de tecnologia.
Ramo:
Recursos Humanos/ Recrutamento e seleção
(EF)