Job Summary
We are seeking a highly skilled Senior Data Engineer to join our team.
About the Role
* Design, build and manage complex data pipelines using various architectures;
* Collaborate with cross-functional teams to ensure compliance with data governance, architecture, and security requirements;
* Develop and maintain data models, transformation, schemas, metadata, and workload management;
* Drive automation through effective metadata management;
* Participate in ensuring compliance and governance during data use;
* Maintain the data catalog and collaborate with data owners and stakeholders.
Key Requirements
* Strong experience with various Data Management architectures, mainly AWS environment;
* Strong experience with Data Modeling;
* Ability to design, build, and manage data pipelines;
* Experience working with commercial and open source message queuing technologies;
* Highly creative and collaborative;
* Excellent communication skills and ability to work effectively with cross-functional teams;
* Solid knowledge in Databricks, Spark, Python, Terraform, SQL, Scala, S3, AWS Glue, AWS Athena, AWS Lambda, AWS Step Functions; Big data, Relational Databases;
* Experience in EMR, Gitflow, Agile Methodology; DevOps.
Technical Skills
* Databricks: Strong understanding of Databricks and its ecosystem;
* Spark: Experience with Apache Spark and its applications;
* Python: Proficiency in Python programming language;
* Terraform: Experience with infrastructure as code using Terraform;
* SQL: Strong understanding of SQL and its applications;
* Scala: Experience with Scala programming language;
* S3: Familiarity with Amazon S3 and its applications;
* AWS Glue: Experience with AWS Glue and its applications;
* AWS Athena: Experience with AWS Athena and its applications;
* AWS Lambda: Experience with AWS Lambda and its applications;
* AWS Step Functions: Experience with AWS Step Functions and its applications;
* Big data: Strong understanding of big data concepts and technologies;
* Relational Databases: Experience with relational databases and their applications.
Languages
* English: Advanced or Fluent
Benefits
* Remote work