Job Overview
We are seeking a senior data engineer to build, manage, and optimize data pipelines. This role involves defining data flows and access to data sources while ensuring compliance with data governance, architecture, and security requirements.
Main Responsibilities
* Create, maintain, and optimize data pipelines for various data structures.
* Drive automation through metadata management.
* Collaborate with data owners and stakeholders to ensure proper use of defined data architecture and respect security principles.
Tech Skills & Qualifications
* Strong experience in Data Management architectures, especially in AWS environment.
* Familiarity with Databrics, Spark, Python, Terraform, SQL, Scala, S3, AWS Glue, Athena, Lambda, Step Functions, Big Data, and Relational Databases.
* Knowledge of EMR, Gitflow, Agile Methodology, and DevOps.
Key Competencies
* Designing and building data pipelines.
* Experience working with commercial and open source message queuing technologies.
* Collaborative and excellent communication skills.