Job Summary:
We're seeking an experienced Senior Data Engineer to join our team and help design, implement and optimize scalable data architectures on GCP.
This role will involve developing high-performance ETL/ELT pipelines, ensuring data quality and integrity, creating and maintaining data models aligned with business needs, and collaborating with cross-functional teams to support advanced analytics and machine learning use cases.
The ideal candidate will have expertise in BigQuery, Dataflow (Apache Beam), Cloud Storage, and Pub/Sub, as well as experience with SQL, Oracle Database, and PostgreSQL.
Additionally, they should have knowledge of orchestration using Cloud Composer (Airflow) and hands-on experience with CI/CD applied to data pipelines (Git, Terraform).
Key Responsibilities:
* Design and implement data architectures on GCP using services such as BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage, and Composer.
* Develop and optimize scalable, high-performance ETL/ELT pipelines.
* Ensure data quality, integrity, and security end-to-end.
* Create and maintain data models aligned with business needs.
* Collaborate with data scientists, analysts, and software engineers to support advanced analytics and machine learning use cases.
* Stay up-to-date with industry trends and best practices in data engineering.
Requirements:
* Proficiency in BigQuery, Dataflow (Apache Beam), Cloud Storage, and Pub/Sub.
* Experience with SQL, Oracle Database, and PostgreSQL.
* Knowledge of orchestration using Cloud Composer (Airflow).
* Hands-on experience with CI/CD applied to data pipelines (Git, Terraform).
* Cloud cost and performance optimization experience.
* GCP certifications.
* Knowledge of Kubernetes (GKE) and APIs on GCP.
* Experience with Machine Learning pipelines (Vertex AI, AI Platform).
* Previous involvement with Data Mesh and distributed architectures.
* Understanding of Data Lake layers.
* Knowledge of batch and streaming processing.
* Experience with data modeling (relational, dimensional, and NoSQL).