Data Pipeline Architect
As a senior software engineer, you have the opportunity to work on high-impact projects building robust, cloud-native data infrastructure that supports machine learning and analytics workflows at scale. This role is ideal for someone who has hands-on experience with Databricks, enjoys building efficient and scalable data pipelines, and is eager to grow their platform skills in a dynamic, multi-cloud environment.
About the Role:
* You will be part of a cross-functional team including platform engineers, DevOps, and data scientists, collaborating on ingestion, transformation, orchestration, and data reliability for production-grade pipelines.
Key Responsibilities:
1. Build and maintain scalable ETL/ELT pipelines on Databricks using Spark, Delta Lake, and Python.
2. Develop orchestration logic using tools such as AWS Step Functions, Lambda, or Databricks Workflows.
3. Contribute to medallion architecture layers (Bronze, Silver, Gold) for structured data processing.
4. Collaborate on infrastructure provisioning and pipeline automation using Terraform and GitHub Actions.
5. Troubleshoot Spark job performance and help ensure reliable, efficient data pipelines.
6. Support cross-functional teams (data scientists, ML engineers) by delivering curated, production-ready data.
Requirements:
* 3–6 years of experience in data engineering or data platform roles.
* Solid experience with Databricks and Delta Lake, including job and cluster setup.
* Strong in PySpark, SQL, and scripting for data transformation.
* Familiarity with AWS services: S3, Lambda, Step Functions, IAM, CloudWatch.
* Exposure to CI/CD practices and infrastructure automation using Terraform.
Benefits:
* Remote work option.
* Coworking space financial coverage.
* Flexible working hours.
* B2B benefits package.
* Paid days off annually.
* Workspace program: financial support for work equipment.
* English language lessons on all levels.
* Performance-based financial incentives.
* Paid courses and certifications.
* Participation at international conferences.