About the Role
As a Data Engineer, you will play a critical role in building robust and scalable data infrastructure that supports machine learning and analytics workflows at scale. This is an ideal opportunity for someone who has hands-on experience with Databricks and enjoys building efficient and scalable data pipelines.
You will be part of a cross-functional team including platform engineers, DevOps, and data scientists, collaborating on ingestion, transformation, orchestration, and data reliability for production-grade pipelines. Your primary responsibilities will include:
* Building and maintaining scalable ETL/ELT pipelines on Databricks using Spark, Delta Lake, and Python.
* Developing orchestration logic using tools such as AWS Step Functions, Lambda, or Databricks Workflows.
* Contributing to medallion architecture layers (Bronze, Silver, Gold) for structured data processing.
* Collaborating on infrastructure provisioning and pipeline automation using Terraform and GitHub Actions.
* Troubleshooting Spark job performance and ensuring reliable, efficient data pipelines.
* Supporting cross-functional teams (data scientists, ML engineers) by delivering curated, production-ready data.
This role requires a strong foundation in data engineering principles, including data transformation, pipeline design, and scalability. You should also have experience working with cloud-based technologies such as AWS and Databricks. Additionally, you will need to demonstrate excellent communication skills and be able to collaborate effectively with a distributed team.
Key Requirements
* 3–6 years of experience in data engineering or data platform roles.
* Solid experience with Databricks and Delta Lake, including job and cluster setup.
* Strong skills in PySpark, SQL, and scripting for data transformation.
* Familiarity with AWS services: S3, Lambda, Step Functions, IAM, CloudWatch.
* Experience designing pipelines in multi-tenant or highly-regulated environments.
Benefits and Perks
* Flexible working hours and remote work options.
* Ongoing professional development opportunities.
* A comprehensive benefits package, including paid time off and health insurance.
* The chance to work with a talented and diverse team of professionals.
* A competitive salary and bonus structure.
We're Looking for Someone Who
* Has a passion for data engineering and a desire to stay up-to-date with industry trends.
* Is a strong communicator and collaborator with excellent problem-solving skills.
* Can adapt quickly to new technologies and processes.
* Is committed to delivering high-quality results in a fast-paced environment.