Data Engineering Position
We are seeking a skilled Data Engineer to join our team in building robust, cloud-native data infrastructure that supports machine learning and analytics workflows at scale.
Key Responsibilities
* Build Scalable ETL/ELT Pipelines: Utilize Databricks with Spark, Delta Lake, and Python to create efficient and scalable pipelines.
* Develop Orchestration Logic: Leverage tools such as AWS Step Functions, Lambda, or Databricks Workflows for streamlined process execution.
* Contribute to Medallion Architecture Layers: Participate in structured data processing for Bronze, Silver, and Gold layers.
* Collaborate on Infrastructure Provisioning: Use Terraform and GitHub Actions for pipeline automation and infrastructure setup.
* Troubleshoot Performance Issues: Identify and resolve Spark job performance problems to ensure reliable data pipelines.
* Serve Cross-Functional Teams: Deliver production-ready data to support data scientists and ML engineers.
Required Skills and Qualifications
* Data Engineering Experience: 3–6 years of experience in data engineering or data platform roles.
* Databricks Expertise: Strong knowledge of Databricks, Delta Lake, including job and cluster setup.
* Programming Skills: Proficiency in PySpark, SQL, and scripting for data transformation.
* AWS Services: Familiarity with S3, Lambda, Step Functions, IAM, CloudWatch.
* CI/CD Practices: Exposure to infrastructure automation using Terraform.
Benefits
* Remote Work: Flexible work arrangement allowing remote work.
* Coworking Space Support: Financial coverage for coworking space expenses.
* Flexible Hours: Ability to work flexible hours.
* Benefits Package: Comprehensive benefits package, including paid time off and professional development opportunities.
* Professional Growth: Opportunities for career advancement and continuous learning.