Job Overview
This key role involves designing and managing Continuous Integration/Continuous Deployment pipelines for Data Lake components, implementing infrastructure automation with Terraform, enforcing branching discipline, and integrating DevOps processes with enterprise security standards.
Key Responsibilities:
* Design and manage CI/CD pipelines for Data Lake components: Data Factory, Databricks, Functions, Synapse, Spark workloads, storage configurations.
* Implement infrastructure automation with Terraform for provisioning storage accounts, networking, compute, identity, and security layers.
* Enforce branching discipline, artifact integrity, automated testing, and controlled release gates.
* Automate environment provisioning, ACL management, key rotation, lifecycle policies, and cluster configuration.
Maintain backup, restoration, disaster-recovery patterns and test them for reliability. Eliminate configuration drift through standardized templates and environment baselines. Maintain and optimize agents, service connections, and deployment runtimes. Perform incident response and root-cause analysis, document systemic fixes. Deliver reusable automation modules for data engineering teams. Optimize workload performance and cost within the Data Lake environment. Ensure compliance with governance, audit requirements, and data protection mandates. Drive continuous reduction of manual operational work through automation.
Required Skills & Qualifications:
Technical Skills:
* Expertise in architecting and maintaining CI/CD pipelines.
* Strong knowledge of Terraform and infrastructure automation.
* Familiarity with Data Factory, Databricks, Functions, Synapse, Spark workloads, and storage configurations.
* Understanding of branching discipline, artifact integrity, automated testing, and controlled release gates.
Soft Skills:
* Excellent problem-solving and analytical skills.
* Strong communication and collaboration skills.
* Ability to work in a fast-paced environment and adapt to change.
What We Offer:
Opportunities for Growth & Development:
* Collaborative and dynamic work environment.
* Ongoing training and professional development opportunities.
* Challenging projects that promote skill growth and expertise.
Benefits Package:
* Competitive salary and benefits package.
* Generous paid time off and holidays.
* Access to cutting-edge technology and tools.