Job Overview
We are seeking a seasoned DevOps professional to join our team. As a senior-level DevOps Engineer, you will play a key role in architecting and maintaining continuous integration and delivery pipelines for data lake components. This includes designing and implementing CI/CD pipelines for data factory, Databricks, Functions, Synapse, and Spark workloads. Additionally, you will implement infrastructure-as-code with Terraform for provisioning storage accounts, networking, compute, identity, and security layers.
Key Responsibilities
* Design and maintain CI/CD pipelines for data lake components: data factory, Databricks, Functions, Synapse, Spark workloads, storage configurations.
* Implement infrastructure-as-code with Terraform for provisioning storage accounts, networking, compute, identity, and security layers.
* Enforce branching discipline, artifact integrity, automated testing, and controlled release gates.
* Automate environment provisioning, ACL management, key rotation, lifecycle policies, and cluster configuration.
* Integrate DevOps processes with enterprise security: RBAC, managed identities, Key Vault, private networking, encryption controls.
* Build observability: logging, metrics, alerting, dashboards for pipelines and platform components.
* Maintain backup, restoration, disaster-recovery patterns and test them for reliability.
* Eliminate configuration drift through standardized templates and environment baselines.
* Maintain and optimize agents, service connections, and deployment runtimes.
* Perform incident response and root-cause analysis, document systemic fixes.
* Deliver reusable automation modules for data engineering teams.
* Optimize workload performance and cost within the data lake environment.
* Ensure compliance with governance, audit requirements, and data protection mandates.
* Drive continuous reduction of manual operational work through automation.