Job Overview
The ideal candidate will have experience with Azure DevOps pipelines, Git repositories, artifact management, Terraform, Infrastructure-as-Code governance patterns, Azure Data Lake Storage Gen2, hierarchical namespace, ACL models, Azure Data Factory, Databricks, Synapse, Spark, Azure Functions, Key Vault, networking (VNets, private endpoints, firewalls), monitoring stacks: Log Analytics, Application Insights, Azure Monitor, scripting: PowerShell, Python, Bash, security controls: RBAC, managed identities, secrets management, encryption, CI/CD patterns, release strategy design, automated testing frameworks, Jira, Confluence, ServiceNow.
Key Responsibilities
1. Design and maintain CI/CD pipelines for Data Lake components: Data Factory, Databricks, Functions, Synapse, Spark workloads, storage configurations.
2. Implement Infrastructure-as-Code with Terraform for provisioning storage accounts, networking, compute, identity, and security layers.
3. Enforce branching discipline, artifact integrity, automated testing, and controlled release gates.
4. Automate environment provisioning, ACL management, key rotation, lifecycle policies, and cluster configuration.
5. Integrate DevOps processes with enterprise security: RBAC, managed identities, Key Vault, private networking, encryption controls.
6. Build observability: logging, metrics, alerting, dashboards for pipelines and platform components.
7. Maintain backup, restoration, disaster-recovery patterns and test them for reliability.
8. Eliminate configuration drift through standardized templates and environment baselines.
9. Maintain and optimize agents, service connections, and deployment runtimes.
10. Perform incident response and root-cause analysis, document systemic fixes.
11. Deliver reusable automation modules for data engineering teams.
12. Optimize workload performance and cost within the Data Lake environment.
13. Ensure compliance with governance, audit requirements, and data protection mandates.
14. Drive continuous reduction of manual operational work through automation.