Sr. Cloud Automation Engineer
Job Summary:
We are seeking a seasoned Cloud Automation Engineer to join our team and work on our Azure Data Platform.
Key Responsibilities:
* Critical Tasks: Design, implement, and maintain CI/CD pipelines for data lake components: Data Factory, Databricks, Functions, Synapse, Spark workloads, storage configurations.
* Automation Expertise: Implement Infrastructure-as-Code with Terraform for provisioning storage accounts, networking, compute, identity, and security layers.
* Delivery Excellence: Enforce branching discipline, artifact integrity, automated testing, and controlled release gates.
* Environment Management: Automate environment provisioning, ACL management, key rotation, lifecycle policies, and cluster configuration.
* Integration and Security: Integrate DevOps processes with enterprise security: RBAC, managed identities, Key Vault, private networking, encryption controls.
* Observability: Build observability: logging, metrics, alerting, dashboards for pipelines and platform components.
* Disaster Recovery: Maintain backup, restoration, disaster-recovery patterns and test them for reliability.
* Configuration Management: Eliminate configuration drift through standardized templates and environment baselines.
* Agent and Service Optimization: Maintain and optimize agents, service connections, and deployment runtimes.
* Incident Response: Perform incident response and root-cause analysis, document systemic fixes.
* Automation Modules: Deliver reusable automation modules for data engineering teams.
* Workload Optimization: Optimize workload performance and cost within the Data Lake environment.
* Compliance: Ensure compliance with governance, audit requirements, and data protection mandates.
* Continuous Improvement: Drive continuous reduction of manual operational work through automation.