DevOps Engineer
This role is responsible for designing, implementing, and maintaining large-scale data engineering systems on Azure.
* Technical Skills:
* Azure DevOps pipelines, Git repos, artifact management
* Terraform, Infrastructure as Code governance patterns
* Azure Data Lake Storage Gen2, hierarchical namespace, ACL models
* Azure Data Factory, Databricks, Synapse, Spark
* Azure Functions, Key Vault, networking (VNets, private endpoints, firewalls)
* Monitoring stacks: Log Analytics, Application Insights, Azure Monitor
* Scripting: PowerShell, Python, Bash
* Security controls: RBAC, managed identities, secrets management, encryption
* CI/CD patterns, release strategy design, automated testing frameworks
* Jira, Confluence, ServiceNow
Key Responsibilities
1. Design and Implement CI/CD Pipelines: Architect and maintain continuous integration and continuous deployment pipelines for data engineering components on Azure.
2. Implement Infrastructure-as-Code: Utilize Terraform to provision storage accounts, networking, compute, identity, and security layers for the data engineering environment.
3. Enforce Discipline and Integrity: Enforce branching discipline, artifact integrity, automated testing, and controlled release gates to ensure the quality and reliability of the data engineering pipeline.
4. Automate Environment Provisioning: Automate environment provisioning, ACL management, key rotation, lifecycle policies, and cluster configuration to reduce manual operational work.
5. Integrate with Enterprise Security: Integrate DevOps processes with enterprise security controls, including RBAC, managed identities, Key Vault, private networking, and encryption controls.
6. Build Observability: Build logging, metrics, alerting, and dashboards for pipelines and platform components to ensure visibility into the data engineering environment.
7. Maintain Backup and Disaster-Recovery: Maintain backup, restoration, disaster-recovery patterns, and test them for reliability to ensure business continuity.
8. Eliminate Configuration Drift: Eliminate configuration drift through standardized templates and environment baselines to ensure consistency across the data engineering environment.
9. Maintain and Optimize Agents: Maintain and optimize agents, service connections, and deployment runtimes to ensure smooth operation of the data engineering pipeline.
10. Perform Incident Response: Perform incident response and root-cause analysis, document systemic fixes to improve the overall reliability of the data engineering environment.