Job Title: Data Engineering Lead
Key Responsibilities:
* Design and maintain data pipelines for our data lake components.
* Implement infrastructure as code using Terraform to provision storage accounts.
* Enforce best practices in branching, artifact integrity, automated testing, and controlled release gates.
* Automate environment provisioning, ACL management, key rotation, lifecycle policies, and cluster configuration.
* Integrate DevOps processes with enterprise security controls.
* Maintain backup, restoration, disaster recovery patterns, and test them for reliability.
* Eliminate configuration drift through standardized templates and environment baselines.
* Maintain and optimize agents, service connections, and deployment runtimes.
* Perform incident response and root cause analysis, document systemic fixes.
* Deliver reusable automation modules for data engineering teams.
* Optimize workload performance and cost within the data lake environment.
* Ensure compliance with governance, audit requirements, and data protection mandates.
Requirements:
* Azure DevOps pipelines, Git repositories, artifact management.
* Terraform, IaC governance patterns.
* Azure Data Lake Storage Gen2, hierarchical namespace, ACL models.
* Azure Data Factory, Databricks, Synapse, Spark.
* Azure Functions, Key Vault, networking (VNets, private endpoints, firewalls).
* Monitoring stacks: Log Analytics, Application Insights, Azure Monitor.
* Scripting: PowerShell, Python, Bash.
* Security controls: RBAC, managed identities, secrets management, encryption.
* CI/CD patterns, release strategy design, automated testing frameworks.
* Jira, Confluence, ServiceNow.