Data Engineer
Hiring Now - Cloud Data Platform Professional
Technical Requirements
* Azure DevOps pipelines, Git repositories, artifact management systems
* Terraform, infrastructure-as-code governance models
* Azure Data Lake Storage Gen2, hierarchical namespace management, access control lists
* Azure Data Factory, Databricks, Synapse Analytics, Spark workflows
* Azure Functions, Key Vault, networking protocols (virtual networks, private endpoints, firewalls)
* Monitoring stacks: Log Analytics, Application Insights, Azure Monitor services
* Scripting languages: PowerShell, Python, Bash programming
* Security controls: Role-Based Access Control, managed identities, secrets management, encryption standards
* CI/CD patterns, release strategy design, automated testing frameworks
Roles and Responsibilities
1. Design and maintain CI/CD pipelines for data lake components: Data Factory, Databricks, Functions, Synapse, Spark workloads, storage configurations.
2. Implement infrastructure-as-code with Terraform for provisioning storage accounts, networking, compute resources, identity, and security layers.
3. Enforce branching discipline, artifact integrity, automated testing, and controlled release gates.
4. Automate environment provisioning, ACL management, key rotation, lifecycle policies, and cluster configuration.
5. Integrate DevOps processes with enterprise security: RBAC, managed identities, Key Vault, private networking, encryption controls.
6. BUILD OBSERVABILITY: logging, metrics, alerting, dashboards for pipelines and platform components.
7. Maintain backup, restoration, disaster-recovery patterns and test them for reliability.
8. Eliminate configuration drift through standardized templates and environment baselines.
9. Maintain and optimize agents, service connections, and deployment runtimes.
10. Perform incident response and root-cause analysis, document systemic fixes.
11. Deliver reusable automation modules for data engineering teams.
12. Optimize workload performance and cost within the data lake environment.
13. Ensure compliance with governance, audit requirements, and data protection mandates.
14. Drive continuous reduction of manual operational work through automation.