Job Title: Data Engineering Architect
We are seeking an experienced Data Engineering Architect to join our team. The ideal candidate will have a strong background in designing and implementing scalable data processing systems, as well as expertise in cloud-based infrastructure.
Required Skills and Qualifications:
* Azure DevOps pipelines, Git repos, artifact management
* Terraform, IaC governance patterns
* Azure Data Lake Storage Gen2, hierarchical namespace, ACL models
* Azure Data Factory, Databricks, Synapse, Spark
* Azure Functions, Key Vault, networking (VNets, private endpoints, firewalls)
* Monitoring stacks: Log Analytics, Application Insights, Azure Monitor
* Scripting: PowerShell, Python, Bash
* Security controls: RBAC, managed identities, secrets management, encryption
* CI/CD patterns, release strategy design, automated testing frameworks
* Jira, Confluence, ServiceNow
Benefits:
The successful candidate will have the opportunity to work with a talented team of engineers and contribute to the development of cutting-edge data processing systems. We offer a competitive salary, comprehensive benefits package, and opportunities for career growth and professional development.
Others:
1. Architect and maintain CI/CD pipelines for Data Lake components: Data Factory, Databricks, Functions, Synapse, Spark workloads, storage configurations.
2. Implement Infrastructure-as-Code with Terraform for provisioning storage accounts, networking, compute, identity, and security layers.
3. Enforce branching discipline, artifact integrity, automated testing, and controlled release gates.
4. Automate environment provisioning, ACL management, key rotation, lifecycle policies, and cluster configuration.
5. Integrate DevOps processes with enterprise security: RBAC, managed identities, Key Vault, private networking, encryption controls.
6. Build observability: logging, metrics, alerting, dashboards for pipelines and platform components.
7. Maintain backup, restoration, disaster-recovery patterns and test them for reliability.
8. Eliminate configuration drift through standardized templates and environment baselines.
9. Maintain and optimize agents, service connections, and deployment runtimes.
10. Perform incident response and root-cause analysis, document systemic fixes.
11. Deliver reusable automation modules for data engineering teams.
12. Optimize workload performance and cost within the Data Lake environment.
13. Ensure compliance with governance, audit requirements, and data protection mandates.
14. Drive continuous reduction of manual operational work through automation.