Data Architect Specialist
The organization seeks a highly skilled data architect to lead the design and implementation of a cutting-edge data warehouse instance.
About the Role:
This position involves creating scalable pipelines, optimizing lakehouse performance, and integrating real-time and batch data sources across AWS cloud services.
Key Responsibilities:
* Design and deploy a Databricks Lakehouse instance tailored to the client's product-level data needs for high-performance analytics.
* Architect and implement robust data ingestion pipelines using Spark (PySpark/Scala) and Delta Lake for efficient data processing.
* Integrate AWS-native services like S3, Glue, Athena, Redshift, and Lambda with Databricks for optimized performance and scalability.
* Define data models, optimize query performance, and establish governance best practices to ensure data quality and integrity.
* Collaborate cross-functionally with product teams, data scientists, and DevOps to streamline data workflows and improve overall efficiency.
* Maintain Continuous Integration/Continuous Deployment (CI/CD) processes using GitOps and Infrastructure-as-Code for streamlined software development and deployment.
* Monitor data jobs and resolve performance bottlenecks or failures across environments to ensure seamless operations.