Data Warehouse Architect
The role entails designing and implementing a new data warehouse instance utilizing Databricks (DBX) and AWS-native data services to provide optimized performance and scalability.
* Develop a tailored Databricks Lakehouse instance catering to the client's product-level data requirements.
* Design and deploy robust data ingestion pipelines using Spark (PySpark/Scala) and Delta Lake for seamless integration with AWS-native services.
* Integrate S3, Glue, Athena, Redshift, and Lambda with Databricks for enhanced performance and efficiency.
Key Responsibilities:
* Determine data models, optimize query performance, and establish warehouse governance best practices for streamlined data workflows.
* Collaborate cross-functionally with product teams, data scientists, and DevOps to enhance data quality and productivity.
* Maintain CI/CD, preferably DBX for data pipelines using GitOps and Infrastructure-as-Code for continuous improvement.
* Monitor data jobs and resolve performance bottlenecks or failures across environments for uninterrupted operations.
Required Skills & Experience:
* Databricks / Lakehouse Architecture
* AWS Native Integration
* Data Warehousing & Modeling
* Automation & DevOps