Data Architect Position
As a Data Architect at our organization, you will play a pivotal role in designing and implementing scalable data pipelines using cutting-edge technologies such as Azure Databricks, Azure Data Factory, and SQL Server.
Main Responsibilities:
* Data Pipeline Development: Develop, optimize, and maintain ETL/ELT pipelines using Azure Databricks (PySpark, Delta Lake) and Azure Data Factory (ADF).
* Data Flows & Transformations: Create efficient data flows, transformations, and complex logic using ADF, PySpark, and T-SQL for seamless data extraction, transformation, and loading.
* Data Processing: Design Databricks Python notebooks for tasks such as joining, filtering, pre-aggregation, and data processing.
* Database & Query Optimization: Optimize database performance through SQL query tuning, index optimization, and code improvements to ensure efficient data retrieval and manipulation.
* SSIS & Migration Support: Maintain and enhance SSIS package design and deployment for legacy workloads; contribute to migration and modernization into cloud-native pipelines.
* Collaboration & DevOps: Work with cross-functional teams using Git (Azure Repos) for version control and Azure DevOps pipelines (CI/CD) for deployment.
* Data Governance & Security: Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking, and role-based security.
* API & External Integration: Implement REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability.
* Automation: Automate ETL processes and database maintenance tasks using SQL Agent Jobs, ensuring data integrity and operational reliability.
* Advanced SQL Expertise: Craft and optimize complex T-SQL queries to support efficient data processing and analytical workloads.
Required Skills:
* Azure Databricks, Python, PySpark, and Delta Lake expertise.
* Azure Data Factory for orchestrating and monitoring pipelines.
* Strong SQL Server / T-SQL experience with focus on query optimization, indexing strategies, and coding best practices.
* Demonstrated experience in SSIS package design, deployment, and performance tuning.
* Hands-on knowledge of Unity Catalog for governance.
* Experience with Git (Azure DevOps Repos) and CI/CD practices in data engineering projects.
Nice to Have:
* Exposure to Change Data Capture (CDC), Change Data Feed (CDF), and Temporal Tables.
* Experience with Microsoft Purview, Power BI, and Azure-native integrations.
* Familiarity with Profisee Master Data Management (MDM).
* Working in Agile/Scrum environments.
Preferred Qualifications:
* Microsoft Certified: Azure Data Engineer Associate (DP-203)
* Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification
* Databricks Certified Data Engineer Associate or Professional
* Addtional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise