Job Opportunity
We are seeking a skilled Data Architect to build and validate five advanced data quality scorecards using existing SAP ECC data within our cloud-based lakehouse.
* Design and implement profiling logic.
* Develop rule-based data quality checks in PySpark.
* Generate field-level and row-level results.
* Publish business-facing dashboards in Power BI.
This role will also establish reusable templates, naming conventions, and repeatable processes to support future scorecard expansion (47 more) and help transition our organization away from Informatica IDQ.
Responsibilities:
* Rebuild Data Quality scorecards in Databricks.
* Develop profiling logic for nulls, distincts, and pattern checks.
* Build PySpark-based Data Quality rules and row/column-level metrics.
* Curate DQ datasets for Power BI scorecards.
* Establish reusable DQ rule templates and standardized development patterns.
* Work with SAP ECC data models.
* Support and mentor a junior developer on rule logic and development standards.
Requirements:
* Strong Databricks engineering experience (PySpark, SQL, Delta Lake).
* Hands-on experience building Data Quality rules, frameworks, or scorecards.
* Experience in profiling large datasets and implementing metadata-driven DQ logic.
* Ability to mentor, review code, and explain concepts clearly.
* Excellent communication skills in English.
* Familiarity with SAP ECC tables and key fields.
* Experience with Unity Catalog or Purview.
* Exposure to Lakehouse Monitoring or DQX accelerators.