Job Opportunity
We're seeking a seasoned Big Data Developer with expertise in building scalable data architectures using Databricks, Spark, PySpark, SQL, and Delta Lake.
This role focuses on data ingestion, processing, and publishing high-quality datasets and visualizations for business intelligence.
Responsibilities:
* Design and optimize ETL/ELT pipelines in Databricks using PySpark, Spark SQL, and Delta Lake.
* Integrate diverse data sources (APIs, databases, cloud storage, SAP/legacy systems) and process them into actionable insights.
* Develop reusable pipeline frameworks, data validation logic, and performance-tuned transformations.
* Create curated datasets and deliver business-driven insights through data visualization tools.
* Implement best practices for data governance, orchestration, and version control.
* Troubleshoot pipeline performance issues and ensure data accuracy, reliability, and quality.