Job Title: Data Engineering Professional
A leading organization seeks a highly skilled Data Engineer to develop and maintain robust data pipelines, utilizing cutting-edge technologies such as Azure Data Services, Apache Spark, and Azure Databricks. This is an exciting opportunity for an experienced professional to drive business growth through data-driven insights.
Key Responsibilities:
1. Design and implement scalable ETL/ELT pipelines using Azure Data Factory, Azure Synapse, and other related tools.
2. Collaborate with cross-functional teams to deliver high-performance data processing solutions that meet business requirements.
3. Develop and maintain clean, reliable datasets by working closely with data scientists, analysts, and stakeholders.
4. Optimize data workflows for performance, cost efficiency, and security in cloud environments.
5. Implement best practices around data governance, compliance, and security protocols.
6. Develop automated CI/CD pipelines for data engineering workflows, ensuring seamless deployment and monitoring of data solutions.
7. Troubleshoot and enhance existing data solutions for reliability, performance, and scalability.
8. Document design patterns, best practices, and operational procedures for future reference.
Requirements:
• Proven experience in designing, developing, and maintaining large-scale data pipelines using Azure Data Services and related technologies.
• Strong understanding of Apache Spark, PySpark, and Azure Databricks, with expertise in building high-performance data processing solutions.
• Excellent collaboration and communication skills, with the ability to work effectively with data scientists, analysts, and stakeholders.
• Strong analytical and problem-solving skills, with experience in troubleshooting and enhancing data solutions.
• Proficiency in Agile methodologies, with experience in developing and maintaining CI/CD pipelines.
• Strong understanding of data governance, compliance, and security protocols, with experience in implementing best practices.
What We Offer:
• Competitive salary and benefits package.
• Opportunities for career growth and professional development.
• Collaborative and dynamic work environment.
• Recognition and rewards for outstanding performance.