Senior AWS Data Engineer - Python, SQL, Airflow
Location: Remote
Contract: 3 months initial
We are seeking a skilled Data Engineer with a strong background in Quality Assurance to join our client's team. The ideal candidate will be responsible for developing and executing a comprehensive data quality strategy as they migrate their SQL databases to AWS. Your expertise in Python, SQL, and ETL frameworks like Airflow and dbt will ensure high-quality data pipelines and effective diagnosis of pipeline failures.
Key Responsibilities:
* Data Quality Strategy Development: Design and implement a robust data quality strategy that addresses current challenges in SQL to AWS migration, ensuring data integrity and reliability throughout the process.
* Data Quality Assurance: Conduct thorough quality assessments, validate data transformations, and ensure that all data quality metrics are met. Focus on identifying gaps and areas for improvement within existing ETL workflows.
* Technical Proficiency: Utilize Python and SQL for extracting, transforming, and loading data. Implement automated testing frameworks (pytest) to validate data integrity and functionality across pipelines.
* ETL Management: Leverage tools like Airflow and dbt to orchestrate data workflows and conduct regular checks to ensure pipeline performance. Identify and troubleshoot sources of failures.
* Cloud Infrastructure: Manage data storage and processing on AWS platforms, specifically with Redshift and S3. Optimize data storage solutions and ensure best practices for data security and accessibility.
* Root-Cause Analysis: Perform strong root-cause analysis to diagnose and resolve pipeline failures effectively. Document issues and fixes to improve future processes and communication with stakeholders.
* Collaboration: Work closely with cross-functional teams, including data scientists and business analysts, to continuously improve data processes and quality metrics. Communicate findings clearly and provide recommendations to enhance system performance.
* Documentation: Maintain comprehensive documentation of data quality processes, testing outcomes, and any identified issues, along with their resolutions.
Qualifications:
* Bachelor’s degree in Computer Science, Data Science, Engineering, or a related field.
* Proven experience as a Data Engineer, Data Analyst, or in a similar role with a focus on QA.
* Strong proficiency in Python, SQL, and experience with data pipeline tools such as Airflow and dbt.
* Familiarity with AWS services, particularly Redshift and S3.
* Excellent English speaking and communication skills, with experience presenting technical concepts to non-technical stakeholders.
* Strong analytical skills with a keen ability to perform root-cause analysis and troubleshoot complex data issues.
* Knowledge of data quality frameworks and methodologies is a plus.