Overview
Our client is a U.S.-based company that provides technical expertise, testing, and certification services to the global food and agricultural industry. The role is to build, maintain, and modernize data pipelines that process large-scale regulatory data and transform it into usable datasets for downstream applications and APIs. The engineer will work hands-on with Python, SQL, and related tools to untangle legacy pipelines, migrate processes to maintainable platforms such as Airflow, and ensure data quality for client-facing products. The role requires strong technical ability and a consulting mindset to learn undocumented systems, troubleshoot gaps, and design scalable solutions.
Responsibilities
* Design, build, and maintain data pipelines and ETL/ELT processes using Python and SQL (Snowflake, DBT, Airflow).
* Migrate legacy pipelines to maintainable platforms and improve data reliability, accuracy, and scalability.
* Monitor data jobs and pipelines, respond to alerts, and minimize downtime.
* Maintain and enhance DBT models and SQL scripts to support evolving business needs and data accuracy.
* Oversee data warehouse operations, including access control, query performance, and resource utilization.
* Perform root cause analysis of data job failures and implement preventive measures.
* Collaborate with data engineers, analysts, and stakeholders to support operational data needs and troubleshooting.
* Identify opportunities to automate manual tasks and improve pipeline efficiency.
* Document operational procedures, job schedules, and incident logs; provide regular updates on system health and performance.
Qualifications
* Minimum 7 years’ experience using Python for data analysis, extraction, transformation, and handling large datasets.
* Proficiency in Python 3+ with libraries/tools for data engineering (Pandas, NumPy, Jupyter Notebooks).
* Strong SQL skills with Oracle, Postgres, or MS SQL Server; solid database design, data modeling, and data warehousing concepts.
* Experience with Snowflake, DBT, Airflow; knowledge of data governance and data quality practices.
* Familiarity with Agile software development and collaborative, cross-functional teams.
* College degree or equivalent in computer science, software development, engineering, information systems, math, or related field.
* Preferred: NoSQL (MongoDB, DynamoDB, Cosmos DB), cloud platforms (AWS, Azure, GCP), Infrastructure as Code (Terraform).
What We Offer
* Professional development opportunities and ongoing skills evolution.
* Remote-friendly environment with potential opportunities outside the country.
* Collaborative and innovative culture with emphasis on teamwork.
#J-18808-Ljbffr