High-Performance Data Processing Engineer
Design, develop, and maintain high-performance data processing pipelines using Python libraries such as Pandas and Polars. Pipeline efficiency is critical for generating dynamic reports.
Develop RESTful APIs using FastAPI or similar frameworks to expose pipeline outputs.
Consume normalized Parquet files from multiple upstream sources to generate reports.
* Optimize report generation logic for speed and scalability.
* Integrate with messaging and storage mechanisms.
* Collaborate on infrastructure-as-code automation.
Participate in design discussions for future migration to Snowflake and/or a data lake architecture to improve data accessibility and scalability.