Job Title: Python Data Engineer
Job Description:
* We are seeking an experienced Python data engineer to join our team.
* The ideal candidate will have strong proficiency in Python for data processing, expertise in Pandas, and experience building backend services or APIs using frameworks like FastAPI.
* The successful candidate will be responsible for designing, building, and maintaining high-performance data processing pipelines using Python libraries (Pandas, Polars) and developing RESTful APIs using FastAPI or similar frameworks.
* They will also be responsible for consuming and processing normalized Parquet files from multiple upstream sources to generate dynamic Excel reports.
* Additionally, the candidate will contribute to a spec-driven development workflow (using GitHub Copilot, Claude, etc.) to scaffold and generate API/data pipeline code.
* They will optimize report generation logic for speed and scalability, targeting sub-20 second response times, and integrate with messaging and storage mechanisms (e.g., Service Bus, Storage Accounts).
* The candidate will also collaborate on infrastructure-as-code automation using Bicep (or similar IaC tools) and participate in design discussions for future migration to Snowflake and/or a data lake architecture.
* Finally, they will contribute to CI/CD pipelines using GitHub Actions.
Required Skills and Qualifications:
* Strong proficiency in Python for data processing (must have expertise in Pandas, nice to have: Polars, openpyxl).
* Experience building backend services or APIs using frameworks like FastAPI.
* Solid understanding of data modeling principles (Star Schema) and handling normalized datasets.
* Familiarity with enterprise messaging patterns and data integration from various sources (API-based and file-based).
* Experience working with GitHub and CI/CD pipelines (GitHub Actions or similar).
* Infrastructure-as-Code experience with Bicep or comparable tools (Terraform, AWS CDK).
* Comfort with spec-driven development and leveraging AI tools like GitHub Copilot for scaffolding.