About this role:
We're looking for a senior software engineer with expertise in Large Language Models (LLMs) to join our team. The ideal candidate will have experience working at the intersection of software engineering, open-source ecosystems, and AI.
The main focus of this project is curating verifiable software engineering challenges from public GitHub repository histories using a human-in-the-loop process. This involves reviewing and comparing model-generated code responses, evaluating code diffs for correctness, quality, style, and efficiency, and providing clear rationales explaining the reasoning behind each ranking decision.
Required skills and qualifications include 7+ years of professional software engineering experience, strong fundamentals in software design, coding best practices, and debugging, excellent ability to assess code quality, correctness, and maintainability, proficient with code review processes and reading diffs in real-world repositories, and exceptional written communication skills to articulate evaluation rationale clearly.
Bonus points are given for experience in LLM research, developer agents, or AI evaluation projects, background in building or scaling developer tools or automation systems, and commitment to high-quality work.
Engagement details include a part-time contractor position (~20 hours/week), partial PST overlap required, and a duration of 1 month (potential extensions based on performance and fit).