Software Engineering Role Overview
A skilled software engineer is needed to collaborate with AI researchers and contribute to the development of high-quality evaluation and training datasets for Large Language Models (LLMs). The successful candidate will work on curating verifiable software engineering challenges from public GitHub repository histories using a human-in-the-loop process. This involves reviewing and comparing model-generated code responses, evaluating code diffs for correctness, quality, style, and efficiency, and providing detailed rationales for ranking decisions.
Key Responsibilities
* Collaborate with AI researchers to shape the future of AI-powered software development
* Work with high-impact open-source projects to evaluate LLM performance on real bugs, issues, and developer tasks
* Influence dataset design that will train and benchmark next-gen LLMs
Requirements
* 7+ years of professional software engineering experience at top-tier product companies
* Strong fundamentals in software design, coding best practices, and debugging
* Excellent ability to assess code quality, correctness, and maintainability
* Proficient with code review processes and reading diffs in real-world repositories
* Exceptional written communication skills to articulate evaluation rationale clearly
Desirable Skills
* Prior experience with LLM-generated code or evaluation work
* Experience in LLM research, developer agents, or AI evaluation projects
* Background in building or scaling developer tools or automation systems