Senior Software Engineer Position
We are looking for an experienced Senior Software Engineer to join our team. The successful candidate will have a strong background in software engineering, with a focus on large language models (LLMs). This role requires collaboration with AI researchers, working on high-impact open-source projects, and evaluating LLMs performance on real bugs, issues, and developer tasks.
About the Project
The project involves building evaluation and training datasets to improve how LLMs interact with realistic software engineering tasks. A key focus is curating verifiable software engineering challenges from public GitHub repository histories using a human-in-the-loop process.
Key Responsibilities
* Review and compare model-generated code responses for each task using a structured ranking system.
* Evaluate code diffs for correctness, code quality, style, and efficiency.
* Provide clear, detailed rationales explaining the reasoning behind each ranking decision.
* Maintain high consistency and objectivity across evaluations.
* Collaborate with the team to identify edge cases and ambiguities in model behavior.
Requirements
* 7+ years of professional software engineering experience.
* Strong fundamentals in software design, coding best practices, and debugging.
* Excellent ability to assess code quality, correctness, and maintainability.
* Proficient with code review processes and reading diffs in real-world repositories.
* Exceptional written communication skills to articulate evaluation rationale clearly.
Preferred Qualifications
* Prior experience with LLM-generated code or evaluation work.
* Experience in LLM research, developer agents, or AI evaluation projects.
* Background in building or scaling developer tools or automation systems.
Engagement Details
* Commitment: ~20 hours/week.
* Type: Contractor.
* Duration: 1 month (potential extensions based on performance and fit).