Senior Software Engineer for AI-Assisted Software Development
About the Role:
We are seeking an experienced software engineer to contribute to a project evaluating how Large Language Models (LLMs) interact with real code. The ideal candidate will have expertise in high-quality public GitHub repositories and can help design and identify repositories and issues that challenge LLMs.
About the Projects:
We are building LLM evaluation and training datasets to train LLMs to work on realistic software engineering problems. Our approach involves creating verifiable tasks based on public repository histories in a synthetic approach with human-in-the-loop, while expanding dataset coverage to different types of tasks.
Key Responsibilities:
* Analyze and triage GitHub issues across trending open-source libraries.
* Set up and configure code repositories, including Dockerization and environment setup.
* Evaluate unit test coverage and quality.
* Modify and run codebases locally to assess LLM performance in bug-fixing scenarios.
* Collaborate with researchers to design and identify repositories and issues challenging for LLMs.
Requirements:
* Strong experience with at least one programming language: Python, JavaScript, Java, Go, Rust, C/C++, C#, or Ruby.
* Experience working with well-maintained, widely-used repositories with 500+ stars.
* Proficiency with Git, Docker, and basic software pipeline setup.
* Ability to understand and navigate complex codebases.
* Comfortable running, modifying, and testing real-world projects locally.
Nice to Have:
* Previous participation in LLM research or evaluation projects.
* Experience building or testing developer tools or automation agents.
Work Arrangement:
* Fully remote environment.
* Opportunity to work on cutting-edge AI projects.