About the Project
Our team is working on building evaluation and training datasets for Large Language Models (LLMs) to train them to work on realistic software engineering problems.
* We're focusing on building verifiable Software Development Tasks based on public repository histories in a synthetic approach with human-in-the-loop.
* We aim to expand dataset coverage to different types of tasks in terms of programming language, difficulty level, and more.
-----------------------------------
About the Role
We're seeking experienced software engineers with a tech lead level background who are familiar with high-quality public GitHub repositories.
* You should have experience working with well-maintained, widely-used repositories with 500+ stars.
* This role involves hands-on software engineering work, including development environment automation, issue triaging, and evaluating test coverage and quality.
-----------------------------------
Day-to-Day Work
As a Senior Software Engineer, you'll be responsible for:
* Analyzing and triaging GitHub issues across trending open-source libraries.
* Setting up and configuring code repositories, including Dockerization and environment setup.
* Evaluating unit test coverage and quality.
* Modifying and running codebases locally to assess LLM performance in bug-fixing scenarios.
* Collaborating with researchers to design and identify repositories and issues that are challenging for LLMs.
-----------------------------------
Required Skills
To succeed in this role, you'll need:
* Strong experience with at least one of the following languages: Python, JavaScript, Java, Go, Rust, C/C++, C#, or Ruby.
* Experience working with well-maintained, widely-used repositories with 500+ stars.
* Proficiency with Git, Docker, and basic software pipeline setup.
* Able to understand and navigate complex codebases.
* Comfortable running, modifying, and testing real-world projects locally.
-----------------------------------
Benefits
As a contractor assignment, you'll enjoy:
* Working in a fully remote environment.
* Opportunities to work on cutting-edge AI projects with leading LLM companies.
-----------------------------------
Nice to Have
While not required, previous experience in LLM research or evaluation projects, building or testing developer tools, or automation agents can be beneficial.