About this project, we are building large language model (LLM) evaluation and training datasets to train LLMs on realistic software engineering problems.
One approach involves creating verifiable software development tasks based on public repository histories using a synthetic method with human oversight.
The dataset coverage will be expanded to include various task types in terms of programming languages, difficulty levels, and other factors.
We are looking for experienced software engineers who can contribute to this project. Ideal candidates have experience working with well-maintained GitHub repositories and can analyze codebases effectively.
This role involves hands-on software engineering work, including developing automation scripts, triaging issues, and evaluating test coverage and quality.
We collaborate with researchers to design and identify challenging issues for LLMs, allowing us to refine our approach and improve model performance.
* Analyze and prioritize GitHub issues across trending open-source libraries.
* Set up and configure code repositories, including environment setup and testing.
* Evaluate unit test coverage and quality, ensuring that the codebase meets high standards.
* Run experiments locally to assess LLM performance in bug-fixing scenarios.
* Collaborate with researchers to identify challenging issues and design new projects.
The ideal candidate has strong experience with at least one of the following languages: Python, JavaScript, Java, Go, Rust, C/C++, C#, or Ruby.
Proficiency in Git, Docker, and basic software pipeline setup is also required. Additionally, you should be able to navigate complex codebases and understand how to optimize them for performance.
A strong understanding of software engineering principles and practices is essential for success in this role.
This position offers an exciting opportunity to contribute to cutting-edge AI research while gaining valuable experience as a software engineer.
Benefits of this role include working in a fully remote environment, collaborating with a talented team of engineers, and having the opportunity to work on diverse projects.
The duration of this contract is expected to be 1 month, with some flexibility in scheduling to accommodate different time zones.
Additional benefits may be discussed during the hiring process.