Responsibilities
- Prepare, clean, convert, validate, and maintain benchmark datasets and flag ambiguities that could compromise results.
- Build infrastructure and pipelines that run evaluations consistently across model APIs and terminal agents.
- Create lightweight containerized evaluation environments and viewers for tasking and tool-use assessment.
- Support fine-tuning experiments and compare baseline with post-training model performance.
- Build scoreboards and leaderboards showing model-, benchmark-, task-, domain-, and rubric-level results.
- Develop tools to identify failure modes, compare models and agent scaffolds, and track capability changes over time.
- Build HTML viewers and internal tools for task authoring, expert annotation, review, quality control, and structured data collection.
- Collaborate with researchers and engineers to ensure evaluation data and published outputs are accurate, consistent, and integrated.

Requirements
- 4+ years of professional experience building and maintaining complex systems.




- Strong Python engineering skills and experience working in existing codebases and infrastructure.
- Experience preparing, cleaning, validating, and maintaining datasets.
- Experience with Docker and reproducible execution environments.
- Ability to collaborate with researchers and scientists and translate methodology into working systems.
- Hands-on AI evaluation experience or experience with Harbor, Terminal-Bench, or Inspect is a strong plus.
- Experience fine-tuning or post-training open-source LLMs, agentic or tool-use evaluation, LLM-as-judge or rubric-based grading, data-heavy dashboards, leaderboards, visualizations, or related open-source or published benchmark work is preferred.

Benefits
- Medical, dental, and vision coverage.
- 401(k).
- Monthly wellness and fitness stipend.
- Paid time off and company holidays.
- Annual company off-sites in Tahoe, Mendocino, Mexico City, San Diego, and Park City.
- Parent-friendly policies, paid fam

📌 Software Engineer (España)
🏢 Office Hours
📍 España