Research Engineer, Benchmarks
- Singapore
- Full-time
- $100,000 to $170,000
- Posted
Skills
Job description
ABOUT THE ROLE
Join a small, technical AI evaluation team building rigorous benchmarks for frontier AI agents. You will own the design and implementation of domain-specific evaluations that help research teams and customers understand how agents perform on realistic workflows.
WHAT YOU'LL DO
•Design, implement, and maintain internal benchmarks for evaluating frontier agents on domain-specific tasks.
•Partner with subject-matter experts to translate real-world workflows into benchmark tasks and evaluation criteria.
•Build reliable infrastructure to run models and agents against benchmark tasks at scale.
•Develop metrics and analyses to assess benchmark difficulty, reliability, and failure modes.
•Validate whether benchmark results align with real-world evaluations and user needs.
•Write clear technical documentation and reports explaining benchmark methods and results.
WHAT WE'RE LOOKING FOR
•Two to four years of experience in software engineering, machine learning engineering, or research engineering, including at least two years building AI benchmarks, evaluation infrastructure, or agent environments.
•Technical education or strong fundamentals in a technical field.
•Proficiency with Python, Docker, and Linux for research or production infrastructure.
•Hands-on experience designing and operating evaluations or environments for AI agents or large language models.
•Experience building reliable model or agent execution infrastructure, developing evaluation metrics or statistical analyses, and validating benchmark performance.
•Ability to work with subject-matter experts and understand workflows across technical or business domains.
•Strong technical writing, attention to detail, independent problem-solving, and comfort in an early-stage environment.
•Published research or technical writing on AI benchmarks, evaluation methods, or model failure modes is valued. Reinforcement learning pipelines, data generation, and agent evaluation experience are also relevant.
COMPENSATION & BENEFITS
Base salary: $100,000 to $170,000 USD annually. Visa sponsorship is available.
LOCATION
On-site in Singapore, Singapore.