Research Engineer, Model Performance & Quality

As a Research Engineer on the Model Performance team, you will help solve one of our greatest challenges: systematically understanding and monitoring model quality in real-time. This role blends research and engineering responsibilities, requiring you to train production models, develop robust monitoring systems, and create novel evaluation methodologies. 

Note: For this role, we conduct all interviews in Python.

Representative Projects

• Build comprehensive training observability systems - Design and implement monitoring infrastructure to keep an eye on how model behaviors evolve throughout training.
• Develop next-generation evaluation frameworks - Move beyond traditional benchmarks to create evaluations that capture real-world utility. 
• Create automated quality assessment pipelines - Build custom classifiers to continuously monitor RL transcripts for complex issues
• Bridge research and production - Partner with research teams to translate cutting-edge evaluation techniques into production-ready systems, and work with engineering teams to ensure our monitoring infrastructure scales with increasingly complex training workflows.

• Are proficient in Python and have experience building production ML systems
• Have experience with training, evaluating, or monitoring large language models
• Are naturally curious about debugging complex, distributed systems and thinking about failure modes
• Enjoy collaborative problem-solving and working across diverse teams - you’ll work on virtually all stages of our model training pipeline
• Can balance research exploration with engineering rigor.
• Have strong analytical skills for interpreting training metrics and model behavior
• Want to directly impact the quality and safety of deployed AI systems

• Experience with reinforcement learning and language model training pipelines
• Experience designing and implementing evaluation frameworks or benchmarks
• Background in production monitoring, observability, and incident response
• Experience with statistical analysis and experimental design
• Knowledge of AI safety and alignment research

• Formal certifications or education credentials
• Academic research experience or publication history
• Prior experience in AI safety or evaluation specifically