About the role
Evaluation is one of the load-bearing pieces of Aether. Per-discipline benchmarks against measured experimental data, refusal-corpus regression, long-rollout-stability tracking, customer-specific calibration — all of it lives in code that has to run reliably on every commit and every release.
We do not ship a model that has not passed eval. The eval team is the gate.
What you'll do
- Maintain and extend the per-discipline benchmark suite — cases, meshes, reference data, scoring scripts.
- Build the eval-as-first-class-object infrastructure inside the agent runtime.
- Co-author the refusal-corpus methodology and the per-release benchmark papers.
- Work with applied engineers to extract customer-specific calibration data into the production eval pipeline.
Who we're looking for
- Strong empirical instincts. Comfortable saying "the experiment did not show what we claimed."
- Comfortable with the operations of evaluation at scale — flaky tests, distribution shift, reproducibility under non-determinism.
- Detail-oriented; the bug in the eval pipeline is often the bug.
Bonus
- Prior experience with adversarial evaluation or red-team methodology.
- Familiarity with benchmark-design practice in any scientific discipline.