Frontier models completing expert legal tasks end to end (Harvey LAB)

Share of holdout tasks the best frontier model completes end to end under Harvey's strict all-pass standard on its Legal Agent Benchmark, with the aggregate figure alongside.

Why it matters. The productisation valve for professional work. A model that passes bar-style questions but completes 7% of real matters is a tool, not a lawyer.

Proxy types

benchmark

Unit

share

Cadence

quarterly

Valve

invention to product

• series harvey_blog.lab_holdout.all_pass_top_model.pt
• series harvey_blog.lab_holdout.all_pass_aggregate.pt
• source Harvey research and insights posts · default tier 7 · Harvey; short quotation

Single digits under an all-pass standard. The benchmark is a vendor's, so the reading stays emerging until an independent run lands.

2 observations. Hollow points are disputed (see counterevidence). Every point links to its observation.

First scoring. Best frontier model completes 7.1% of Legal Agent Benchmark holdout tasks end to end under the all-pass standard (Harvey, 26 May 2026), inside the normal band, but the only source is the vendor (tier 7), which caps the status at emerging until an independent run exists.

• 2026-05-26 lab_holdout · 7.1%as of 2026-05-26

1. 2026-09-10○unmeasured →to ◐emergingconf — → 40 · evaluate

   First scoring. Best frontier model completes 7.1% of Legal Agent Benchmark holdout tasks end to end under the all-pass standard (Harvey, 26 May 2026), inside the normal band, but the only source is the vendor (tier 7), which caps the status at emerging until an independent run exists.

   obs:723ff70f

40 / 95 — limited or vague evidence

Confidence is independent of status: 90–95 multiple strong independent sources; 70–89 good evidence, some ambiguity; 50–69 mixed or hard to operationalise; below 50 limited or vague.

• Agent reliability across repeated runs (tau-bench pass^k) ◐emerging
• METR 80% time horizon ▲faster than normal
• Crosswalk: Products ⇄ Deployment & application (same valve)