Skip to content
Slow Variables
Ask

Products · Deployment & application

Frontier models completing expert legal tasks end to end (Harvey LAB)

Share of holdout tasks the best frontier model completes end to end under Harvey's strict all-pass standard on its Legal Agent Benchmark, with the aggregate figure alongside.

Reason through this

The reading

Single digits under an all-pass standard. The benchmark is a vendor's, so the reading stays emerging until an independent run lands.

Every readingeach dot links to its record
lab holdout · 7.1% · 2026-05-26 · reported
a readingnormal rangefast range
reported

1 reading drawn from harvey_blog.lab_holdout.all_pass_top_model.pt, one of 2 series listed under How it is measured; 2 observations in the evidence.

Source: Harvey research and insights posts. Harvey.

The numbers
as ofreadingvaluerecord
2026-05-26lab holdout7.1%reported

Why it matters

The productisation valve for professional work. A model that passes bar-style questions but completes 7% of real matters is a tool, not a lawyer.

What cuts against it

A vendor benchmark scored by the vendor (tier 7); the all-pass standard is deliberately harsh; partial completions have value in practice.

How it is measured
  • series harvey_blog.lab_holdout.all_pass_top_model.pt
  • series harvey_blog.lab_holdout.all_pass_aggregate.pt
  • source Harvey research and insights posts · default tier 7 · Harvey; short quotation
  • proxy benchmark · cadence quarterly · valve invention to product

Normal = the best frontier model completes under 30% of expert tasks end to end; fast = above 70%, where substitution rather than assistance becomes the product. Between is `emerging`.

Applied to harvey_blog.lab_holdout.all_pass_top_model.pt.

Latest readings
  • 2026-05-26 lab_holdout · 7.1%as of 2026-05-26
Why this status, and its history

First scoring. Best frontier model completes 7.1% of Legal Agent Benchmark holdout tasks end to end under the all-pass standard (Harvey, 26 May 2026), inside the normal band, but the only source is the vendor (tier 7), which caps the status at emerging until an independent run exists.

  1. 2026-09-10unmeasured to emergingconf — → 40 · the evaluator (automatic)

    First scoring. Best frontier model completes 7.1% of Legal Agent Benchmark holdout tasks end to end under the all-pass standard (Harvey, 26 May 2026), inside the normal band, but the only source is the vendor (tier 7), which caps the status at emerging until an independent run exists.

    source1
Why this confidence

Rests on 2 observations from Harvey research and insights posts; best evidence tier 7, grade D; current.

Confidence is independent of status: 90–95 multiple strong independent sources; 70–89 good evidence, some ambiguity; 50–69 mixed or hard to operationalise; 0–49 limited or vague evidence.

Predictions that lean on this