We measure our legal AI in public

Answer accuracy on statute questions

The assistant answers 100 statute questions — 25 each for Norway, Sweden, Denmark and Germany — through the same answer pipeline as the product: retrieval over the official statute corpora, the answer model, and the citation check that runs after every grounded answer. Each question is written in the jurisdiction’s own language, the way a lawyer would put it, and never names the governing statute.

Every answer is broken into legal claims. A claim that cites a source is checked against the text of that source; a legal statement without a citation counts as ungrounded. Deterministic checks catch citations that point to no retrieved source, to an act outside the corpus, or to a section that does not exist in the cited act. Each question also has required points — periods, amounts and conditions taken from the statute text — which a fixed pattern matcher looks for in the answer.

Last run Sep 28, 2026 · Build 37f924ff6 · Models claude-haiku-4-5-20251001, claude-opus-4-8, claude-sonnet-4-6

One run so far — the trend appears after the next one.

Questions are shown in the language they were asked in — they are the benchmark’s source data.

Download per-question results (JSON)Every answer, claim, verdict and source, unedited.

External benchmark: German case retrieval (GerDaLIR)

Every other topic on this page grades tasks we wrote, with gold we picked. This one does not. GerDaLIR is a public German legal-retrieval dataset built by a third party from Open Legal Data: a query is a passage of a court decision that cites another decision, with the reference itself masked, and the cited decision is the gold.

Each task makes the call the case “Rett” tab makes for a German matter, with the masked passage as the query, and passes when a cited decision is within the first 10 results — the product’s own ranking, source-type quotas included, not a retrieval-only ceiling.

Sample: 150 of the 309 test-split queries whose every cited decision our federal-court corpus holds (of 12298 test queries), drawn with a fixed seed. Every task links to the public record so it can be checked.

Dataset and licence (MIT)How the sample was drawn

0 of 153 cited decisions carry vectors in our store

None of the German decisions carried vectors for this run, so retrieval was keyword search plus the LLM re-rank. When they are embedded, the change will show in the trend above.

Without the LLM re-rank (search fusion order only): 1% (95% CI 0–4%) · with it: 2% · 150 tasks graded on both passes

External benchmark: German statute retrieval (GerLeRB)

GerLeRB is a public benchmark over German federal legislation, built by a third party: 367 legal questions, each paired with the one statutory paragraph that answers it, out of 90,798 paragraphs on gesetze-im-internet.de — the source our German statute corpus is synced from. The questions were generated with GPT-4.1 from court-decision contexts and then verified by legal experts. Nobody at Attorly wrote a question or picked a gold. German law only: it says nothing about the Nordic jurisdictions.

Each task makes the call the case “Rett” tab makes for a German matter, with the question as the query, and passes when the gold law is within the first 5 results — the product’s own ranking, source-type quotas included. A stricter grade records whether the result for that law also named the gold paragraph.

Every one of the 367 questions is a task (367 in this set) — no sampling. Each gold links to the paragraph on gesetze-im-internet.de so it can be checked.

Dataset and licence (CC BY 4.0)How the task set was built

Without the LLM re-rank (search fusion order only): 88% (95% CI 84–91%) · with it: 91% · 367 tasks graded on both passes

Assistant source citations

Measures whether the assistant’s source citations stay on relevant law: for each question, the top sources are graded against a list of clearly off-domain statutes. Citing e.g. maritime-labour law on an ordinary dismissal question counts as noise.

Without the relevance gate (top results ungated): 67% (95% CI 39–86%) · with it: 100% · 12 tasks graded on both passes

Download results (JSON)

Citation verification accuracy

After every grounded answer, the assistant checks each citation against the text of the source it points at and marks it verified, partly supported or not supported. This benchmark measures that verifier itself on labeled pairs of a claim and a verbatim statute excerpt with a known answer.

Half the pairs are near-misses: the right statute but the wrong period, amount or party — the case a lenient verifier waves through. A false accept, an unsupported claim marked verified, is the number that matters; a false reject only costs the reader a double-check.

Download results (JSON)

Playbook redlining

Each task gives our redlining engine a contract and a playbook of position ladders, the way a firm would. The engine grades every term and proposes edits; we check each output against gold answers written with the governing statute beside them.

Tasks cover employment contracts, NDAs and SaaS terms in Norway, Sweden, Denmark and Germany. The contracts are synthetic and written in-house so no client material is used. Every run uses the production prompt, and the checks are graded automatically without a model judging another model.

Download results (JSON)

The independent VLAIR study reports a lawyer baseline of 79.7% on its own redlining tasks. That task set is private and different from ours, so the figure is context, not a head-to-head comparison.