Skip to content
← Money

Protocol

LLM-as-a-Judge Extraction Evaluation

Define a rubric for what a correct extraction looks like for your documents, then use an LLM as a judge against that rubric to evaluate extraction outputs.

Advanced3 actions

Creator's principle

Define a rubric for what a correct extraction looks like for your documents. Use an LLM as a judge against that rubric to evaluate extraction outputs.

Creator's example

  • Adit Abraham recommends LLM-as-a-judge for document extraction evaluation, conditioned on being thoughtful about the rubric and how to approach it.

This is what the creator describes doing. It is not assigned to you.

Your version

3 total

Evaluate your document extraction outputs with an LLM judge built on a rubric you define for your own documents.

Steps 1, 2, 3 and 4 are implementation scaffolding: they structure the user's own rubric, judge prompt and review loop around the source-backed recommendation to use LLM-as-a-judge with a thoughtful rubric. The rubric content, document type, extraction fields and run cadence are user-supplied parameters; the source gives no schedule for running the judge.

  1. Write a rubric for what a correct extraction looks like for your [kind of documents are you extracting from] documents, stating the context ([kind of documents are you extracting from], [fields or information are you extracting from those documents]) and the explicit criteria the judge should check.

    • State the context: what kind of document this is and what a correct extraction is expected to contain.
    • State the explicit criteria the judge should check, so it can make a yes or no decision on each one.
    • Do not write 'a good response is helpful and accurate' — the judge cannot make a yes or no decision from that.
    • Do not use overly strict rules: the same information can be represented faithfully two different ways and you don't want to penalize that.

    Setup · onceDone when: A written rubric exists that states the document context and the explicit criteria a correct extraction must meet.NuthinButta scaffolding

  2. Run the judge on your extraction outputs and read its explanations of what was wrong and what was missing.

    • Read the explanation: it tells you what was wrong, what was missing, and often what to fix in your prompt to get this right.

    Recurring · on your cadenceDone when: The judge has produced a verdict and an explanation for the extraction outputs evaluated in this run.How-to from AI Engineer

  3. Check that the judge is judging correctly by reviewing its verdicts against failures you have seen before.

    • Use humans for the kinds of failures you haven't seen before, and to check that your LLM judges are judging correctly.

    Review · on your cadenceDone when: The judge's verdicts from this run have been checked against known failures.How-to from AI Engineer

Scaling and safety

Don't

  • ×Don't use overly strict rules in your benchmarking — the same information can be represented faithfully two different ways and you don't want to penalize that.
Where this came from

Provenance not established.

This protocol is marked source-derived, but its persisted source, analysis, or verification chain cannot be resolved. We won't claim a source.

Effort
HIGH
business