How we measure extraction recall — and why most pipelines can't
Filings contain a built-in answer key: numbers that are both XBRL-tagged and written out in the prose. Rendering every tagged fact into the string forms it would take in a sentence, then searching the text for them, gives a ground-truth set to score the extractor against. On the reference filing the sweep found 545 of 545.
Most extraction pipelines report what they found. Almost none report what they missed, because missing is invisible — a number the regex never matched leaves no trace.
Filings solve this for us. Detail-tagging means thousands of figures appear twice: once as a tagged XBRL fact, and once written out in a sentence. That overlap is a free answer key.
The check runs in three steps. Take every tagged numeric fact. Render each into the string
forms it would plausibly take in prose — 4,231, $4.2 billion, 4.2 million. Search the
document text for those forms; the ones that actually appear are the ground-truth set,
because a competent extractor must find them. Then score the sweep’s character spans
against those positions.
On the reference filing that came back 545 of 545 — 100% measured recall, printed on every run. A filing that scores below threshold is flagged rather than quietly ingested.
The point isn’t the number. It’s that the number exists at all: completeness stops being a claim and becomes a measurement, per filing, every time.