As Filed, Part Two

Part one ended with the extractor about half built and a promise that the next post would cover how it's put together and how well it scores, so this is that post, and I'm writing it from a slightly awkward spot where the numbers are real but still moving week to week. Everything below is as of September 5. The short version is that there are now 214 financial maintenance covenants sitting in a table, pulled out of 162 credit agreements. Every one of them is backed by a verbatim quote that appears in exactly one place in the filed document, and the whole thing cost about $23 of model time to produce. It scores 0.87 against my label set, which sounds fine until you learn that the label set was drafted by the same family of model and that I haven't reviewed it yet. I'll get to why I decided that was acceptable, because I think it's the more interesting part.

Counting the corpus again

In part one I said the corpus was 552 documents, of which 77 were credit agreements, and while both numbers were technically correct, the second one turned out to be close to meaningless. The classification came from the document's title, and titles in EDGAR exhibits are unreliable in both directions. When I sorted by length, 33 of the 77 "credit agreements" were cover pages and exhibit indexes under 10,000 characters, while 149 of the 344 "amendments" were full amended-and-restated agreements running well over 100,000 characters, with the entire covenant package inside them. The real count of documents that actually contain an agreement is 162, and getting there took one more correction, because 22 documents that looked like agreements by every structural test (length, numbered sections, a definitions article with hundreds of defined terms) turned out to be guarantee, collateral, intercreditor and pledge agreements. Those have the shape of a credit agreement and no maintenance covenants at all, and I only found them because the extractor, when it was handed them, mostly declined to say anything (on seven of them it asserted covenant-lite, which is wrong in a way it had no means of knowing, since a security agreement is covenant-lite only in the sense that a cookbook is).

I'll mention one wrong turn here, because it says something about how much weight a document's caption can bear. The first version of the classifier required the caption to say "credit agreement", which sounds reasonable, and it threw out 88 documents captioned "Amendment No. 3" or similar that attach the whole restated agreement as an exhibit. Those 88 had yielded 110 covenants in the batch run, more than the 69 documents that were captioned as credit agreements had (98), so the caption is good for telling you what a document isn't, and I wouldn't lean on it for much beyond that.

The segmentation from part one worked at the article level, which turned out to be the wrong grain for this problem, since a financial covenant sits in a numbered section inside the negative covenants article, "Section 7.11 Financial Covenants" or thereabouts, rather than having an article to itself. In maybe a quarter of agreements the heading names the ratio instead of the concept ("Leverage Ratio", "Fixed Charge Coverage Ratio"), which means routing by heading needs a vocabulary rather than a phrase. So the segmenter now works at the section level under both numbering conventions the corpus uses (the "SECTION 7.11" style and the bare "7.11 Heading" style, and some documents mix them), and it also indexes every defined term to the character span of its definition, which will matter later on. Across the full agreements, 95% have ten or more numbered sections, 84% have a section whose heading names a covenant (this was 66% until the segmenter learned that some agreements keep the covenants in an unnumbered annex and some amendments rewrite the section as a lettered clause), and 97% have at least fifty defined terms indexed. My first pass at this kept the table-of-contents entry rather than the body heading on six of the first seven documents I checked, for a reason that was obvious in hindsight, which is that a table of contents is by construction the longest ascending run of section numbers in a document, so a rule that preferred long runs was a rule that preferred the table of contents. What fixed it was a density test, on the theory that a heading with another heading within 250 characters on either side is a list rather than a section, and that theory turned out to hold across the corpus.

How it reads a document

The one rule I'd keep if I had to throw everything else away is that the model has to quote the document. Every covenant it returns has to carry a quote of at least twenty characters that occurs exactly once in the frozen text of the filing, or it's rejected. If a quote fails, the model gets one chance to repair it, with the nearest matching passage shown back to it, and a second failure drops the covenant and logs why. That's the whole defense against the model inventing something. It has no way to put a covenant into the table itself and can only point at one, and the pointer (the quote becomes a character span, which becomes the citation) gets re-checked every time the table is rebuilt. Across five runs and 302 model calls, the number of quotes that failed to locate after a repair attempt is zero, which I'd like to put down to a good prompt, but it's mostly because the model is good at copying text when you insist on it.

I try not to show the model the whole agreement if I can avoid it, partly for cost and partly because a 290,000-character document is a bad place to ask a precise question. For each document the routing works down a list in order. It starts with the sections the segmenter typed as financial covenants, and if there aren't any, it takes the negative covenants article. Failing that, it collects every sentence containing a covenant verb ("shall not permit", "will at no time exceed", "make or commit to make") alongside a ratio term, each with 3,000 characters of context on either side, and only when all of that fails does it send the whole document in chunks. On the full run, 29 of 184 documents fell all the way through to the last option, and they accounted for most of the bill.

Definitions go in with the text, and that's where the defined-term index comes in. If the candidate spans mention "Consolidated First Lien Net Leverage Ratio", the definition of that term comes along from the index (up to ten definitions per document, ranked by how often the term shows up in the spans and whether it's the object of a covenant verb). The first version didn't have this ranking and just took the first six definitions it happened to find, and on at least one document that left out the one definition that mattered, which I'll come back to below.

What comes back is a label in a fixed schema, with the covenant type from a closed list, the defined term, the numeric threshold, the ratio text as written, the test frequency, the springing trigger if there is one, a step schedule, the quote and a free-text note. One model does the extraction, and a second, smaller one independently reads a sample of the same documents as a check, for reasons I'll get to.

Scoring, and the problem with the labels

Part one said the labels would be mine, and they aren't, at least not yet. Thirty-one documents were labeled by the model in a single pass at the end of August, with every quote span-validated the same way, and I set aside seven blind holdouts that no model has seen. I haven't had the hours to go through the thirty-one properly, and rather than let that stall everything, I'm treating the labels as provisional gold. I froze the scorer and I'm scoring the extractor against the labels anyway, on the understanding that my review will improve the score when it happens and doesn't hold up the work in the meantime.

The obvious objection is that a model grading a model can agree with itself for bad reasons, and that's true. What makes it less bad than it sounds is that the extractor reads the document differently from the labeler (it has the sections and the ranked definitions routed in, and a much narrower question), so when the two disagree, the document itself usually settles it. So far the label set has been wrong outright once. Caesars' 2022 agreement carries an $850 million minimum liquidity covenant during a covenant relief period, the labels missed it entirely, and the label schema didn't even have a type for it (it does now). Two other labels were stale rather than wrong: they'd been typed "other" for a monthly EBITDAR test and a cash-on-hand floor because the schema had no better value when they were written, and they were retyped once it did.

Here's how the score moved on the same ten documents, using a second version of the scorer that I'll explain further down:

Two of the run-one misses are worth spelling out, because they're the kind of error that would look completely fine on a spreadsheet. The Caesars covenant is written on the "Total Leverage Ratio", which sounds like a gross number, but the definition says Total Net Debt over EBITDA, and the model typed it gross from the name because the definition never reached it (the routing had attached the first six definitions it found, and this wasn't among them). Fixing it took two things, a sentence in the prompt saying the definition wins over the name, and the ranking, so that the right definition actually gets attached. The second miss was Fluor's 2004 agreement, where the model returned a $400 million cap on subsidiary debt as a maintenance covenant. That's a basket, a limit on incurring a category of debt, and it belongs with the negative covenants rather than the financial ones, which is a distinction a credit analyst makes without thinking about it and a model has to be told explicitly.

I versioned the scorer because I didn't want to quietly loosen the grading to make the number go up. The original scorer matches a covenant by type and by overlap of the quoted spans, and it was marking things as misses that were plainly the same covenant quoted from a different part of the same sentence (the label quoted the "shall not permit" head, the extractor quoted the threshold clause), along with defined-term strings that differed by a parenthetical and step schedules that differed by date format. The second version matches on the same sentence, normalizes the term and date strings, and collapses step schedules to their change points before comparing, because the model returns every test date and threshold while the labels only recorded the dates the threshold changes (I kept the full schedule, since that's what the document says). Under the original scorer run 3 is 0.80, and I report both numbers for every run so that neither one is hidden.

The runs also showed one more thing I hadn't expected to matter at this scale, which is that adaptive reasoning isn't deterministic. On identical inputs one covenant's type flipped between two runs, and at fifteen covenants that's 0.07 of F1 on its own. The more stable number is the score on the twenty-nine labeled documents inside the full run, and that also comes out at 0.87.

What the full run looked like

The 184 documents went through the batch API in a single submission, which came to 231 calls and about three minutes of wall clock with no errors, and it cost $17.82 against the $10 I'd estimated from the ten-document runs (29 documents fell through to whole-document reads, and the model wrote more than I'd assumed, mostly step schedules). The second model then read twenty of the same documents for $0.61 and agreed with the first on the covenant set in eighteen of them, with eleven field-level disagreements and none at all about whether a covenant existed. Total model spend on the project to date is $22.85, which I mention because I'd assumed going in that this part would be the expensive one.

Ninety-six of these are tested quarterly, 53 are springing, 43 apply at all times, and the rest are monthly or annual or something the schema doesn't have a name for yet.

Close

The 19 documents with no valid output split two ways. Fifteen of them weren't credit agreements at all, as above, and the other four are routing misses, where the sentence windows surfaced reporting covenants or baskets instead of the covenant article. Those four are the next thing I'd fix on the reading side. Over the 162 documents that actually are agreements, 158 produced either a covenant or a covenant-lite finding, which is a better number than I expected and probably a bit flattering, since "produced a finding" and "produced the right finding" are different things and only 29 of the 162 have labels to check against.

Per agreement it works out to around ten cents, and half that through the batch API, so the whole corpus is a one-time expense of under fifty dollars, and a document only ever needs reading once per version. The routing does more to keep it cheap than the discount does, since a whole-document read costs five to eight times what a routed one does, and the batch pricing sits on top of that.

Where it's still wrong

It's mostly amendments, which I think is the bigger of the problems left. A sentence like "Section 7.11 is hereby amended by replacing '4.50 to 1.00' with '5.00 to 1.00'" edits a covenant without restating it, and reading it correctly means resolving the edit against the parent agreement, which nothing here does yet. The full amended-and-restated agreements are handled because they carry the whole text. The short amendments, which is where the actual step-downs over a facility's life tend to live, aren't handled, and I suspect they're the harder half of the problem rather than the tail of it.

Then there are the four routing misses, and two documents whose covenant section the segmenter still can't find. One puts the covenants in an unnumbered annex, and the other rewrote the section as a lettered clause inside an amendment, so the only candidate heading is the table-of-contents entry. The second model, reading on its own, made two errors the first didn't (a threshold in millions where the schema wants dollars, and a "Consolidated Secured Debt Ratio" typed as other rather than senior secured leverage), and the prompt fixes for both are now in.

And the labels are still drafted by the model. Every score in this post is against a label set I haven't read, and the corrections made to it so far were made because the document text confirmed them rather than because I did.

What's usable now

Locally, the same app from part one has grown a covenants register with the whole table in it, which you can filter by type, frequency, confidence and review status. Next to it there's a per-company view that lists each agreement's covenants and links into the document at the exact character, and a labeling workbench that loads the extractor's draft into a form, checks every quote against the text and saves the label. The workbench is where the review will happen, whenever I get to it. There's also an MCP server over the same tables, so a coding assistant on my machine can ask for a company's covenants, its filed statements as of a date, or the exact text behind a citation, and get the answer straight from the parquet with the source table and license attached.

Close

Close

The project is still private, and part three will be a hosted version of the application, with the statements and the covenant table behind it, once the amendments are handled and I've had a chance to review the labels.