Give three attorneys the same suggested contract edit (known in the legal industry as a redline), and you typically get three different outcomes. One might accept it as is, another may adjust the language because of their interpretation of the risk it creates, and the third might rewrite in their own words to be clearer, without materially changing the legal substance. Only the first counts as “accepted as written,” while the other two get a “rejected as written” label for completely different reasons.
At Crosby, our AI agents produce the first pass of a contract review, and an attorney makes edits where needed before the redlined contract goes back to the client. Every one of those attorney decisions is a potential training label. This post covers how we tapped into that potential: why legal data is hard to train on, what it took to build a Verifier model to predict attorney decisions, and how this model scales with data and model size.
Why working with legal data is hard
Post-training Large Language Models (LLMs) for legal applications is challenging because the “quality” of model-generated text is hard to determine without a human in the loop. This leaves no verifiable reward signal of the kind that Reinforcement Learning (RL) depends on. The same algorithms that helped models make great strides in the domains of mathematics and computer programming, where checking a solution is cheap and deterministic (e.g., running a unit test), cannot be used out of the box here.
Our attorneys usually know things the model doesn’t: how impatient the counterparty is, how past similar deals went, what’s broadly standard for this kind of transaction. Sometimes the model reasons correctly and still reaches the wrong conclusion because it didn’t have access to all the information it needed. Either it didn’t fit in the prompt, or it was never in context at all.
So we wanted to start with a toy problem: take a single redline our agents suggest, and try to predict whether an attorney would send it to a client as written. Whether a redline is “good” is hard to agree on, multiple lawyers might reasonably debate this. But whether a single lawyer sends an AI-generated redline to a client is something we already record.
A Verifier for legal judgment
We call this model the Verifier. For each redline our agents suggest, we track whether the reviewing attorney sent it to the client unchanged or edited it. Given the deal context, the relevant playbook rules, and a suggested redline, the Verifier predicts the probability that an attorney will accept the AI-generated redline exactly as written.
A good Verifier can unlock three things for Crosby:
- Escalation: If the Verifier can reliably identify the redlines our attorneys are going to accept, we can focus attorney time and attention on the edits that require a higher level of lawyer judgment.
- Evals: The judges we described in our evals post need a reference, an attorney-reviewed version of the same contract to compare against. The Verifier does not need a reference, so it can score our agents in production on contracts that no attorney has reviewed yet.
- Automation: If the Verifier’s predictions prove reliable enough, we could use its scores as a reward signal for training agents to generate better redlines that attorneys are more likely to accept.
Scaling the Verifier with data and parameters
To fine-tune multi-billion parameter LLMs with a binary classification objective in our domain, you need tens of thousands of contract reviews.1 Even with this volume, this data is noisy; as mentioned earlier, two attorneys can disagree on the quality of the same redline, so similar suggestions can end up with opposite labels. To find out whether the labels have consistent signal, we tested whether the Verifier improves predictably as data and model size grow. If it does, “more data should help” becomes a measured relationship we can build plans on, in the same spirit as the scaling laws for neural language models (Kaplan et al., 2020).
We ran two experiments: one varying model size and one varying training set size. Specifically, five models ranging from 0.8B to 27B parameters, fine-tuned on a training set of ~13,000 labeled redlines, and six training-set sizes ranging from 1/32 of the data up to all of it, with the 27B model. Each of these models were open-weight dense decoders, fine-tuned with a classification head to output an acceptance score, and evaluated on the same held-out set of production redlines that none of the models had seen before. The metric we tracked was PR-AUC, which is the area under the precision-recall curve. It ranges from 0.0 to 1.0 and summarizes how well the model separates accepted suggestions from edited ones across different score thresholds.
Scaling with Qwen3.5 model size
- Observed
- Least-squares fit
Validation PR-AUC
Non-embedding parameters, billions (log₂ scale)
| Label | Non-embedding parameters, billions | Validation PR-AUC | ±1 bootstrap SD bounds |
|---|---|---|---|
| 0.8B | 0.546 | 0.6539 | 0.633–0.6748 |
| 2B | 1.491 | 0.7021 | 0.6838–0.7204 |
| 4B | 3.364 | 0.6895 | 0.6701–0.7089 |
| 9B | 6.966 | 0.7015 | 0.681–0.722 |
| 27B | 24.457 | 0.7297 | 0.7109–0.7485 |
Scaling with training data
- Observed
- Least-squares fit
Validation PR-AUC
Training examples (log₂ scale)
| Label | Training examples | Validation PR-AUC | ±1 bootstrap SD bounds |
|---|---|---|---|
| d32 | 425 | 0.6383 | 0.6176–0.659 |
| d16 | 840 | 0.6493 | 0.6309–0.6677 |
| d8 | 1667 | 0.6673 | 0.649–0.6856 |
| d4 | 3351 | 0.6854 | 0.6663–0.7045 |
| d2 | 6665 | 0.7163 | 0.6985–0.7341 |
| d1 (full) | 13325 | 0.7297 | 0.7109–0.7485 |
In both slices, validation PR-AUC increased approximately linearly in the log of the quantity being scaled: roughly 1.3 percentage points per doubling of parameters and 1.9 percentage points per doubling of training data. The data-size experiment showed the clearest pattern. Increasing the training set from ~500 to ~13,000 redlines raised PR-AUC from ~0.64 to ~0.73. These results gave us a reason to keep collecting examples, but they also put the gains in perspective: an improvement of ~9 percentage points took roughly thirty times as much data, and because PR-AUC is bounded at 1.0, the line has to bend eventually.
We have evidence of steady progress over the range we tested, but not a guarantee that it continues. We suspect disagreement between attorneys sets a ceiling on how far it can go. But much of that disagreement isn’t random. There is some ground truth. Disagreement tends to track client context and deal history the model doesn’t have, as well as lawyer preferences/dispositions (which we think could be predictable and are testing in forthcoming research). This means ground truth can possibly be recovered by changing the model’s context or customizing agents around specific lawyer personas. So we’re working on two things in parallel: moving along the line, which happens on its own almost every time we review another contract, and raising the ceiling by getting more signal out of the reviews we already have.
Getting more signal out of contract reviews
We wanted to know whether we could make the examples we already had more useful. Most of the architecture and training changes we tried had little effect outside of model size. Changes to label construction and input context made larger differences. Reviewing disagreements with our attorneys helped us understand where to look, and let us categorize them into three sources:
- Preprocessing mistakes
- Deal context the model hadn’t seen
- Differences in attorney preferences.
All three can produce the same rejection label, but each calls for a different response; we share one example of each below.
Scaling Still Helps
Our domain is noisy in a way that coding and math are not. Some of that noise was a preprocessing bug, and we fixed it. Other parts of that noise come from disagreement between attorneys, and that isn’t a defect at all – it’s precisely these differences in legal judgment that we’re trying to model. And despite the disagreements, our Verifier still improves predictably with scale. This means that reviewing more contracts with our systems means a better Verifier, and that leads to better evals for our agents. Critically, each of these feeds back into the next review.
This is why we believe Crosby gets better for everyone the more contracts we review with our AI systems and our law firm’s oversight. Just as importantly, when experts disagree, don’t average the disagreement away. Work out which disagreements come from mistakes in the pipeline, which come from context the model is missing, and which are a matter of taste.
We still have a lot to learn about measuring legal judgment at scale. If that sounds interesting, we’re hiring at https://crosby.ai/careers.
Acknowledgements
Thank you to Teddi Worledge for her work on the technical implementation, and to Tricia Matibag and Ross Weiser for shaping our datasets.
