At Crosby, we want our agentic harness to help lawyers review contracts quickly while maintaining an extremely high quality bar. In support of this goal, our research team has been post-training models to improve surgicality: the precise, targeted nature of an edit that minimizes cognitive load for attorneys while maintaining its correct legal substance. We worked closely with Crosby's attorneys to define surgical standards, ensure they’re represented in our data, and translate them into verifiable rewards. We would like to introduce our first post-trained model adapted for surgical reasoning, which outperforms frontier language models on our internal benchmark dataset at a fraction of the cost.
Why is surgicality challenging?
When redlining a contract, there are typically multiple ways of achieving the desired text with varying degrees of surgicality. Consider the following example:
Input document: “Either party may terminate the agreement after thirty (30) days.”
Desired redline: “The counterparty may terminate the Agreement after ten (10) days”.
Targeted edits: 39 characters changed, 6 tracked changes.
Of the options in the example above, which one is the most surgical? There are a few factors to take into account; most importantly, whether the redline offers a clear view of each proposed change, side by side. But other considerations, like the number of mouse clicks required to create or resolve the tracked changes, or the extra complexity that comes with comparing many edits, sometimes pull in the opposite direction. Moreover, lawyers take pride in finding elegant solutions to surgicality problems in ways that can’t always be codified– making incisions that fall cleanly alongside the existing punctuation, carving out redundant words, and altering the document without disrupting, for lack of another word, its flow. Some lawyers are only half joking when they call surgicality an art form, and that’s why modeling it is so hard.
Surgicality is an important concept for the AI research team at Crosby because we want our agentic harness to help lawyers review contracts quickly, without decreasing the quality of their work. Both of those words are tracked by a metric: “quickly” is measured by HuRT (human review time), and “quality” is measured by CSAT (client satisfaction). Our goal is to decrease HuRT while maintaining exceptional CSAT scores. If our system proposes non-surgical redlines, one of the two metrics will suffer: either our own attorneys take on the burden of fixing the redlines themselves (HuRT up), or they send untidy redlines to the client/counterparty (CSAT down). We estimate that over a quarter of our harness’ suggested redlines are semantically correct, but not surgical. Getting that number as close to zero as possible will chip off a large block of lawyers’ review time, and help pave the way towards automation.
Codifying the desirable properties of surgical edits isn’t easy. If they could be exhaustively specified, then we could provide a language model with a list of rules, or simpler yet, produce the optimal redline directly via code. Levenshtein distance (character insertion/deletion count) can be used as a heuristic approximation of whether a redline’s edits are precise, for instance– but the example above illustrates why minimal edits aren’t always more surgical. Option 4 achieves a lower Levenshtein distance than option 3 (23 vs. 39); yet it produces more tracked changes, and edits the numbers in a way that’s less readable.
The alternative to a heuristic-based approach is to let a model learn about surgicality through post-training. Then, accuracy is limited by the inability to extract consistent training labels. Lawyers often disagree on whether a redline is surgical or not (is it better to strike through an entire number, or just the relevant digits?), making alignment with all attorneys’ preferences an issue. Label disagreement is also a challenge for us, as AI researchers. Small but inaccurate mark-ups, like a redundant whitespace deletion and re-insertion, matter more to attorneys than they do to most reward functions. Training a model requires AI researchers and lawyers to work in tandem to define surgical standards, and translate them into verifiable reward signals.
To our knowledge, the legal AI community has yet to make much public progress on the task. A few conceptually similar research areas are growing in other fields, like precise code repair, but the overall landscape is relatively barren. We hope to help populate the space by describing our approach to post-training, sharing results, and establishing a pareto frontier.
Designing a surgical verifier
Our last blog post about verifier models shared a recipe for converting open ended questions about quality into tasks a model can learn. We followed the same process here, while addressing the issues discussed over there. Here’s a side-by-side comparison of the two approaches:
| Training design | Legal judgment verifier | Surgicality verifier |
|---|---|---|
| Training data | Harness traces | Harness traces + synthetic examples |
| Test data labels | Programmatically generated | Human generated |
| Model architecture | 27B-parameter language decoder | 27B-parameter language decoder |
| Model output | Binary prediction score | Thinking trace + yes/no verdict |
| Training algorithm | Supervised finetuning (SFT) on binary labels | Autoregressive SFT + reinforcement learning (RL) |
The key difference is that our surgical verifier is trained to output reasoning alongside its yes/no prediction. The purpose of the binary output is the same in both verifiers: it can be used for offline evaluation, real-time alerts and monitoring, or be part of an auto-acceptance pipeline. The addition of reasoning traces is helpful for a few reasons:
- It allows our researchers to understand failure modes, and builds our attorneys’ and engineers’ trust in the model.
- It steers our agentic harness to automatically produce more surgical redlines in cases where it initially suggests a non-surgical one.
- It improves the binary verdict: letting the model work through each tracked change step by step (known as chain of thought reasoning) increases the likelihood it reaches the correct conclusion.
Below, we go into the details of how model reasoning is enabled through RL, and how synthetic training data improved model performance.
Training Data
As with the legal judgment verifier, we used our agentic system’s suggestions as training data. In order to assign those redlines ground truth labels (surgical=1, non-surgical=0), they were compared against the redlines our attorneys actually submitted for review. We restricted the dataset to cases where the agent’s proposed redline and the attorney’s sent-out-redline produced the same text; in other words, instances where the attorney agreed with the legal substance of the agent’s edit. Within this subset, we assigned a label of 1 to the proposed redlines if and only if they had the same exact tracked changes as the attorney’s sent-out version; we assigned a label of 0 otherwise.
This process generates labels for thousands of redlines without human intervention, but it also results in large class imbalance. More than 90% of the labels were positive, biasing early training runs towards the target class. This comes from the fact that the process described above excludes a significant portion of data; namely, redlines where the agent and lawyer’s plain-text versions don’t match. Some fraction of these dropped cases are valid negative class examples, but would have been impossible to include without human verification.
In our last blog post, we discussed how dataset noise was the main damping force on our model performance. Carrying this lesson over, we addressed label noise in two ways. Training data was augmented with synthetic examples, and validation data was relabelled with the help of Crosby’s attorneys. The latter increased the proportion of negative samples in our validation data to 38% (from 2% initially); we hypothesize that class imbalance was initially large because lawyers reviewing contracts in production are willing to accept a trade-off between review speed and surgicality.
Modeling Surgicality
We trained our model using a two step pipeline: auto-regressive SFT followed by an RL algorithm called group relative policy optimization (GRPO).
1. Supervised autoregressive finetuning: GRPO works by rewarding the model for generating better reasoning traces relative to others. If all the traces generated during a GRPO step are equally poor, the model receives small gradient updates. By warm-starting the GRPO phase with an epoch of SFT first, we can engender desirable reasoning patterns within the model. This improves stability, convergence speed, and reasoning style during the second training phase, and distills the reasoning capabilities of a much larger model into our own.
Our SFT training data was generated using a technique known as rejection sampling. The process follows three steps. First, prompts are gathered: each prompt contains a single redline, and a statement instructing the downstream model to assess whether or not the redline is surgical. Next, a powerful teacher model generates responses to those prompts. As a post-processing step, responses with verdicts contrary to the ground truth label are discarded; in this way, only high quality responses are kept.
During autoregressive SFT, the model is given a prompt, and must try to predict the teacher model’s generated response, using the standard autoregressive training objective. In this manner, it learns the structure, length, and reasoning patterns of the teacher model’s outputs, which improves downstream GRPO performance.
2. Group Relative Policy Optimization: GRPO hinges on having a good reward function. The reward must faithfully measure what we actually care about (in this case: whether an attorney would send the redline as marked), and be cheap enough to compute many times per training step. If the reward drifts from that target, the policy learns to exploit the gap (reward hacking) rather than to get better at the task.
We began by training with a simple binary reward indicating whether the model’s yes/no verdict matched the ground truth label. The policy trained with this reward achieved high balanced accuracy, but its reasoning traces weren’t satisfactory. We improved the reward by using an LLM to assign an additional score to the reasoning trace. This approach is known as using an LLM as a judge. In our reward function, the LLM judge’s score is conditioned on the binary verdict being correct, so that the model is not rewarded for producing overly confident, incorrect answers.
How does our model stack up?
We benchmarked our model against ten frontier language models, comparing them across their balanced accuracy scores on our gold-standard validation dataset. Balanced accuracy measures average recall across both the positive and negative class, making it a good choice in settings where class imbalance is high. All models were scored with the same prompt. We also include a heuristic score as a baseline, which gauges edit minimality and word boundary alignment.
Our model achieves a balanced accuracy of 0.906, outperforming the second best model, Gemini 4 Argon, by 6 balanced accuracy percentage points. Among the closed-weight models, there is a coherent trend between model size and performance: GPT-6 Astra outperforms Sol and Luna; Opus outperforms Fable and Haiku. Gemini 4 Argon’s second-highest score is consistent with their own reported performance on public legal benchmarks compared to Anthropic and OpenAI models, and their emphasis on legal usage in the model’s system card.
In practice, generating accurate predictions and reasoning traces isn’t our only consideration. Given the goal of integrating this verifier into our agentic redline proposal loop, it’s also important for the model to cost a fraction of what a powerful frontier LLM would. We plot all the benchmarked models’ cost, per million tokens, below, on a semi-log scale. Closed-model costs are taken from each provider’s published list prices. Our model offers higher accuracy at roughly a 10-100x lower cost than most private models, forming a pareto frontier alongside Jev.
Given all models received the same prompt, we attribute our model’s performance gains to our post-training procedure. We hypothesize that surgicality verification is a challenging problem that cannot be solved by detailed prompting or providing few-shot examples; instead, improvements on this benchmark will be achieved through improvements in dataset size, diversity, and signal.
Training a model to improve redline surgicality is just one way we can help drive automation in this space. If you’re interested in helping us break this problem down, we’re hiring at https://crosby.ai/careers.
Acknowledgements: we would like to thank Teddi Worledge for her help on the technical implementation, and Leah Yared, Tricia Matibag, Tenzin Dolkar, and Ross Weiser for their legal expertise.
