Benchmarks and Legal LLMs: Where are we today? Link to heading

Highlight: On the AIR-Bench 2024 benchmark, the first AI safety benchmark aligned with government regulations, Claude Haiku 4.5, Anthropic’s low-end model has the highest score of 93% correct compared to all other models tested, a result that I reproduced here.

Project Context Link to heading

The BlueDot Impact Technical AI Safety Project Sprint is a 5-week facilitated sprint with the end goal of making a contribution to AI Safety research or engineering.

For my project, I planned to replicate and extend a paper, Safety Compliance: Rethinking LLM Safety Reasoning Through the Lens of Compliance, which went through the process of creating a benchmark, creating and training legal LLMs and running that benchmark on general-purpose and safety models for comparison. This project was both a success and a failure in that I went above and beyond in accomplishing my personal goals, but failed to replicate the full project, due to a lack of resources.

Terminology Link to heading

• LLM/model: A large language model. This is a type of AI that has been trained on billions of words which then allows it to create sentences and conversations by predicting what word should come next. I generally use the words LLM and model interchangably in this project.
• local model: A model that I run locally on my personal machine using a tool called ollama.
• frontier model: A model that is hosted by someone else and I have to access programatically through an API (Application Programming Interface) or a webpage with chat functionality.
• legal model: A model that has been specifically trained on laws and legal documentation meant to improve the quality of its legal and regulatory responses.
• safety model: A model that has been trained to be more conservative in its responses, self-filtering and generally providing safer outputs.
• general-purpose model: A model that has been trained on vast quantities of data, but without any specific purpose.

Summary Link to heading

Benchmark results from the original paper, my investigation here and other sources, show that we still have a long way to go to have reliable legal LLMs. LLMs varied widely in their ability to understand nuanced approaches to legislation: in multiple cases, the same LLM would review a scenario several times and contradict its original results and reasoning. On AIR-Bench 2024, modern LLMs showed a wide variety of results spanning from 58% to 93%. Interestingly, the 93% result was on one of the low-end frontier models (Claude Haiku). The worst frontier model had scores of only 10% higher than the best local model which indicates that it’s possible that as local models improve, their scores will eventually meet those of current-day frontier models.

Why: My personal goals entering the project Link to heading

My goal when entering this project was to answer 4 questions:

1. Do I want to pursue technical AI research?
2. Do I want to pursue software development?
3. What research exists in the Technical AI Governance space and how can I use it?
4. What kind of skills would be useful in all of the above tracks?

What I chose and how it worked Link to heading

My original plan was to replicate and extend a paper, Safety Compliance: Rethinking LLM Safety Reasoning Through the Lens of Compliance.

This paper makes a case for training a safety+legal LLM which is an expert in law. In this case, the focus was on EU AI Act and GDPR. My plan was to replicate this and then extend it to include the California AI Transparency Act.

Safety Compliance went through 3 phases to create and test this LLM:

1. Create a benchmark to test legal knowledge.
2. Create and train a custom legal+safety LLM.
3. Run the new benchmark on the new LLM(s) and compare it to results when run on other models.

What I Did (and Didn’t) Do Link to heading

I ran into a lot of surprises as I went through the process and used this as an opportunity to constantly adapt, while still aiming to achieve my original goals.

Creating the benchmark - what they did vs. what I did Link to heading

Challenge: Source code and data wasn’t available and the original authors didn’t reply with requests for access.
Solution: Create the datasets myself.

Challenge: I used Claude to generate a single scenario with the purpose of testing quality. I had a legal expert review it, who determined that the scenario wasn’t legally accurate enough to justify burning tokens to create the full 2500 test scenarios.
Solution: Manually run that single case on local and frontier models and compare the LLM feedback with that of a human lawyer. Use a pre-existing legal benchmark instead and compare my results with already published results.

Training New Models - what they did vs. what I did Link to heading

Challenge: The paper used a separate dataset for training that wasn’t clearly defined.
Solution: In addition to their own dataset, they also trained with some existing safety datasets which were publicly available. I would simply use these and leave out their custom dataset.

Challenge: Insufficient resources for full training process.
Solution: Use LoRA to create an adapter that works along side a base model.

Challenge: Existing safety models and custom-trained models didn’t produce expected output.
Solution: Limit scope to models that produced coherent output.

Challenge: Initial calculations estimated 10+ days to run 10 epochs of SFT+GRPO using all available datasets.
Solution: Train 1 epoch of one dataset of SFT and GRPO. (8.5 hr)

Benchmarking - what they did vs. what I did Link to heading

Results Link to heading

Due to the fact that I didn’t trust the quality of the scenario that I generated, but I had an excellent legal review of the nuance, I decided that I would test various models on the scenario that I generated and compare the results to the scenario that was in the original paper for the same clause.

I detailed the process I used to get my single scenario and the prompts I used for testing the models.

The expected responses for my new scenario was probably not permitted, so the answer should be prohibited with lower confidence rating and a reasoning that explains the nuance.

The expected response for the original scenario was listed as permitted, so that’s the response we expect there.

I have combined the results into a single table for easier readability. New scenario is the scenario that I generated, original scenario is from the original paper.

Confidence rating reflects how confident it is in its answer. This can range from 0 showing absolutely no confidence in its rating to 10, being perfectly certain that the answer is correct.

Results below in green are interesting because they are the only ones that correctly described the correct answer in my scenario of probably not permitted and the answers in red are interesting because a DeepSeek model was the creator of the scenario from the original paper and yet models within the same family still judged it as being prohibited.

Air Bench Benchmark

Scores below reflect overall average of questions correctly answered. Scores for individual questions can be 0 (wrong), 0.5 (either the final answer or logic behind it is wrong) or 1 (completely correct answer and logic).

The most surprising result here is that Haiku is the winner here, despite being a “low-end” frontier model.

Notes:

1. Stanford’s latest results are from 2024, so a lot of models that they tested on are no longer available today. I tried to get models as close as possible for comparison.

What I Learned Link to heading

• I like applied research. Throughout this project, I was constantly thinking of ways we could use this process in AI safety. I kept going back to ideas for products and processes that could use it. I would be frustrated doing pure research because I want to see the results of the research in the real world.
• I also enjoyed the development side of things, but I continually had to redirect myself from designing AI Compliance systems that used this work in my head. I really need something more high-level that looks at compliance systems which I can then delegate dev work for. (This is what I did in my last job.)
• There are a lot of excellent technical+legal AI safety ideas out there that could be used in the compliance space. During this project I collected an additional 22 papers that I want to read that have more ideas.
• This project was incredibly useful and I met all of my personal goals. I now have more insight into the research process, the technologies used behind the scenes in AI software development, including setting up local models, configuring frontier models access + API usage, basic model training and setting up and using the Inspect framework.
• I’m disappointed that I didn’t actually fully repeat the process of the original paper but I think that I learned more from the steps that I went through than I would have if I had had access to all of their data and code and just reproduced it.

Possible Next Steps Link to heading

• Investigate benchmark scenario creation. Work with lawyers to do more in-depth review of the scenarios and judge quality between various models. If we want to use this type of process, we probably need to add in some RLHF with legal experts to get good scenarios. This would be expensive.
• Investigate why my models consistently got stuck in repetitious loops after training. Suspicion: training data too easy.
• Test the impact of using different prompts. How does prompt creation impact results? A lot of study has been done on this so I’m very interested in learning more about this.
• Test breaking down this process and creating more dedicated LLMs that each has its own purpose. Does this improve the results?
• Repeat the tests (many) more times. Are the results consistent? How much do they vary if we run each test 10 or more times?
• Try to get in contact with researchers again for clarification about what they used for training vs testing data to better understand how they created their training data.
• Some models didn’t produce valid output when being tested, yet they have results in other places and so should run. Investigate how to get correct output. This was specifically some Qwen versions due to the structure of its thinking output and the safety models default to reporting whether the input is safe or not, not actually solving the tasks.
• Log a bug with Inspect framework because they have a bug with reporting final results.
  • Even better! Fix the bug and submit it myself!
• Take an existing benchmark that doesn’t exist in Inspect and convert it so that it can run in Inspect. Submit to be added.
• My experience in dev + program mgmt helped me stay organised and was a huge benefit. I am thinking about putting something together that BDI could provide to people without the experience. This could consist of:
  1. A project outline that someone could follow to learn the basics, especially if they’re new to development or are from the governance side and just want to go through the process of setting up one of these environments so they understand what they’re trying to govern.
  2. How to do iterative work. Start small, reiterate. This caught me up in the first week and could have saved me a lot of pain.

References Link to heading

BlueDot Impact Technical AI Safety Project Sprint
Original project plan
My code repo - this site includes all information related to the project, including this page
UK AISI Inspect Framework

Wenbin Hu, Huihao Jing, Haochen Shi, Haoran Li, and Yangqiu Song. Safety Compliance: Rethinking LLM Safety Reasoning through the Lens of Compliance, 2025. URL https://arxiv.org/abs/2509.22250v1.

Yi Zeng, Yu Yang, Andy Zhou, Jeffrey Ziwei Tan, Yuheng Tu, Yifan Mai, Kevin Klyman, Minzhou Pan, Ruoxi Jia, Dawn Song, Percy Liang, Bo Li. AIR-Bench 2024: A Safety Benchmark Based on Risk Categories from Regulations and Policies, 2024. URL https://arxiv.org/abs/2407.17436

Acknowledgements Link to heading

I would like to thank the following people for helping me throughout my first six months in AI Safety. They all helped direct and guide me in which direction to go in.

• Johannes Taraz, my BlueDot Impact Technical AI Safety Project Sprint facilitator
• Leticia Prados, a lawyer and EU AI Act practitioner who reviewed my Claude-generated legal scenario
• Jess Bergs, from UK AISI who gave me some advice and ideas about using the Inspect framework
• Sudhanshu Kasewa, my first AI Safety advisor from 80,000 hours
• Justin Shenk, my BlueDot Impact Technical AI Safety facilitator
• Bart Kubiak, my BlueDot Impact AGI Strategy facilitator