February 1, 2025
BC's #a2jweek, from February 3 to 7, explores "What does the future of access to justice look like?" From where we sit, that future includes opportunities enabled by generative AI to scale legal help. We're seeing this taking shape every day with Beagle+ , our generative AI-powered chatbot, answering questions and guiding British Columbians through their everyday legal problems — no lines to wait in, no appointment necessary.
Call it "on-demand a2j." That's what the future looks like 🔮
We're recognized as a leader in using generative AI to increase A2J
We went live with Beagle+ in February 2024. It was one of the first legal chatbots to be powered by generative artificial intelligence. A year on, having helped over 6,000 British Columbians navigate legal problems, Beagle+ is recognized as a leading example of a GenAI-powered chatbot.
In January, we were invited to share our Beagle+ journey with the National Center for State Courts, an organization dedicated to driving innovation and progress in the court system in the United States. The webinar, Applications of AI to Increase Access to Justice, was attended by 856 people. The feedback has been fantastic. In a companion article, the webinar organizers wrote:
"Central to the success of Beagle+ is its thoughtful design and user-centric approach. The team prioritized creating a system that is both empathetic and informative with a primary focus of providing users with clear, actionable guidance."
A sampling of the audience feedback:
“Thank you for hosting such an informative session. The real-world examples of generative AI in legal contexts were particularly helpful, and I appreciated the tips on structuring content to reduce hallucinations.”
“I really liked the discussion of what instructions/prompts you used for Beagle+ to push it toward your needs and correct issues. Excellent work!”
Also in January, we presented at the annual conference on innovations in technology hosted by the Legal Services Corporation, the umbrella agency for legal aid delivery in the US. This year’s gathering in Phoenix was attended by 700+ folks working on access to justice across North America. Our workshop, Generative AI Using Your Content, was the most viewed livestream at the conference.
And again, the feedback has been fantastic. For example, Sam Harden, Senior Innovation Manager at Pro Bono Net, posted on his substack:
"The folks at the People’s Law School in British Columbia came and gave a presentation at the main ITC conference about Beagle+, the AI bot they have rolled out. I was really impressed by their practical approach, especially their willingness to test and evaluate the bot’s performance. Also they have been running QA on the bot’s responses and documenting it — this will be really valuable for other organizations that want to do something similar. Granted most orgs won’t have the people power to do as extensive QA, but they can point to Beagle’s results as a type of level-setting."
We set high standards before giving access to the public
Let's talk more about our approach to quality assurance. The foundational layer is retrieval-augmented generation (RAG). That's a fancy term for saying Beagle+ uses our own People's Law School content to provide its responses. We instruct ChatGPT 4o, the model that powers Beagle+, to use the content from our websites — and if we don't have any relevant content, to say so.
In our development journey, we ran a set of 42 test questions through dozens of iterations of Beagle+, trying different ChatGPT models and settings. We found the introduction of GPT-4 offered a big step forward, as it was much more disciplined in not going beyond our content. And we continued to refine our system prompt to make it even more disciplined.
Until we were very confident with Beagle+’s ability to answer legal questions accurately, we held back on introducing it to the public.
🙅♂️ April 2023: 70% accuracy; no launch yet
🙅♀️ October 2023: 80% accuracy; no launch yet
🤔 December 2023: 90% accuracy; still no launch yet
😄 February 2024: 99% accuracy; public launch
This took a lot of testing and refinement. But we weren’t comfortable to ship our product until it was ready.
We constantly review Beagle+ conversations, and make further adjustments as needed
From the day we launched, we’ve reviewed every conversation the public has had with Beagle+. Our content development lawyers have judged 99% of the conversations to be legally accurate. And our review is not just about legal accuracy; we’re looking at tone, helpfulness, and the language used to ensure that Beagle+ can empower our users to take a next step in resolving or avoiding a legal issue.
With the 1% of conversations that our team has flagged as ones in need of improvement, we've been able to address every single one. With roughly half, we made tweaks or additions to the underlying content, which resulted in excellent responses when we reran the same inputs afterwards. With the other half, we made further refinements to our system prompt to address the issue; for example, we now instruct Beagle+ to ask a question clarifying what level of court is involved when the context doesn't make that clear.
We're encouraged by the results we're seeing. Users can provide feedback on whether Beagle+ was helpful (via a 👍 or 👎). In 2024, of the 6,000+ people who used Beagle+, 79% said it was helpful. Compare that to Statistic Canada’s Canadian Legal Problems Survey, where people found legal professionals to be helpful 77% of the time.
Stay tuned as Beagle+ expands its reach and ability to fetch
The future of legal technology is bright, and People’s Law is proud to be part of the movement to enhance access to justice.
The US National Center for State Courts has invited us back to do a part 2 webinar on February 19. We'll dive deeper into the technical aspects of building and monitoring Beagle+, including using retrieval-augmented generation as well as an LLM (large language model) engineering platform to help in the review of chatbot conversations.
In addition, we’re in conversations with Stanford Law School's Legal Design Lab to work together on establishing performance benchmarks for what makes for a high quality conversational chatbot experience in the legal space — which would further improve the user experience of Beagle+ and support other non-profits in using generative AI-powered tools in ways that are safe and empowering for people seeking to access justice.