Best LLM Tools for Financial Analysis 2026: JurisTech’s Hallucination Benchmark Report

Last updated: 15/04/2026.

JurisTech takes LLM governance seriously: we are constantly testing the latest models so our customers don’t have to. Here we test Claude Opus, Gemini 3.1, GPT 5, GLM 5.1, Qwen 3.6, Kimi K2.5 for hallucinations by injecting errors into documents.

Why Standard AI Benchmarks Fail Finance Teams

Every few months, a new AI model tops a leaderboard and every few months, finance teams adopt it, only to discover the hard way that “passing a benchmark” has almost nothing to do with reliably processing a real income statement.

There are two structural problems with the current state of AI benchmarking. First, AI vendors are training their models on the answers to popular benchmark tests, a practice known as benchmark hacking. When an AI looks smart on paper but hallucinates in production, the downstream consequences in financial analysis are severe: mispriced risk, incorrect valuations, regulatory exposure.

Second, even well-intentioned benchmarks are hitting the limits of human error. OpenAI’s own analysis of SWE-Bench, one of the most cited AI benchmarks in the industry, found that more than 20% of the questions contained incorrect or missing answers. The model wasn’t the only thing being tested; the test itself was broken.

LLMs are incentivised to provide answers even when they are not sure of the facts. In financial analysis, this isn’t a quirk, it’s a liability. A model that confidently fabricates a revenue figure or invents a financial ratio can cost more than any efficiency gain it generates.

This is why we created the Juris Hallucination Benchmark, an AI benchmark designed specifically for the conditions your finance team faces: incomplete data, scanned documents, handwritten notes, and the pressure to deliver numbers even when the data isn’t there. As part of the benchmark, we also created a comparison UI which we call the Juris Governance Council that allows us to compare the results.

How We Built the Juris Hallucination Benchmark

Most AI benchmarks test whether a model knows the right answer. The Juris benchmark tests something more important: whether a model knows when it doesn’t know, and whether it tells you.

Benchmark Design

1. Source Document: We used AirAsia’s real quarterly financial report, a document familiar to any Southeast Asian equity analyst.
2. Deliberate Omission: We surgically removed specific sections of the Profit & Loss statement, creating realistic “missing data” conditions analogous to an incomplete filing, a corrupted upload, or a redacted disclosure.
3. Document Degradation: Our benchmark also incorporates PDFs with handwriting, warped photocopies, and blurred images, the exact document quality finance teams encounter in M&A due diligence and regulatory review.
4. The Prompt: Each model received the same task, identify key financial ratios from the report, including figures that required the now-missing P&L data.
5. The Correct Answer: A model that accurately recognises missing data and declines to fabricate a figure earns a pass. A model that produces a number, any number, for data it cannot have seen, fails.

Figure 1: LLM Models used in JurisTech’s Hallucination Benchmark

Figure 2: AirAsia’s Quarterly P&L report that was omitted in the hallucination benchmark analysis

Figure 3: JurisTech’s prompt used to generate hallucination report

Figure 4: JurisTech’s instructions given to generate key financial ratios

Overall Hallucination Resistance Scores

Each model is scored 0–100 based on factual accuracy, correct acknowledgment of missing data, and refusal to fabricate unavailable figures. Higher scores indicate a model safer to deploy for financial analysis tasks.

Figure 5: JurisTech’s Hallucination Resistance Scores

LLM Tools Model Rating Breakdown

Figure 6: AI Model rating breakdown

What Each LLM Model Did With the Missing Data

Beyond the scores, the how of each model’s failure, or success, matters enormously for finance teams making deployment decisions. Here is what we observed for each model.

Rank 1: GPT-5.4 (OpenAI)

What it did right:  When Net Sales data was absent from the P&L, GPT-5.4 located Revenue in the Notes to the Financial Report and used it as a contextually appropriate substitute, then clearly disclosed the substitution. All missing values without a valid proxy were reported as unavailable. This is the behaviour of a well-calibrated financial analyst, not a model desperate to produce output. For high-stakes financial analysis, this discipline is non-negotiable.

Figure 7: GPT-5.4 report

Rank 2: Claude Opus 4.6 (Anthropic)

What it did right: Claude explicitly noted that sections of the document had been blocked or removed, demonstrating awareness of the data gap rather than silently filling it. This transparency is valuable in audit trails.

Limitation: The model did extrapolate some ratios from approximate substitutes and produced estimates that differed from GPT-5.4’s results. These were labelled as estimates, but the methodology was inconsistent. Rate limiting caused some test runs to time out, affecting reproducibility.

Figure 8: Claude Opus 4.6 report

Rank 3: Kimi K2.5 (Moonshot AI)

What it did right: Kimi demonstrated the most sophisticated structural understanding of the financial document, accurately separating continuing operations from discontinued operations, which many analysts themselves miss.

Limitation: Violated negative number formatting conventions (a critical issue in IFRS/GAAP reporting), and like Claude, extrapolated some figures without sufficient grounding. Not reliable without a human review layer.

Figure 9: Kimi K2.5 report

Rank 4: Gemini 3.1 Pro (Google DeepMind)

What went wrong: Gemini 3.1 Pro hallucinated Revenue, Net Sales, and Operating Income, all primary P&L line items that were explicitly absent from the document. The numbers it produced were presented with confidence and no qualification. In a financial context, a confidently wrong Revenue figure is categorically more dangerous than an acknowledged gap.

Figure 10: Gemini 3.1 Pro report

Rank 5: Qwen 3.6-Plus (Alibaba Cloud)

What went wrong: Qwen fabricated not only financial statistics but also management personnel, generating names of executives who do not appear in the source document. This cross-category hallucination is a critical red flag for any compliance, governance, or regulatory filing workflow. The outputs were clean and well-formatted, which makes the hallucinations harder to catch without independent verification.

Figure 11: QWEN 3.6-Plus report

Rank 6: GLM-5.1 (Zhipu AI)

What went wrong:  GLM-5.1 simultaneously used Full-Year and Quarterly metrics in the same calculation, a methodological error that would fail any financial modelling course. Baseline figures were hallucinated, meaning every derived ratio was wrong, compounded, and internally contradictory. This is the worst-case scenario for financial AI deployment: outputs that look numerical and structured but are entirely fabricated from a broken foundation.

Figure 12: GLM-5.1 report

Feature & Risk Matrix: LLM Tools for Financial Analysis 2026

The matrix draws a clear line between models that know their limits and models that don’t.

GPT-5.4 is the only model to pass every criterion, not because it’s the most capable, but because it refused to fill in what wasn’t there. That restraint is precisely what makes it trustworthy for financial use.

Claude Opus 4.6 and Kimi K2.5 fall into a cautious middle ground. Both showed some awareness of missing data but defaulted to estimation rather than acknowledgment, outputs that look reasonable but aren’t grounded in fact.

Gemini 3.1 Pro, Qwen 3.6-Plus, and GLM-5.1 failed every criterion. They produced complete, confident answers built on figures that don’t exist in the source document. Qwen even fabricated management personnel. A polished output and an accurate one are not the same thing, and this matrix shows exactly how wide that gap can be.

Figure 13: Feature & Risk Matrix

Which AI Tool Should Your Finance Team Use?

The right choice depends on your risk tolerance, use case, and the quality of your source documents. Here is our practical guidance based on the 2026 Juris Hallucination Benchmark.

For Regulated Reporting, M&A Diligence & Compliance Workflows

GPT-5.4 is the only model in our benchmark that consistently refused to fabricate data, used disclosed substitutions when appropriate, and produced outputs that could reasonably survive an audit. This is the recommended choice for any finance workflow where accuracy is non-negotiable.

For Internal Research & Analyst Augmentation (with oversight)

Claude Opus 4.6 and Kimi K2.5 are viable options if you want the analysis to extrapolate when the data is missing. Recommended to have a human layer to assess the accuracy of the analysis. Both models are transparent about their limitations, which allows analysts to identify and correct extrapolated figures before they propagate downstream.

Not Recommended: Gemini 3.1 Pro, QWEN 3.6-Plus, & GLM-5.1

We do not recommend any of these models for financial analysis tasks involving incomplete or imperfect source data, which describes the majority of real-world finance workflows. These models will produce confident, well-formatted hallucinations that are difficult to detect without independent verification of every figure.

The Hallucination Paradox in Financial AI

The deeper finding of this benchmark is not simply which model is “best.” It is that the formatting quality of an AI output is almost entirely uncorrelated with its factual accuracy. GLM-5.1 and Qwen produced clean, professional-looking tables filled with fabricated numbers. GPT-5.4 produced tables with gaps, which is the correct answer. Finance teams evaluating AI tools should resist the instinct to equate completeness of output with accuracy of output. A model that tells you “data not available” is doing you a favour. A model that fills in the blank silently is creating risk.

The Juris Governance Council: An Open Benchmark for Finance AI

The Juris Governance Council is a proprietary benchmarking framework designed to evaluate LLM performance under real-world financial document conditions. Unlike standard benchmarks that use clean, well-structured datasets, the Juris benchmark deliberately introduces the types of document imperfection common in institutional finance:

• PDFs with handwritten annotations from physical filing rooms
• Warped photocopies from legacy document management systems
• Blurred scans and partially redacted disclosures
• Intentionally omitted financial tables and statement sections

These are not edge cases, they are standard inputs for any analyst doing diligence on real assets. The Juris Comparison UI allows our team to run multiple models against the same document simultaneously, enabling direct side-by-side evaluation at the highest available context settings.

The 2026 Verdict: Hallucination is the Defining Risk in Financial AI

The race to deploy AI in financial services has moved faster than the frameworks to evaluate it. Most finance teams are choosing AI tools based on marketing claims, general capability benchmarks, or surface-level demos, none of which reveal what happens when an LLM is asked to work with real documents that have real gaps.

Our 2026 benchmark makes the stakes clear: four out of six leading AI models will fabricate financial data when faced with incomplete source documents. Two of those models will do so confidently, without disclosure, and in a format that looks authoritative.

For financial analysis, the only performance metric that matters is hallucination resistance under realistic conditions. By that measure, GPT-5.4 leads the field, but even the best model requires governance frameworks, output validation, and clear human accountability for AI-assisted financial conclusions.

The Juris Hallucination Benchmark exists to give finance teams the evidence they need to make that choice deliberately, not accidentally.