Tech company Saturn tested free and paid versions of ChatGPT, Claude, Copilot, Grok and Gemini in research reported by the FT, finding that the models gave incorrect answers to financial questions an average of 57% of the time. The error rate rose to 88% for complex queries involving multiple calculations, highlighting the potential financial consequences of relying on generative AI for tax, pension and investment decisions.

AI models struggled with complex financial questions

Researchers tested more than 100 money-related questions across 18 models. They submitted more than 10,000 prompts in total, with each question repeated up to five times.

The study identified errors in financial calculations, failures to account for future changes to tax legislation, made-up rules and incorrect responses to complex questions requiring multiple calculations.

For some models, the error rate on complex questions reached 99%. Claude Opus 5 delivered the best result when used in “reasoning” mode, but it still answered 39% of questions incorrectly.

Amal Jolly, Saturn’s CEO, said widespread use of AI for financial guidance created additional risks.

“Millions of people are trusting the AI models for money advice, but they are getting wrong answers that can lose them money,” he said.

Errors could carry substantial costs

In one test, researchers asked the free Claude Haiku 4.5 model about pension taxation. According to the study, its error could have resulted in an HMRC charge of £17,500, or about $23,430, for a pension saver.

In another case, Claude “made up a rule” about student loans and claimed that a graduate could stop making payments after moving abroad.

Paid models were generally more accurate than free versions, while newer models produced better results. Sarah Coles, head of personal finance at investment platform AJ Bell, said AI could still help users with budgeting and finding information.

“It’s a good example of why people who need support need proper advice rather than trusting AI,” she said.

The findings align with a recent study by the UK Financial Conduct Authority, which found that one in five UK adults was willing to let AI make financial decisions on their behalf. Interest was strongest in complex areas including debt, pensions and investments.

Jolly said AI financial advice was currently unregulated, leaving users without the same protections and compensation available when they consult a human professional.

OpenAI recently launched ChatGPT for the financial sector.

Source: Incrypted