logoalt Hacker News

AI agents ask better questions than KPMG accountants in audit trial

98 points • by Betelbuddy • today at 3:49 PM • 52 comments • view on HN

Comments

prometheus1992 • today at 6:25 PM

READ TFA- The experiment is conducted by KPMG itself. The result of this experiment is the equivalent of 'most of the code in faang today is written by AI'.

The task of auditing a business is extremely complicated by virtue of data/information being all over the place. AI had not gotten to a point of where it could outperform humans but in this experiment they find out a group of agents are good enough for this task. I still think there are so many poorly defined tasks that auditors just understand implicitly, eg data discovery; before the audit even starts, that humans will always be needed to initiate and conclude audits just like software projects.

➕ show 2 replies
trentnix • today at 6:14 PM

KMPG accountants are generally inexperienced, overworked, and poorly incentivized. The entire big firm public accounting model is exploitative at multiple levels.

origini • today at 5:21 PM

Even in normal jobs we have such a HUGE difference between two people. The only organization which is working against are pilots with procedures and checklists.

And every year a new human comes along and has to learn all of it again.

Of course ONE AI agent and teaching ONE AI is massivly in an advantage as soon as its as good as the avg person in the job and it blows everyone else out of the water as soon as it gets a tag better.

➕ show 2 replies
chilmers • today at 5:38 PM

I think properly-trained AI agents will have a huge advantage against humans in such contexts: They will have no intrinsic tendency towards laziness. I think most humans have a natural inclination to optimise the amount of work we do in a professional setting, unless we really happen to enjoy it doing it for the sake of it. We can fight against this by setting up processes and incentives, but we'll always be going against a tendency deeply ingrained by evolution to minimise our mental energy expenditure and to avoid repetitive, tedious or boring tasks.

On the other hand, with AI we'll use reinforcement learning to create agents without such evolved-in tendencies. Where (so long as we can solve alignment and cheating) they are trained to do the job, no matter how much effort it takes, or what's involved.

➕ show 1 reply
chirau • today at 5:58 PM

I am not surprised at all.

AI labs were working with companies like Mercor for more than a year paying the consultants and accountants themselves from these very firms a good amount per typical task they work on everyday, some even providing actual documents.

The expectations were more rigorous, and the rubrics pretty expansive. so yeah, the agent trained from tens of thousands of consultants would eventually ask better questions than an individual consultant.

intended • today at 6:41 PM

Im quite sceptical of articles making claims, as opposed to data that explains the productivity over a large enough scope.

OUTSIDE OF TECH: From what I am personally seeing, there are more incremental improvements in productivity than step changes in firms.

For the degree of hype, we would expect more step changes.

For the actual capability of these tools, it should be more incremental changes.

The greatest determinant of a step change in productivity is the presence of 3 things, typically in one person.

1) Domain expertise

2) project expertise/knowledge

3) familiarty with AI tooling

All of which are useful because this allows the operator to identify when a session is going of the rails quickly, discard the output, and switch to a better prompt.

Otherwise, the productivity gains are incremental, and arise out of capability expansion in doing early career work. i.e., AI removes the communication friction of reaching out to another firm or team.

There are going to be MANY counter examples and anecdotes on how this is not true. Good counter examples are people using LLMs to assist with legal issues or handle translation. This is usually within the boundaries of early stage work, which the operator has enough experience and common sense to understand.

Another boundary is review - people are getting spammed from inside now, because the cost/ease of generation has gone down, while verification capacity has not kept pace. The ratio of availble V/G vs required V/G for a process depends on how mission critical that process is. Most firms are stuck in a hype driven cycle to just produce more, and review is not a valued part of the process.

I expect that this is going to change as soon as people start asking who is responsible and accountable for things going wrong. Velocity is great as long as you don't break someone else's things.

I am going to regret writing this.

joe_the_user • today at 6:08 PM

I think AIs can ask good questions and lead a person to important insights.

But they can also fail to ask "obvious" questions and get stuck at points that an average person wouldn't be stuck in.

Which makes them for adversarial processes - like auditing or hacking. But no so good at producing defect-free software or planning without guidance. With guidance, they're great aids in this too. But when people AI processes to "turn off their brains" or act autonomously, then you get problems. And you get a lot of pressure to very lightly if at all scan AI output when AI investment is supposed to show productivity gains.

jordanb • today at 5:43 PM

KPMG's nickname in Germany is Kinder Prüfen Meine Gesellschaft (children auditing my company). Not exactly a stamp of excellence. The KPMG folks were probably also asking questions they got from gpt.

➕ show 1 reply
shye • today at 5:04 PM

I suspect this says more about KPMG accountants than AI.

➕ show 1 reply
letmeguessok • today at 6:09 PM

Let me guess.. auditing and doing “accounting” never was the “hard part” anyway?

pembrook • today at 5:37 PM

Is anybody surprised that AI can do a better job in a field that is literally just counting beans and reading written down rules and rulings (laws, precedent) and following them during the process of bean counting?

➕ show 3 replies
bell-cot • today at 5:11 PM

From coverage of a fair number of accounting scandals over the decades, my understanding is that a "Big Four" audit of a major enterprise is too often a cut-rate whitewashing exercise - the auditors overlook all the dubious accounting, and in exchange their firm gets various lucrative consulting contracts and such.

Most famously - https://en.wikipedia.org/wiki/Enron_scandal

Is anyone in this space, to speak to whether AI agents asking "better" questions is a benefit, vs. a bad product-market fit?

➕ show 3 replies
redwood • today at 5:53 PM

What this means is we Face a future in which instead of a sample of accounts auditing moves into being a comprehensive assessment. That's what near free at scale Brute Force enables that humans simply could never do.

There are many downsides to this in that business cannot ever be perfect and a sample-based regime is useful because it allows for a few mistakes here and there it's not waste everyone's time at audit time

So similar to cybersecurity where you're going to need to apply AI offensively and defensively I suspect we'll see the same in accounting

➕ show 1 reply
jdw64 • today at 5:36 PM

In my country, Korea, a perfect score was achieved on a tax-related exam that even practicing tax professionals find difficult to score perfectly on—and it turned out that AI had been used. Regulations on AI cheating are now being drafted in Korea. Looking at this article, at what I’ve seen and felt around me, and at similar problems in other East Asian countries, I often think that AI is better than humans in areas where specific procedures are fixed. In other words, it feels stronger than humans in problems where the search space is defined.

j45 • today at 5:09 PM

Given AI's tendency to start with the average and most common questions, I wonder if that's what's outperforming accountants and management accountants.

The accountants will not have this ability to use AI because their fantasy of managing and overseeing IT wasn't able to be handled.

➕ show 1 reply