AI in finance went mainstream this year, and it made the evidence collection process for audits even more complex. If your company took the leap and decided to put AI into SOX scope, you won't know whether your auditor accepts it until early 2027. And you won't know whether the Public Company Accounting Oversight Board (PCAOB) agrees with your auditor until 2028.
That's how the cycle works. Controllership teams and their audit firms can have conceptual discussions about what might be acceptable, and audit firms are in constant dialogue with the regulator. But the rubber meets the road during the audit and ensuing PCAOB inspection cycle.
For a calendar-year company, your auditor issues its opinion on your 2026 controls in early 2027. The PCAOB inspects the audit firms between roughly March and October 2027, and as feedback and findings trickle in during the inspection cycle, audit partners do what they can to shore up their 2027 audits. The public sees what the PCAOB thought of the 2026 audits when inspection reports come out in 2028. And we’re only seeing the feedback on the public companies that put AI in scope for their controls and happened to be selected, likely a minority of the inspection population. At the pace AI is moving, we may hit the singularity first!
So finance leaders aren't waiting. In June, Financial Executives International's Committee on Corporate Reporting published AI Framework: Internal Control Over Financial Reporting, written by controllers and chief accounting officers from Fortune 500 companies. I see it as the practitioners' first volley to take control of the narrative.
I spent 21 years at EY, the last eight as an audit partner to high-growth tech companies, so I read the whitepaper from the audit chair. It's sound and thoughtful, and perhaps its most important idea is also the easiest to skim past. For traditional systems, auditors get comfortable by understanding the logic. With most AI, you can't. There's no rulebook to read. So the paper shifts the question from how the system reached an answer to whether its answers hold up, through human review, testing against known answers, comparing independent models, and analytics. That's the bridge from how we've always tested applications to how you can test and rely on AI.
Here's where I'd push the framework, and where I think it leaves gaps you'll have to fill yourself.
"Can I test it once and rely on it?"
For an automated control, that's always been the auditor's question. A three-way match in your ERP, or a tax calculation from your billing system or a dedicated tax platform, follows rules someone wrote. Test the logic, confirm through change management that nobody changed it, and you can rely on it all year.
AI breaks that in a way the paper’s vocabulary can hide. The paper describes AI systems as “probabilistic,” “deterministic but complex,” or a mix of both. An AI model learns from examples, and once that training is locked, some models give the same answer every time they see the same input. A model trained to code invoices, for example, will code an identical invoice the same way every time. The paper calls that deterministic but complex. Others, like the large language models behind most AI assistants, have some randomness built in, so the same question can get a different answer. The paper calls those probabilistic. But giving the same answer every time says nothing about whether it’s the right answer. Many finance leaders hear “deterministic” as “reliable” and assume the AI works like traditional software.
We know software is wrong when it has a bug. If someone wrote a rule incorrectly, you can read it, fix it, and find every transaction it touched. An AI model that works by inferring its answer from patterns in its training is wrong some percentage of the time by design, even with clean inputs. A model that gives the same answer every time can be consistently wrong on a certain kind of transaction, and you can't read its logic to predict which kind. You find out by testing.
So you test software to confirm its logic, and you test an AI model to measure its error rate. When whoever provides the model (frontier lab or a software vendor, for example) updates it, the "test once" part is gone. It's like the person who performs a control being replaced by someone with different training: same process and sign-off, different judgment.
A test set fixes that. You can rerun your test after every model change, and compare the answers to a key you trust.
Shadow reliance isn't new, and we already know how to catch it
Pre-AI, what the paper coins shadow reliance already existed. A preparer can be so competent and thorough that the reviewer gets comfortable, and turns the review into an empty sign-off. Good control design guards against this by spelling out what the reviewer has to do each time the control runs and what evidence is sufficient, directly within the control description and related attributes.
The FEI paper worries about the AI version of this. What happens when human nature kicks in, and a reviewer who's supposed to be checking the AI's work stops applying the required level of critical thinking? There's already a pre-AI framework that sets the evidence bar, so a review of an important control that requires judgment doesn't become a rubber stamp.
If a review is objective, like a total in one system matching a total in another, or a reconciliation that never has reconciling items, a sign-off and a date is enough. If it requires judgment, like a complex reconciliation with reconciling items, a flux analysis that depends on knowing the business, or the judgments behind an estimate, the evidence has to show the review was thorough enough for the risk. That means a documented precision level, evidence of the outliers and unusual items identified, assumptions challenged, questions asked, answers received, and follow-up actions taken.
Apply the same bar to AI. Objective AI work, like tie-outs or other simple comparisons, can be evidenced like an objective review. AI output that requires judgment needs the full record of the human's review. The paper points this way when it says human review should be documented like a management review control. I'd make the scale explicit: the more judgment, the more evidence required.
How much testing is enough? The paper doesn't say
The paper expects AI to be tested at least every reporting period, either by having someone recheck a sample of the AI's real work or by running a curated set of transactions with known answers through it. But it never says how many items to test. It points you to existing sampling practice and your auditor.
Instead of assigning a fixed number, I'd fill that gap with a risk assessment. In most cases, AI is operating below the financial statement caption level, and other controls downstream are there to catch errors that add up to something material. In that situation, I'd accept a standard controls sample: 25 to 60 transactions, maybe 90 if volumes were very high during the period or the control is more pervasive to the financial statements.
The sample you can accept depends on what else you're relying on. A clean sample of 25 supports a deviation rate under roughly 10%, and a clean 60, under roughly 5%. That's reasonable when your downstream reviews are precise enough to catch what an error rate like that could produce.
The paper also separates two questions that classic control testing doesn't. In control testing, one exception counts against the control. In the paper's decision tree, an AI error within your tolerance doesn't by itself undermine reliance on the system, though it still gets investigated, and the transaction may still need fixing. Whether that holds up depends on the AI's role. If AI prepares and a human reviewer catches the error, that's the control working. If AI is the control and it misses, that's a deviation.
When the AI gets one sample wrong, find the cluster
If one of those samples came back wrong, I’d want to understand the nature of the item so I could start to form an opinion about why it was a problem for the AI. Is it unique or different from the rest of the population? How many others are there? Would the AI get those wrong too?
Start by ruling out bad source data. Often, though, the data is fine and the AI isn’t broken. The transaction is just something the model wasn’t trained for, like a new vendor, a new kind of contract, or an expense the company has never booked before. Well-designed software can be built to stop and flag an input it can’t handle, like a product with no tax code mapping. An AI model will usually give you an answer anyway, with no sign it’s outside what it was trained on. So one wrong answer usually points to a type of transaction the model handles badly, and it will tend to get others like it wrong too.
If the AI caused the error and it’s outside your tolerance, the paper says to work out how far it reaches, stop relying on the AI for the affected transactions, and ask whether the error points to a control deficiency. Where it’s thin is on how to find and size the affected transactions. Here’s how I’d do it. Pin down what made the item different: a new vendor, an unusual contract term, an account the AI hasn’t seen before. Then pull every transaction in the population that shares that trait, and count them and total the dollars. Recheck a sample from that group to see whether the AI got those wrong too. If it did, you’ve found your cluster and you know how big it is.
This is where audit discipline helps. An auditor treats an error in a sample as representative of the whole population unless there’s strong evidence it isn’t. So if you can’t explain what made the item different, you can’t call it a one-off. Treat it as something that could affect the whole population until more testing says otherwise.
Once you find the cluster, add transactions with that fact pattern to your test set, so you’ll know right away if the weakness comes back after a model update.
Two models agreeing doesn't mean they're right
One of the paper’s control approaches runs the same work through two independent AI models and flags the items where they disagree. It suggests making them independent with different vendors, different versions of a model, different prompts, or separate context, and asks management to show that the two are unlikely to make the same mistake.
That’s the right test, but if two models were trained to do the same job on similar data, couldn't they both miss when a pattern neither was trained on shows up? Research suggests they can. A 2025 study of more than 350 language models and found that when two models were both wrong on one benchmark, they picked the same wrong answer about 60% of the time. Models from the same provider agreed more often, and the more accurate models had highly correlated errors even across providers. That research used benchmark questions and resume screening, not accounting transactions, but the direction is clear.
On simple, black-and-white tasks, two models flubbing the same item is unlikely, so comparing models is good at catching random slips. It's weak against shared blind spots. To show the risk is low, run both models against a test set with known hard cases and count how often they miss the same ones. Treat the items where models disagree as the most valuable output, and don't let agreement replace a test set with known hard cases.
Test sets: pick your holy grail, and check the answer key
The paper's performance testing, running a curated set of transactions with known answers through the AI, is a lot like a walkthrough. When a system, process, or control changes, you walk it through again to confirm it's still designed correctly and working the way you understand it. A test set does the same job for AI: rerun it after every model change and confirm the answers still match the key. There are two differences. You only see what the AI decided, not how, and it can treat two transactions on the same path differently, so you need several variations of each scenario instead of one. And you won't always know when the model changed, so you also rerun it on a regular schedule.
Given how fast models change, even fast-growing companies may want a test set so a vendor update doesn't catch them off guard. But time is limited, so choose which processes are the holy grail: the ones with high risk of material misstatement that need a test set and a review of model cards and release notes. For lower-risk processes, you can punt that decision until you have more resources, or rely on downstream detective controls.
Can AI help build the test set? I think so. Auditors already use software to select samples. AI picking transactions, some at random and some with errors designed in, is fine if a human reviews the set and agrees it’s appropriate. Include every exception cluster you’ve found, plus a control set of normal transactions. What needs a human is the answer key: the correct outcome for each test transaction. For the errors you planted, you already know the answer. For everything else, a person has to confirm it.
What's still open
There's a lot to be determined. The American Institute of Certified Public Accountants recently published its own AI practice aid for auditors. That guide, and any others that follow, will take time to digest and implement on the fly, and it doesn't contemplate everything auditors will see in the field in the coming months. Experience tells us we will get our first indicators of what auditors accept this year's audits are done, with a PCAOB view of those audits after they complete their inspection cycle. Early alignment with your auditor is never a bad idea, and it matters more than usual here, because FEI's paper is a management framework, not an auditing standard. Your auditor isn't bound by it.
What I'm confident about is that none of this takes experienced people out of the loop. Even with everything in this paper working, you need humans who understand the business reviewing the work and monitoring the systems. More of the job becomes review, and less of it is the tedious work.
There's a payoff if we get this right. Finance teams are already chasing a zero-day or continuous close, and the natural companion is an audit that keeps pace. Much of what makes the audit painful is reconstructing evidence after the fact. If AI flags exceptions as they happen and reviewers document their questions and follow-up in real time, much of the audit work can move into the year. We're not at an audit that's done on January 1st yet, but that's the direction worth building toward.
.png)
.png)
.png)