A controller's questions for evaluating AI accounting software
By Mara Voss · October 24, 2026
Category: competitive-landscape
Demos show the clean path. These questions force a vendor to show the exceptions, the limits and the logs, which is where AI accounting software succeeds or fails.
Key takeaways
The problem Demos run on clean data, so every product looks accurate and every vendor sounds the same.
Core insight The honest signal comes from three things a demo avoids: your worst documents, the product's refusals, and the record it leaves behind.
Practical outcome Run a paid pilot on your own exceptions with written pass criteria, and ask for the log before you ask for the price.
Every demo of AI accounting software works. The invoices are clean, the bank feed is tidy, and the match rate is high. Then you load your own data, and the product meets the supplier who writes totals by hand.
A buying process should therefore be built to find the failures. Here is the framework we would use as a controller.
Start with the problem, not the category
Write one paragraph that names the process, the volume and the pain. "Our AP team spends most of its week on invoices that fail matching" is a requirement. "We want AI" is not.
That paragraph becomes your test. Any product you consider must show it improving that specific pain on your specific data.
Six areas to test
1. Accuracy on your worst documents
Send the vendor the ugly set: skewed scans, handwritten amounts, multi-page invoices, foreign languages, credit notes. Keep a hidden answer key. Compare results yourself, and ask for per-field accuracy, not a single headline score.
2. What it does when it is unsure
This is the most revealing area. Ask the vendor to show a case the system declined to decide. Ask how it reports confidence and whether you can set the threshold. A system with no visible uncertainty will guess, and a guess in your ledger is an error you pay for later.
3. The record it leaves
Ask to see the log of a single transaction from arrival to posting. You want the inputs, the decision, the reasoning or rule applied, the time, and the identity that acted. Ask whether your auditors can export it in a readable form. Treat a thin log as a disqualifier.
4. Control and permissions
Ask what the system can read and what it can write. Look for a separate service identity, limits by amount, and respect for your period locks. The principle of separating who initiates from who approves, often called separation of duties, should hold for software too.
5. Integration with your ledger
Ask which interfaces it uses and what happens when the ledger changes. Find out who maintains the connection. A fragile integration becomes your support burden.
6. Data handling
Ask where documents are stored, who at the vendor can see them, whether your data trains a shared model, and how deletion works. Get the answers in writing and have your security team read them.
Questions that expose marketing
| Ask this | A weak answer | A strong answer |
|---|---|---|
| What is the error rate? | A single percentage with no method | Per-field rates, measured on a stated sample, with failure cases shown |
| What does the system do when unsure? | "It rarely is" | It routes to a person and reports the handoff rate |
| Can I see the audit trail? | A dashboard screenshot | An exportable record per transaction |
| How do I turn it off? | A support ticket | A switch you control, plus a documented fallback |
| Who is accountable for a wrong posting? | Silence or "you review everything" | A clear split of what the system owns and what you own |
Run a pilot that can fail
Choose one workflow and one entity. Write pass criteria before the pilot begins: the share of items that post without a touch, the maximum reversal rate, the time to first action on exceptions. Agree on how you will count them.
Run the system in proposal mode first, with your team approving each decision. Compare its proposals to what your team would have done. Disagreements are the data. Some will show system errors and some will show that your own process was inconsistent. Both are useful.
Weigh the references carefully
Ask to speak with a customer of similar size who runs a similar workflow. Ask them what broke in the first ninety days and what the vendor did about it. Ask what they would change. Discount any reference that cannot name a problem.
Decide on the whole cost
Price is one input. Add implementation effort, your review time, data cleanup and the cost of leaving. Ask how you export your rules and history if you change vendors. If the answer is "you can't", price that in.
The NIST AI Risk Management Framework offers a neutral structure if your risk team asks for one. Use it to organize questions, not to replace the pilot.
Your next step
Before you book another demo, assemble your exception set: fifty real documents that your team found hard. Make every vendor process that same set and show the log. The comparison will take less time than the demos you have already scheduled, and it will tell you more.