---
title: "A controller's questions for evaluating AI accounting software"
description: "Demos show the clean path. These questions force a vendor to show the exceptions, the limits and the logs, which is where AI accounting software succeeds or fails."
author: "Mara Voss"
category: "Competitive Landscape"
date: 2026-10-24T15:10:39.615Z
canonical: "https://tasksheriff.com/blog/how-to-evaluate-ai-accounting-software"
---

# A controller's questions for evaluating AI accounting software

![Hand pulling back a smooth sheet of paper to reveal crossed-out entries and tangled loose threads behind it](https://hsppuvezyxmkpzkgfkho.supabase.co/storage/v1/object/public/media/enrichment/39b6f926-5a52-4bf1-99e0-a3cf4d8c6ff8/b4faa85f-e293-4ffb-9715-fb98d81e1cb6/f02ce856-1d6d-4c3f-8e59-4b38914c259d.png)

> Demos show the clean path. These questions force a vendor to show the exceptions, the limits and the logs, which is where AI accounting software succeeds or fails.

Every demo of AI accounting software works. The invoices are clean, the bank feed is tidy, and the match rate is high. Then you load your own data, and the product meets the supplier who writes totals by hand.

A buying process should therefore be built to find the failures. Here is the framework we would use as a controller.

## Start with the problem, not the category

Write one paragraph that names the process, the volume and the pain. "Our AP team spends most of its week on invoices that fail matching" is a requirement. "We want AI" is not.

That paragraph becomes your test. Any product you consider must show it improving that specific pain on your specific data.

## Six areas to test

### 1. Accuracy on your worst documents

Send the vendor the ugly set: skewed scans, handwritten amounts, multi-page invoices, foreign languages, credit notes. Keep a hidden answer key. Compare results yourself, and ask for per-field accuracy, not a single headline score.

### 2. What it does when it is unsure

This is the most revealing area. Ask the vendor to show a case the system declined to decide. Ask how it reports confidence and whether you can set the threshold. A system with no visible uncertainty will guess, and a guess in your ledger is an error you pay for later.

### 3. The record it leaves

Ask to see the log of a single transaction from arrival to posting. You want the inputs, the decision, the reasoning or rule applied, the time, and the identity that acted. Ask whether your auditors can export it in a readable form. Treat a thin log as a disqualifier.

### 4. Control and permissions

Ask what the system can read and what it can write. Look for a separate service identity, limits by amount, and respect for your period locks. The principle of separating who initiates from who approves, often called [separation of duties](https://en.wikipedia.org/wiki/Separation_of_duties), should hold for software too.

### 5. Integration with your ledger

Ask which interfaces it uses and what happens when the ledger changes. Find out who maintains the connection. A fragile integration becomes your support burden.

### 6. Data handling

Ask where documents are stored, who at the vendor can see them, whether your data trains a shared model, and how deletion works. Get the answers in writing and have your security team read them.

## Questions that expose marketing

Ask thisA weak answerA strong answer

What is the error rate?A single percentage with no methodPer-field rates, measured on a stated sample, with failure cases shown
What does the system do when unsure?"It rarely is"It routes to a person and reports the handoff rate
Can I see the audit trail?A dashboard screenshotAn exportable record per transaction
How do I turn it off?A support ticketA switch you control, plus a documented fallback
Who is accountable for a wrong posting?Silence or "you review everything"A clear split of what the system owns and what you own

## Run a pilot that can fail

Choose one workflow and one entity. Write pass criteria before the pilot begins: the share of items that post without a touch, the maximum reversal rate, the time to first action on exceptions. Agree on how you will count them.

Run the system in proposal mode first, with your team approving each decision. Compare its proposals to what your team would have done. Disagreements are the data. Some will show system errors and some will show that your own process was inconsistent. Both are useful.

## Weigh the references carefully

Ask to speak with a customer of similar size who runs a similar workflow. Ask them what broke in the first ninety days and what the vendor did about it. Ask what they would change. Discount any reference that cannot name a problem.

## Decide on the whole cost

Price is one input. Add implementation effort, your review time, data cleanup and the cost of leaving. Ask how you export your rules and history if you change vendors. If the answer is "you can't", price that in.

The [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) offers a neutral structure if your risk team asks for one. Use it to organize questions, not to replace the pilot.

## Your next step

Before you book another demo, assemble your exception set: fifty real documents that your team found hard. Make every vendor process that same set and show the log. The comparison will take less time than the demos you have already scheduled, and it will tell you more.

---
Source: https://tasksheriff.com/blog/how-to-evaluate-ai-accounting-software