What Should a Good Audit Report Include? How to Read One

A good AI app audit report starts with its scope (URL, date, pages, and exclusions), scores six public product dimensions, and ties every finding to evidence a reader can trace. It keeps "what we tested" apart from "what passed," puts the fixes in order of consequence, and lists what wasn't tested, because it reviews a public site and not code or private data.

The name trips people up, so first a quick note. An AI app audit report is about an AI-built public website or web app. It doesn't evaluate an AI model. It explains what a visitor can understand and do at an authorized public URL, backs each finding with evidence, and helps the owner pick the next fix. It also isn't automatically a code review, a prompt-history review, a security assessment, or a look at private records.

What scope should an audit report state before the score?

Say you hand a teammate a report that says "the booking flow works." They'll want to know which booking flow, on what day, on what screen, and whether anyone actually submitted it. Scope answers all of that up front. A good report opens with the exact public URL, the review date, the environment, the build or version if one is available, the screen sizes, and the pages and actions included. Then it lists the exclusions, like login, live payment, irreversible deletion, or a form that couldn't be submitted safely. Without scope, "works" could mean one path someone watched or the whole product, and a reader has no way to tell which.

So I'll be upfront about ours. Is My App Slop's automated $27 product reviews a public website or web app. It can follow public links and buttons and interact with eligible public forms. It doesn't go behind a login, inspect source code, read prompt history, or verify private database records, and it isn't exhaustive quality assurance.

Which six dimensions should a report cover?

A report should cover six areas: clarity and promise, craft and design, trust, flow and usability, mobile and access, and conversion.

Clarity and promise asks whether a new visitor can tell what the product is, who it's for, what they'll get, and what to do next. Craft and design covers hierarchy, legibility, consistency, complete states, and the small details that affect understanding. Trust looks at visible identity, pricing, terms, support, privacy information, proof, and limitations. Flow and usability follows the main public path and its recoverable errors. Mobile and access checks phone-width behavior, zoom, labels, focus, reading order, and keyboard basics, without claiming accessibility compliance. Conversion follows the path from the promise to the requested or paid action, and stops short of any live charge nobody authorized.

Ours are named Clarity and Promise, Craft and Design Integrity, Trust and Legitimacy, Flow and Usability, Mobile and Access, and Conversion and The Ask.

Sticking to these six keeps the report about the product a visitor actually meets. Code maintainability, private permissions, model behavior, and legal obligations are real questions too, and a separate specialist can take them on when that work is authorized.

How should you read the scores in an audit report?

Read them as a summary of the auditor's judgment under a stated rubric. An overall score and category scores aren't a certification, a warranty, or proof that every path works. A good report explains what pushed each category up or down. It also lets one broken main path outweigh several polished side pages, because that's how a visitor experiences it.

Comparing scores over time only makes sense when the URL, scope, rubric, and key actions are still comparable. If a later review adds login, payments, or a different set of devices, call it a broader review. A trend line drawn between unlike things just makes a tidy chart out of a mismatch.

What is the difference between coverage and outcome?

Coverage is "tested" or "not tested." Outcome is "pass" or "fail," for a check that was actually done. A payment flow left out to avoid a live charge is Not tested, and it doesn't get to call itself Pass. A visible button that led to the expected public page is Tested/Pass. A form that showed a confirmation and then lost the result after a reload is Tested/Fail. Keeping both pairs in view stops the unknown areas from quietly padding your confidence.

How do you separate observations from inferred causes?

Write down what you saw, and label anything past that as a guess. If the confirmation appeared and then the submitted item was gone after a reload, say exactly that. You can add that the cause might involve persistence, as long as it's clearly marked as an inference. Declaring an unseen database fault goes too far. Public evidence supports statements about public behavior, and root-cause claims need access and investigation that may not be part of the product.

How do you make each finding traceable?

Give each finding a row that runs from the visitor's action all the way to the repair and the later retest. The table below shows the shape, using two invented findings for a booking site.

Neither row is an Is My App Slop finding or a customer outcome. They're there to show the format. In a real report, attach only the minimum evidence needed to reproduce the behavior, and leave out personal data.

How should repairs be ranked?

Rank them by the harm they could do to a real person, and then by how often they'll bite. Data exposure, wrong charges, lost work, misleading claims, and a broken main journey go first. Repeated friction a visitor can recover from comes next, and visual polish comes last. Severity should reflect the likely consequence. A dramatic screenshot doesn't earn a higher rating on its own.

For the first invented row, a narrow repair prompt could read: "Fix persistence for a successfully submitted public booking. Preserve the current fields, validation copy, confirmation design, analytics, and unrelated routes. After the change, repeat the same controlled submission, reload, and record whether the booking reference remains." It names one failure, keeps the constraints, and says how to retest. What it avoids is a broad rebuild request that might change things nobody meant to touch.

What should a report say was not tested?

All of it, and plainly: unavailable logins, time limits, side effects that were excluded, dependencies that couldn't be reached, and evidence that couldn't be collected. An automated public audit doesn't inspect code, private access rules, source prompts, internal logs, or database records. It also can't establish legal compliance, security, complete accessibility, or how well the product works for representative real people.

Each of those can be a valuable review of its own, scoped separately. A security specialist can look at threats and technical controls. An accessibility specialist and disabled participants can offer a deeper evaluation. A developer can review source quality and architecture, and a product researcher can watch target users. Naming these options makes the public report more credible, and no less useful.

How do you use the report as a working baseline?

Turn each material finding into a small work item with an owner, a repair, the evidence you'll need, and a retest status. After the changes, repeat the same page, action, expected result, and rubric. A code change alone isn't a retest, so leave "Not retested" exactly as it is until someone has seen the behavior again.

To get a report like this for your own site, find out if your app is slop.

Share the fix-first summary with the people making decisions and the detailed evidence with the people making fixes. The real sample report shows how this product presents its public findings, so I won't invent a score here. The Lovable app audit guide explains the six-dimension review, and the testing workflow covers deeper verification that the owner controls.

A useful report claims only the certainty it earned. It says which public surface it reviewed, keeps observation apart from inference, labels the unknowns as unknowns, and hands the owner a next step they can repeat.