Dummy Run: letting fictional users try your app first
Why I'm building Dummy Run, an open-source tool where AI personas use your app in a real browser and report what broke, with the evidence attached.
“Dummy run” is British English for a trial run. In my new project the dummies literally do the running.
Dummy Run is an open-source tool that lets AI play your users and try your app before your customers do. It’s on GitHub under the Apache 2.0 licence.
The idea #
Most teams now get a preview build for every pull request. Someone’s meant to have a poke around it before it’s merged. In practice they’re flat out, they click through the happy path, and nobody logs in as the awkward customer, the read-only auditor or the person on their phone on the bus.
Dummy Run gives that job to a cast of personas. A persona is a fictional user with a role, permissions, goals, habits and a device. A run signs one persona in to your preview in a real Chrome or Firefox. It then tries to get its job done and reports three things:
- what broke
- what confused it
- what its role should never have been able to see
Evidence, or it didn’t happen #
I’ve seen enough AI tools cheerfully report problems that aren’t there, so this one is built around evidence. Every run produces steps, screenshots and logs. A reviewer checks each claim against that evidence and the code. Only the claims that hold up reach the results board.
And nothing is posted for you. Findings go to the board. A person decides what becomes a ticket.
Your data stays where it should #
You choose the model provider: Azure OpenAI, Anthropic, Gemini, Amazon Bedrock, or anything OpenAI-compatible, including a model running on your own machine. You can pick one per market, so a run for a UK client stays in the UK and an EU one stays in the EU. I’ve spent a fair bit of my day job on UK data residency, so that one wasn’t up for debate.
Observable from day one #
This is the bit I’ve enjoyed most. Every run is an OpenTelemetry trace, with a span per step and an event per finding. So you can see your fictional users in the same place as your real ones. There’s a Grafana integration with dashboards ready to go, and a local stack you can bring up with a single docker compose up.
Who it’s for #
- QA leads who want exploratory testing by role on every pull request, without maintaining scripts.
- Developers who’d like to see their product used the way a customer actually uses it.
- Agencies running one cast against every client’s preview.
- Product and UX people who want friction tied to a step and a screenshot.
- Anyone in compliance or security who wants to know what each role can see or do beyond its rights.
Have a look #
The site is at dummyrun.dev and the code is at github.com/dummy-run/dummy-run. Issues and discussions are open. If you give it a go on your own app, I’d be chuffed to hear what your dummies turned up.