In July, an AI model filled in a police tip form about an unsolved murder. It had no information to give. It saw a form and completed it. For a small business, the lesson is practical: before an AI tool can send, submit, book or update anything on its own, decide what it may do, what it must not do, and who checks the work.
What happened when an AI model completed a police tip form?
On 18 July, an Anthropic AI model (Claude Haiku 4.5) was generating example tasks on randomly selected web pages. It came across a tip form on a Philadelphia police website for unsolved murders and submitted it, indicating it might have information about a case. It did not. The submission was marked as spam and was never passed to investigators. The incident became public on 9 October, when Anthropic released a report and the police announced it.
Anthropic’s own account, as reported, is that the model appears to have been producing example content for the task, not trying to mislead anyone. We agree that this is the more useful way to read it. The AI treated a police tip form like any other form on a webpage: something to complete.
Ceridwen covered the incident in edition 4 of her newsletter, AI Without the Sci Fi: When AI Learns to Lie. It is worth reading for the full story. Here, we are taking the practical side for a business.
Why is checking afterwards not enough when implementing AI?
Because when an AI tool completes the action itself, the action has already happened by the time anyone looks. A sent email cannot be unsent. A submitted form is sitting in someone else’s system. A confirmed booking has a customer attached to it. In this case a spam filter stood between the tip and the investigators. Your customers, suppliers and records may not have one.
Checking afterwards tells you what happened. It does not stop it. What was missing was a combination of two things: clear limits on what the AI could and could not do, and a person checking the work before it took effect.
Here is a hypothetical example. Picture a business with an AI tool that handles customer enquiries. If the tool drafts the replies and a named person reviews and sends them, the limit is built into the process. If the tool can send replies directly, the first check is a customer complaint.
Two terms are worth defining.
Human check: a point in a process where a named person reviews and approves an AI’s work before it takes effect.
Clear limit: a written boundary on what an AI tool is allowed to do, including what it must not do.
What should you do before implementing AI in your business?
We work on the process before the tool. Our approach to AI readiness has five steps.
Map what actually happens today, step by step. Include the workarounds and who really does each task, not just the version in the procedure document. You cannot decide what AI should do in a process you have not mapped.
Decide which steps AI can draft, and which need a person to approve. Drafting is usually low risk. Sending, submitting, paying, deleting and changing records are the steps that need a person to say yes first.
Set clear limits on what the AI may do and, particularly, what it must not do.
Name the person responsible for each check, so it does not depend on someone remembering. A role is a start. A named person, with a backup for when they are away, is better.
Then automate what makes sense. Once the first four steps are in place, you can decide what to automate with a clear view of what could go wrong and who will catch it.
If the honest answer to step 4 today is “nobody, really”, and the limits are not written down, that is the gap to close first.
What should you tell an AI tool before it starts work?
There is one more step worth taking: give your AI the same expectations you would have of an employee. It is not just what gets done, but how. Alongside proper controls, it gives the AI permission to pause rather than push through.
Here is wording you could adapt:
“Be honest about how the work was done. Don’t make anything up, and don’t conceal failed steps or change records to cover errors. Tell me what’s confirmed and what’s estimated, what you checked and what you didn’t, and any evidence that doesn’t support your recommendation. If you can’t complete the task as asked, stop and tell me why.”
The last line matters most. Reporting on the incident describes Anthropic’s explanation that models sometimes work around a restriction instead of stopping when they cannot complete a task as given. An instruction that says “stop and tell me why” is aimed at exactly that.
To be clear about its limits: an instruction like this is one control among several. It does not prevent errors on its own. It sits alongside the limits on what the tool can access and the person who checks its work, and it does not replace either.
Where do your checks and limits sit today?
Look at everything an AI tool in your business can send, submit, book or update on its own. For each one, ask three questions. What is it not allowed to do? Where does a person check the work? Who is that person?
This is not a question about whether the AI can be trusted. It is that the checks, and the limits on what it may do, were never built in. That is a process question, and it can be fixed before the tool goes live.
If you would like to talk through where the checks and limits sit in your own business, get in touch. You can also read Ceridwen’s full newsletter edition, When AI Learns to Lie, for more on the incident.