I assumed no scanner covered my MCP server. deepsec ships three matchers for it.

Four days ago I wrote up a talk by an AI pentest startup and ended it saying I was going to go read some open source. I did that this weekend. Three repos: deepsec from Vercel Labs, Strix, and Anthropic’s defending-code-reference-harness.

I went in with a section already written in my head, about how none of these tools would look at the surface I actually worry about. That section was wrong, and finding out why was worth the weekend.

What the three actually are

They get lumped together as “AI security scanners.” They are not the same tool, and the differences decide where each one earns its keep.

deepsec is a deep static sweep. Regex matchers find candidate sites with no AI calls involved, then coding agents investigate each one. Five stages: scan, process, triage, revalidate, export. It is built for reading all the code in a large existing repo, hunting what has been sitting there for years. The matching is free and the investigating is not. The README says scans “can cost thousands or even tens-of-thousands of dollars for large codebases.” There is a process --diff mode that scans and investigates only files changed in a diff, and a --max-cost-usd flag, and that combination is probably where most teams should live. Apache 2.0.

Strix is a pentester, not a reader. It runs your app, attacks it, and validates what it finds through “actual proofs-of-concept.” Point it at a URL, a repo, an OpenAPI spec or a Postman collection, and it hunts the class of bug that source review structurally cannot find: IDOR, privilege escalation, auth bypass, race conditions, payment manipulation. There is a GitHub Actions workflow, and in CI pull request runs it scopes quick reviews to changed files. Apache 2.0. No published cost per run, and I am not going to invent one.

Anthropic’s harness is not a product and says so. “This repo is not maintained and is not accepting contributions,” and the docs call it “a reference, not a product.” What you get is the shape of the pipeline: the prompts, and the container and grading boundaries around them. The Claude Code skills in it (/threat-model, /vuln-scan, /triage, /patch) are read/write on files only, and the docs say they are “safe to run unsandboxed, as long as you review and approve each tool use.”

Know this before you get excited about that last one: the autonomous pipeline targets C and C++ memory bugs, using Docker and ASAN. It is written to be portable, and there is a /customize skill for that, but out of the box it will not run against a TypeScript monolith. The skills port. The harness is a weekend of work.

I have not run all three against the same target, so I cannot tell you how much their findings overlap or what layering costs you in duplicate triage. What I can say is that they read three different surfaces, and my plan below uses them in sequence rather than picking one.

Verification is the part you pay for

Regex candidate sites are nearly free. Deciding which ones are real is the entire cost, in money and in the attention of a team that does not have a security engineer. A scanner with a bad false-positive rate consumes the same roadmap a real incident would, and worse, it teaches everyone to ignore the security queue.

The best writeup of what to do about that is in the Anthropic repo, and it is specific. A finding does not count until the find agent produces an input that “crashes 3 out of 3 times.” Then a second agent, in a fresh container, re-runs the proof of concept and checks that the crash is real. The detail I keep thinking about:

The only thing that crosses from the find container to the grader is the PoC bytes, so the grader isn’t influenced by the find agent’s reasoning.

That is the idea in one sentence. Their own number: adding an adversarial verifier “roughly halved the rate of non-exploitable findings from the discovery phase.”

I want to be careful about how far that generalises, because it is easy to read it as an industry consensus and it is not. deepsec has a revalidate stage that re-checks findings and cuts the false-positive rate, with its authors putting the remainder at “roughly 10-20%,” but nothing in the repo says that revalidator is blinded to the original investigation. Strix validates by landing an exploit, which is a witness but not a second grader. One of the three publishes the blinding detail. All three pay for verification somehow. Those are different claims and I had them mushed together in my first draft.

The corollary is uncomfortable if you like tidy dashboards. From the same Anthropic write-up:

don’t expect the nth run to have zero new findings. Models are stochastic, and a large codebase can have a long tail of vulnerabilities that continue to trickle in even when the code is unchanged.

Each run is a sample. You union across several, and a clean run is not proof of anything.

Three other things from that document, all of which changed what I plan to do. Run ten agents in parallel with no map and they converge on the same shallow bugs, so have a model do a first pass that partitions the search space by attack surface or component, then feed those partitions to the parallel agents. One team quoted there tried the obvious thing first: “We initially tried to just horizontally scale and send more agents, but saw limiting returns.” Long checklists backfire, which I did not expect: “more prescriptive prompts make discovery worse,” because the model pattern-matches your scaffolding instead of reasoning about the code. And rank targets by blast radius, phrased there as “If the PoC fires, who is affected? One user or all users, one tenant or the platform, userland or the kernel?”

The part I got wrong

Here is the section I sat down to write, before I read the source.

We are building MCP tooling for WordPress. Our per-site server exposes things like execute_php and db_query over an HTTP endpoint behind a bearer token. That is not a bug, it is what makes an agent useful on a site instead of merely conversational. But look at it with a security hat on: we ship remote code execution as a feature, to sites we do not operate, for users who will paste that token into whatever agent they are using this month. Then add prompt injection. An agent reads a post. The post contains text an attacker wrote. The agent has execute_php.

My claim was going to be that none of these tools look at that, and that there is not even a CWE for it. Both halves are wrong.

There is a CWE. CWE-1427, “Improper Neutralization of Input Used for LLM Prompting,” covers exactly the case where a product builds prompts from externally provided data and the model cannot tell developer directives from user input.

And deepsec ships matchers for it. I counted 198 matcher files in packages/scanner/src/matchers/ this morning, and three are pointed straight at my problem. agentic-untrusted-prompt-input describes itself in its own header comment as an “Indirect-prompt-injection sink detector,” flagging prompts that interpolate variables from CRM notes, scraped HTML, warehouse rows, Slack history. agent-tool-definition matches tool definitions with the comment “These are prompt injection attack surfaces: if an attacker controls agent input, tools are the payload,” and labels a shell-exec hit as “RCE via prompt injection.” And mcp-tool-handler exists to flag “every file that registers an MCP tool so a downstream investigation can verify auth + input validation per tool.”

That last phrase is the one that survived my correction. The matcher’s job is to hand a candidate site to a downstream investigation. What investigates it? An agent, writing an argument. Which is the thing the rest of the pipeline is engineered not to trust.

What a witness for this would look like

So I went looking for the oracle. My first instinct was that there is not one, because nothing crashes. That is too strong, and I talked myself out of it by trying to specify the test.

Take a real finding against our server: untrusted post content reaches a prompt in a context where execute_php is callable. There are at least five separate things you might mean by “verified,” and they need different evidence.

1. The data flow exists. Static, cheap, a matcher already does this.
2. The tool is reachable from that agent state. Checkable by inspection.
3. A crafted injection induces a prohibited tool call. Runnable, and this is the interesting one.
4. It fires often enough to matter. Requires repetition.
5. The call causes material harm rather than a no-op.

Step 3 is where the equivalent of ASAN has to go, and it does exist, it is just not the same shape. You do not need the model to admit anything. You instrument the tool layer, plant a canary the injected text tries to exfiltrate or a call the agent should never make, run the scenario N times, and count. The assertion is on the tool invocation, not on the model’s stated reasoning, which is the same discipline as passing only PoC bytes to the grader.

The difference from ASAN is that the answer is a rate, not a bit. “Crashes 3 out of 3” is a gate you can put in CI. “Fires 11 times in 100” is not obviously pass or fail, and I do not know where the line goes. Nobody does yet, as far as I can find. Reducing your injection success rate from 11% to 2% is real work and real progress, and it is also still a shipped remote code execution path that an attacker can retry.

Strix does not list prompt injection among the classes it hunts. I noticed that and briefly wanted it to mean something deep. The boring explanation is more likely: Strix is a web and API pentester, the README list reads as illustrative rather than exhaustive, and running an agent against your agent N times is a different rig from firing HTTP requests at an endpoint. What it does tell me is that nobody has handed me this harness, so it is mine to build.

So the honest version of my worry: detection of agent tool surfaces is further along than I assumed, and better than I would have written myself. Verification of them is a rig that exists in principle, that I have not seen shipped in any of the three, and that has no agreed threshold even once you build it.

What I’m doing

Over the next month, concretely.

Week one, no budget approval needed. Run the read-only skills across each service: threat model, static scan, triage. It costs tokens and about a day of my attention, not a purchase order, which is the only sense in which it is free. The threat model is the artifact I want most, because it is the input that tells the expensive tools where to look.

Week two, the crown jewels only. deepsec on the two repos with the widest blast radius, not the monolith, with --max-cost-usd set. A tight context file naming our auth helpers and middleware. I am deliberately keeping it architectural rather than a checklist of bug categories, given the finding above that prescriptive prompts make discovery worse, though I will admit I am guessing at where that line sits. First thing I read is the output of those three agentic matchers.

Week three, staging. Strix against a staging environment, authenticated as two separate tenants, pointed at cross-tenant access. Staging only. These agents genuinely attack what you point them at, and “I was testing” is not a legal defence.

Week four is the one that matters, and it is the one I would have skipped if I had shipped my original draft. Build the injection rig: a seeded WordPress with attacker-controlled post content, our MCP server attached, the tool layer instrumented, a canary call that should never happen, and a loop that runs each scenario a hundred times and reports how many times it fired. That number is the only security metric I will have about our newest surface, and right now it does not exist.

The stale finding backlog goes in too, and here I have to be consistent with the rest of this post: a model saying an old finding is not real is an argument, not evidence, and I said arguments do not count. So the backlog pass only closes things where the disproof is checkable, a call site that no longer exists, a sanitiser I can see fire. Everything else stays open and annoying.

Everything the patcher produces is a draft. A human reads the diff.

Separately, and this is anecdote rather than data: I have spent time on the other side of a diligence process lately, and the thing that cost us was not the finding, it was not having a dated report and a ledger of what we did about each item. If you expect to be in one within a year, produce those before someone asks for them.