I asked the AI pentest startup which model it runs. The answer was open weights.

I spent this morning at a talk in Bengaluru called Offensive Security in the Age of Frontier Models, run at Basecamp as part of Bengaluru Tech Week. Small room, about ninety people registered on Luma, one company presenting: Greywatch. They sell continuous automated attacks against your product, with a working exploit attached to every finding.

The subtitle on their opening slide: “The attacker got faster, cheaper, and tireless. Most products haven’t noticed yet.”

The line on the event page is the whole pitch:

Two out of three funded startups we assess have gaps serious enough to enable account takeover, sensitive-data access, or database exfiltration.

The argument behind it: agents have gotten good at attacking, and cheap at it. Attackers already have that. You may as well point the same thing at yourself first.

What the public record actually shows

Before taking a security vendor’s word on “attacks are cheaper now,” here is what I could check.

XBOW reached the top spot on HackerOne’s US leaderboard, the first autonomous pentester to do it. Their own writeup puts the volume at nearly 1,060 submitted vulnerabilities, with 54 critical, 242 high, 524 medium and 65 low over the ninety days before that post.

Anthropic published a report on an AI-orchestrated espionage campaign where a jailbroken agent ran the chain from reconnaissance through exploitation and data extraction across roughly thirty targets.

Neither of those tells you what an attack costs now. What they do establish is narrower and still worth something: agents can carry a real offensive operation end to end, at volumes a person cannot match, and at least one crew has already pointed one at companies that did not consent. Greywatch’s version of this is “there’s no longer a cost that makes skipping you worthwhile.” That is a sales line, not a measurement, and I have no number to put against it.

Which model, which harness

The thing I wanted out of the morning was the stack. The answer, from talking to someone there after the talk: open-weight models. GLM, Kimi, DeepSeek. Driven through opencode, which is MIT-licensed and public. On top of that, their own security-oriented harness that turns model output into a finding with a reproduction attached.

Treat that with the weight it deserves. It is one hallway conversation, not a datasheet. I did not record it, I did not get it in writing, and I cannot tell you whether that describes what runs in production, what they were experimenting with last month, or one component of several.

Taking it at face value anyway, since it matches what I would expect: GLM-4.7 on z.ai’s published price list is $0.60 per million input tokens and $2.20 per million output. That does not mean a scan is cheap. Tokens are one input to a cost model that also includes retries, parallel agents, browser and sandbox compute, and the people who read the output. I do not have those numbers and neither does anyone quoting token prices at you.

What it does mean is that “we have a model you cannot get” is not what is being sold here. The part that would be hard to copy is the harness, and none of that is visible from outside.

The part their own copy says out loud

From the Greywatch landing page, describing the “Find it” motion:

A security engineer filters every finding, so you get the few that are real and exploitable. Not 1,000 tickets.

Elsewhere on the same page: “2 humans in the loop on every finding,” and “think of it as security engineers backed by an AI research team.”

They are not hiding this, and I do not think it is a weakness. Human review is a reasonable answer to a real problem. But it does tell you what to ask a vendor in this category, and none of it is answerable from the outside: how many candidate findings does the machine produce for each one you receive, how long does a reviewer spend on each, and what happens to that ratio when you add a customer.

My guess is that cheap models make candidate bugs almost free to produce, so the expensive step moves from finding to deciding what is real. That is a guess and I am not going to dress it up as anything better. Nobody in this category has published a false-positive rate, a queue depth, or reviewer minutes per candidate.

The slide worth stealing

The part I would hand to an engineering team was the lifecycle. Five stages, three controls each, none of it requiring you to buy anything. I photographed the run, and what follows is close to the card text as it appeared. Where I have added anything it is a link.

1. Spec. Authorization model: name who is allowed and the tenant boundaries, as acceptance criteria. Abuse cases: write the ways it breaks, not just the happy path. In the agent’s rules: AGENTS.md, CLAUDE.md, an OWASP ASVS checklist.

2. Generate. Deny rules: an allowed-library list and blocked patterns. Boxed in: a worktree or sandbox, with short-lived, scoped credentials. Provenance: tag every AI-written diff so you can audit it later.

3. Merge gate. SAST blocks the merge: Semgrep and CodeQL, on every commit. Secrets and deps: gitleaks for secrets, Socket for dependencies. Authz test: user A cannot read user B, propose the fix, revert if it breaks.

4. Deploy. Least privilege: tight IAM and Kubernetes RBAC, IAM Access Analyzer. Egress allowlist: stop SSRF and exfiltration, IMDSv2, network policy. Short-lived tokens: no long-lived keys, OIDC for CI.

5. Runtime. Continuous red team: not an annual pentest. ZAP, agentic runs. Monitor AI features: watch for prompt injection and tool misuse in your own app. Canaries and deception: catch and slow intruders, canarytokens.org.

Two things were illegible in my photos and I am not going to guess at them: a tool name at the end of the “deny rules” card, and a second scanner named next to ZAP on the Runtime slide. Projector plus phone camera plus a pale gradient.

Nothing on that list is new. Semgrep and gitleaks have been around for years, and any AppSec engineer could have written the same fifteen cards in 2019. What I took from it is the ordering, and specifically that the “start here” badge sits on the merge gate rather than on the continuous attack loop they sell. The charitable reading is that a talk for founders should open with the cheapest useful step, and that is probably all it means. It is still the recommendation I would follow first, and it costs nothing.

What I’m doing with this

I am building a small scanner on the same shape of stack: open-weight model, opencode as the driver, a thin harness of my own that knows what my products look like and what counts as a real finding for them. I am not trying to compete with anyone. I want to know whether running it on every release is affordable, and the only way to find out is to run it and measure.

The things I plan to write down: tokens and rupees per scan, how many candidates it produces per real bug, how long I spend rejecting the rest, and whether the exploits it claims actually reproduce.

Two open-source projects I am reading first. strix, Apache-2.0, gives its agents an HTTP proxy, a browser, a shell and a Python sandbox, so it exercises a running target rather than reasoning about source. pentagi, MIT, is the heavier option: self-hosted, GraphQL backend, closer to a platform than a tool. I have run neither, so treat both as bookmarks.

The failure I expect first is scope, not discovery. An agent with a shell and credentials, pointed at a target, will follow whatever is reachable from there, and on a cloud account “reachable” is a larger set than you planned for. That is the boring reason sandboxes and scoped credentials exist, and I expect to rediscover it the expensive way. Discovery quality could fail first instead. I will report which one it was.