Home/Journal/Opinion
OpinionSecurity StrategyComplianceContinuous Testing

Quarterly Pentest, Scanner, SOC 2: Why They Fail in 2026

The conventional defenses were built for an attacker who moves slowly. That attacker no longer exists. The Zero Day Clock data makes the gap undeniable.

Greywatch Research··7 min read

The 3 things the industry still treats as “enough” - a quarterly pentest, an automated scanner, a SOC 2 badge - were defensible choices when mean time-to-exploit was measured in years. That window is gone. In 2018, attackers needed 771 days to weaponize a disclosed vulnerability. By 2024 it was 53 days. By 2025, most exploited vulnerabilities were being weaponized before public disclosure. Against that threat model, a 90-day-old report, a wall of unfiltered scanner findings, and a compliance checkbox are not a security posture - they’re a liability dressed up as one.

Zero Day Clock — mean days from disclosure to active exploitation
800d600d400d200d0d771d53dquarterly cadencebecomes indefensibleexploited before disclosure20182020202220242026
Source: Greywatch analysis of CISA KEV, Mandiant M-Trends 2024, and Project Zero data.

Does SOC 2 mean I’m secure?

No. SOC 2 means the right policies are documented. It does not mean your product can’t be broken into - and the difference is not semantic. The audit checks whether you have access controls documented, not whether those controls actually stop an attacker who probes your API endpoints for 6 hours with automated agents. Every one of the critical findings Greywatch has confirmed in the last 7 days would have passed a SOC 2 review. Compliant is not the same as secure.

The quarterly pentest: a historical document masquerading as a security posture

A VAPT report is a snapshot. It is accurate on the day it ran and nowhere else. The day after it was written, your team shipped code. Then they shipped again. And again. 47 deployments later, the attack surface described in that report no longer corresponds to the product your customers are using.

External Vulnerability Assessment Report
Q2 2026 — Acme Corp — Confidential
Prepared by: SecureAudit Partners
Report date: June 2, 2026
Scope: api.acmecorp.com (as of 2-Jun)
Critical findings: 2
High findings: 4
Next review: September 2026
Re-test coverage: 0 of 47 subsequent deploys
VALID 94 DAYS AGO
47
deploys shipped since
0
re-tested

The 90-day audit cadence made sense when exploitation windows were measured in years. If a new vulnerability was disclosed and your attacker needed 18 months to weaponize it, a quarterly review gave you enough lead time to catch it. That math stopped working around 2021 when mean time-to-exploit crossed under a year. By 2023 it was 128 days - already shorter than most audit cycles. In 2024 it was 53 days. The quarterly cadence is not a conservative choice anymore. It is a gap.

The other problem is shipping frequency. The average Series A-C product team ships multiple times per day. A test that runs 4 times per year is testing a product that no longer exists by the time anyone reads the report. You are not getting coverage - you are getting a receipt.

The automated scanner: coverage without judgment is just noise

Point any CVE scanner at a real codebase and it will return findings. Hundreds of them, often over a thousand. They come back unranked, unfiltered, with no sense of what is actually reachable by an outside attacker versus what is a theoretical flag on a dependency no one calls.

scanner-output.txt — 1,247 findingssorted by: detection order
CRITCVE-2024-3094 · liblzma/xz backdoor detected in build dependencies
MEDMissing X-Frame-Options header on 14 endpoints
LOWCookie set without SameSite attribute (session_id)
MEDOutdated jQuery 3.4.1 detected — known XSS vectors
LOWServer version disclosed in response header (nginx/1.18.0)
INFOTLS 1.1 supported on legacy endpoint /api/v1/legacy
CRITCVE-2023-44487 · HTTP/2 rapid reset attack — 3 affected services
MEDContent-Security-Policy header not set on /dashboard/
INFOTRACE method enabled on primary domain
LOWReferrer-Policy not set — 23 pages
HIGHCVE-2024-21733 · Apache Tomcat partial PUT request DoS
MEDCross-origin resource sharing misconfiguration on /api/data
INFOrobots.txt exposes internal path structure
LOWPermissions-Policy header not set
CRITCVE-2021-44228 · Log4Shell — log4j 2.14.0 in classpath
Showing 15 of 1,247 findings — scroll to view all — 0 filtered — 0 confirmed exploitable

The result is predictable: teams stop reading the reports. When 1,247 findings land in a ticket queue with nothing to indicate which 3 of them would actually allow an attacker to exfiltrate customer data, the natural response is to close the tab and move on. Alert fatigue is not a discipline failure - it is a rational response to a signal with no signal.

The scanner’s model of the world is flat: it finds patterns that match known signatures and flags them. It has no model of your specific application logic, no understanding of which endpoints are publicly accessible, no ability to chain a low-severity misconfiguration through 3 hops to a critical data exposure. A CVE flag on a library is not the same as a working exploit. The gap between those 2 things is where the actual attack surface lives.

“A scanner tells you what might be wrong. Greywatch tells you how we got in.”

The compliance badge: the checkbox has never stopped an exploit

SOC 2, HIPAA, ISO 27001, a green GRC dashboard from Vanta or Secureframe - these are real and necessary for enterprise sales. They are not a security posture. The audit verifies that controls are documented, access is logged, incidents have a response plan. It does not attempt to break into your product. No auditor probes your API. No one runs a credential stuffing attack against your authentication layer. No one checks whether your AI-generated code introduced an IDOR vulnerability in the route added last Tuesday.

The compliance attestation answers: “Does this company have security policies?” The attack answers: “Can I get in anyway?” Those are different questions, and the badge only addresses one of them.

The more durable risk for a Series A-C company is this: a breach that happens inside your compliance window is still a breach. The SOC 2 report does not protect customers whose data was exfiltrated. It does not protect you from the regulatory consequence or the customer trust collapse. It documents that you tried - which matters in litigation, but not to the attacker who already has the data.

What actually works: continuous testing with judgment in the loop

The attacker went continuous. The defense has to match that cadence.

The core problem with all 3 conventional defenses is that they were designed for an attacker who moves slowly. The Zero Day Clock data shows that attacker no longer exists. The one who does moves in days, sometimes hours, and is increasingly automated.

The model that holds up against that threat has 3 properties:

01
Continuous
Our AI agents run continuous offensive testing against your external attack surface. It runs as often as you ship code.
02
Reviewed
Every candidate finding goes to a security engineer before it reaches you. What lands is a short list of confirmed, exploitable issues. No noise dump.
03
Reproducible
The exact HTTP request that worked, the data it exposed, reproduction steps your engineers can run, and a recommended fix. Not “this might be vulnerable.” This is how we got in.

Frequently asked

Does SOC 2 mean my company is secure?
No. SOC 2 confirms that your security policies are documented and your controls are in place. It does not test whether your product can be broken into. Greywatch has confirmed critical findings in SOC 2-compliant companies - vulnerabilities that would have passed the audit entirely because no one attempted to exploit them. Compliance and security address different questions.
Is a quarterly pentest enough for a Series A startup?
Not in 2026. Mean time-to-exploit was 771 days in 2018, when a quarterly cadence was reasonable. It is now measured in days - and in many cases, exploitation begins before a vulnerability is publicly disclosed. A 90-day-old report describes a product you no longer have. If your team ships multiple times a day, your security coverage needs to match that cadence.
What’s wrong with automated vulnerability scanners?
The problem isn’t that scanners find too little - it’s that they find too much, with no prioritization and no understanding of what’s actually reachable. A wall of 1,247 findings, with critical CVEs sitting next to informational headers, produces alert fatigue. Real vulnerabilities go unpatched because teams stop reading the reports. Coverage without judgment is noise.
How is continuous offensive security different from a pentest?
A traditional pentest is a point-in-time exercise: a team tests the product as it exists on a given day, writes a report, and leaves. Continuous offensive security runs permanently - the attack surface is probed as your product ships, not once per quarter. The output isn’t a report; it’s an ongoing feed of real, prioritized, proven findings that reflects your current product.
What does “prove it” mean in practice?
Most scanners flag a potential vulnerability and move on. Greywatch confirms whether it’s actually exploitable, then delivers the exact HTTP request that worked, the data it exposed, and step-by-step reproduction steps your own engineers can verify. The distinction matters: a flag that can’t be reproduced gets triaged. A working exploit with data gets fixed.

If your team ships continuously and your security is still tested quarterly, the gap is real. Talk to the founder about what Greywatch's AI agents find on your product.

Talk to the founder →