--- title: "Finding high-severity security issues with publicly available models" description: "We pointed 10,000 coding agents at Ramp's backend and found seven confirmed high-severity vulnerabilities. A simple, model-agnostic pipeline for scaling security research with publicly available models." date: "2026-05-27" authors: "Alex Levinson" tag: "Research" slug: "security-scanning-public-models" canonical: "https://labs.ramp.com/research/security-scanning-public-models/" --- ![White and yellow dotted agent trajectories converge around the Ramp logo.](/research/articles/security-scanning-public-models/cover.webp) We pointed 10,000 coding agents at Ramp's backend, instructing each one to find a security vulnerability. Thousands more agents deduplicated, reproduced, and prioritized the results. The output was seven confirmed high-severity vulnerabilities, along with a number of lower-severity findings. Many were sophisticated and subtle, with attack paths that chained behavior across multiple systems; all were novel, having evaded our penetration tests, bug bounties, and prior AI scans. The scan ran on publicly available models, and the pipeline is model-agnostic: cheaper open-weight models still surface high-severity vulnerabilities. Building and operating the scan did not require closed-access models or a custom security-focused agent harness. ## Ramp Inspect We built our scan pipeline on top of [Inspect](https://builders.ramp.com/post/why-we-built-our-background-agent), Ramp's internal background coding agent, which also powered [parts of our previous AI scans](https://engineering.ramp.com/post/100-vulnerabilities-patched-with-0-humans). Each Inspect session runs in its own hosted sandbox with a full dev environment, so the agent can make real API requests, run tests, and reproduce bugs end-to-end against live code. Inspect uses hosted compute and exposes an API for creating and managing sessions, which makes scaling to hundreds of parallel runs trivial. ## Scaling Agent Compute Rich Sutton argued in [The Bitter Lesson](http://www.incompleteideas.net/IncIdeas/BitterLesson.html) that general AI methods leveraging computation consistently outperform approaches relying on human-engineered knowledge. Work on code generation bears this out: scaling the number of LLM attempts on programming tasks improves success rate log-linearly, a result demonstrated in [Competition-Level Code Generation with AlphaCode](https://arxiv.org/abs/2203.07814) and [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://arxiv.org/abs/2407.21787). ![Coverage increases with logarithmic sample counts across five coding and reasoning benchmarks. SWE-bench Lite reaches 56%, compared with a 24.67% single-attempt GPT-4o baseline.](/research/articles/security-scanning-public-models/scaling-compute.webp) *Success rate scales log-linearly with the number of samples across coding and reasoning tasks. Adapted from Brown et al., Large Language Monkeys (ICLR 2025).* The same should hold for cybersecurity tasks, and vulnerability discovery is an especially strong fit for this paradigm. Unlike traditional coding tasks, where the *final* answer has to be right, an adversarial security scan only requires *one* trajectory across many to land a real hit. Failed passes cost only tokens, and each additional pass is another independent shot at the same target. Following this principle, we designed our scan to be general and to scale with compute. In practice, this meant: - Massively parallel deployment of scanner agents. We ran one Inspect session on every endpoint in Ramp's core backend, with extra passes on security-sensitive surfaces like auth and payments. - A minimal, non-prescriptive prompt that lets the agent rely on its own intelligence. The instruction is essentially "find security issues in Ramp, with this endpoint as a focus." We also provided the agent with a short document detailing our trust boundaries, edge protections, and other cross-repo security context. In an 8-hour run, we spun up roughly 10,000 Inspect sessions against our backend, recording each potential finding as a ticket in an internal issue tracker. To avoid poisoning our downstream triage steps, we told the agent to write neutral, factual tickets. If the agent found nothing worth noting, no ticket was created. ## Triage The result of our initial scan was a large backlog of 6,000 raw tickets. To transform this backlog into actionable security work, we faced two problems: - Sessions hitting shared code paths converged on the same potential finding, resulting in duplicate tickets. In our run, a single UUID resolution issue was identified 29 times. - None of the tickets were confirmed or graded. Real bugs had to be separated from false positives and overstatements, then ranked by severity. ### Deduplication We deduplicated with Inspect. Each scanner had already tagged its finding with the file most relevant to the issue, so we used those tags as a coarse grouping and then ran an Inspect session over each group to merge true duplicates within it. Losing a real issue to a bad merge is worse than allowing a duplicate to survive, so the prompt was tuned to leave borderline cases alone. After deduplication, around 3,000 distinct tickets remained. > [Figure: security-scanning-public-models/triage-pipeline — agent-flow: From scan to distinct findings] *From roughly 10,000 scanner sessions to 6,000 raw tickets and 3,000 distinct findings. Sessions without findings produce no ticket; duplicate findings are merged conservatively.* ### Confirmation & Prioritization While our scanner agents excelled at analyzing the codebase, they struggled to interpret security impact. Every model we evaluated tended to overstate the severity of its findings. For example, a scanner agent flagged a ‘cross-tenant data leak’ in an internal demo tool, even though the tool never touches production and operates entirely on synthetic data. Therefore, we ran a final validation pass on the deduplicated tickets. We spun up an Inspect session for each ticket, in which the agent attempted to reproduce the issue and regrade its severity to a defensible level. To combat overstatement, we used an adversarial prompt. > “Argue against yourself to make sure the surrounding architecture, product scope, or existing control flow does not already make that attack path unreachable, irrelevant, or indistinguishable from capability the realistic caller already has through other sanctioned means.” Each issue got one of five labels: - **High**: Directly exploitable. An attacker could gain unauthorized capability under realistic conditions. All findings at this level have been patched. - **Medium**: Exploitable in principle, but requires chained preconditions or access an attacker is unlikely to obtain. Blast radius excludes sensitive systems and data. - **Low**: A real issue with marginal practical impact. Exploitation grants little or no meaningful new capability. - **Hardening**: A real pattern, but grants no capability beyond what sanctioned callers already have. A code quality or defense-in-depth observation. - **Not applicable**: Threat model doesn't hold. Note that these labels do not directly map to any industry framework or standard: we picked them to help the agent prioritize consistently. After grading, our 3,074 confirmed tickets skewed heavily toward hardening and low-severity findings, with only seven rated at high-severity (all of which we promptly patched). This was expected: almost every code path in production has room for hardening, while serious vulnerabilities are rare. ## Example Finding: Email Lua Injection One of the more interesting high-severity findings sat at the boundary between Ramp's email-receipt ingestion process and the isolated service that renders those emails into images. The renderer used an embedded Lua engine, and the calling code interpolated the email's HTML body directly into a Lua long-bracket string before evaluation. In simplified form, the vulnerable template looked like this: ```lua local tpl = [===[]===] -- render html into an image ``` The scanner agent noticed that nothing prevented an inbound email body from containing the matching closing sequence; an email body that included `]===]` would terminate the string literal early, and any text after it would be parsed as Lua and executed inside the rendering process. The bug was a composition issue spanning multiple components. The email ingress code, the Python helper that assembled the Lua template, and the Lua rendering script each looked safe in isolation when they were written. The agent identified the interpolation pattern in the templating helper, traced the data flow across systems from inbound mail through receipt processing to the evaluation site, constructed an email body that survived upstream parsing intact, and verified the finding end-to-end inside the Inspect sandbox before filing a ticket. ![Five stages from crafted email to Lua execution inside the isolated renderer.](/research/articles/security-scanning-public-models/email-lua-flow.svg) *An attacker-controlled email body travels through email ingress and receipt processing into a Lua template. The closing sequence terminates the string, allowing code to execute inside the isolated renderer.* We patched this finding immediately upon receiving the report, and human review found no evidence of prior exploit. While the issue existed, the Lua engine was isolated from systems that hold customer data, meaning there was no risk of data loss. ## Using Open-Weight Models Our production scan used GPT-5.5 with high reasoning effort. At full price, a scan this size can easily exceed \$20,000 USD. Ramp helps teams [get more out of every AI dollar](https://ramp.com/ai-cost-monitoring), and we apply the same approach to our internal security scans: we wanted to know whether finding high-severity vulnerabilities actually requires an expensive frontier model. Kimi K2.6 and DeepSeek V4 Pro, served by providers like [Fireworks](https://fireworks.ai/models), are strong reasoners and roughly five times cheaper per token than GPT-5.5-high. Using the same prompt and harness from our initial scan, we measured each model's discovery rate on our confirmed high-severity findings. > **Model settings:** DeepSeek V4 Pro was run at high reasoning effort. Kimi K2.6 was run with thinking enabled. ### Chart > [Figure: security-scanning-public-models/detection-rate — chart: Detection rate by security issue] ### Data > [Figure: security-scanning-public-models/detection-data — table] *Detection rate by security issue, with 25 attempts per model per issue. ISS-1: Configuration; ISS-2: API logic; ISS-3: Authentication; ISS-4: User management; ISS-5: Payments; ISS-6: Integrations; ISS-7: Access control.* > [Figure: security-scanning-public-models/model-summary — stat-grid: Overall detection rate] GPT-5.5 is the strongest scanner overall, and the only model to identify a couple of the more subtle bugs. Kimi K2.6 trails in aggregate but stays close on most of the seven, and on one issue, it meaningfully outperforms GPT-5.5. DeepSeek V4 Pro is weaker across the board but still recovers a few bugs reliably. On balance, the hard cases reward a frontier model, but cheaper open-weight models still find high-severity security issues in production code at meaningful rates. ## Learnings Every meaningful surface in our backend has been audited by an adversarial agent. Our system isn’t perfectly efficient, but the simple brute-force approach has many advantages: it’s easy to set up, easy to scale, model-agnostic, and requires minimal maintenance. A custom-engineered security harness may give better results in the short term, but requires continuous refactoring to maintain that edge as model intelligence and behavior evolve. Most of the vulnerabilities we found aren't intellectually difficult. They survive in production because no team has the bandwidth to closely read an entire codebase by hand. A reasonably capable model with enough passes will surface them. At high parallelism, frontier model agent sessions get expensive fast. If we built this again, we'd lean more on open-weight models, closely monitor token spend, and use flex-tier APIs where possible. ## Closing Thoughts Ramp's security work exists to earn and keep the trust of Ramp customers. As AI capabilities expand, so must the scope of that work. Frontier models can now identify and exploit issues at the level of a top human researcher, and our security program should account for these new capabilities. We encourage software teams to run similar scans against their own code. If you have access to a background coding agent and an inference budget, you can build a version of this. Alternatively, [Ramp is hiring](https://jobs.ashbyhq.com/ramp?utm_source=RampLabs). Come build with us. Article by Alex Levinson [@a_levitator](https://x.com/a_levitator) ## Citation ## Citation See article metadata for citation details.