← 返回列表

Fuzzers vs. AI: Comparing Fuzzilli and BigSleep on JavaScript Engines

单站 Google Project Zero 入库 2026-11-17T04:15:28

Summary

Project Zero compared two automated tools used by Google to find vulnerabilities in Chrome’s JavaScript engine, V8, on a regular basis. Fuzzilli is a generative JavaScript engine fuzzer and BigSleep is an LLM-based bug finding system. In the past six months, BigSleep found more vulnerabilities than Fuzzilli (45 versus 37) with a 25% overlap in the specific bugs reported. Since this comparison methodology has a tendency to underestimate overlap, we also performed dedicated experiments to test theoretical overlap in vulnerabilities reported since 2023. These experiments estimated overlap at roughly 60%. Regardless of which value is accurate, a substantial portion of vulnerabilities were discoverable by only LLMs or only fuzzing. While it is extremely promising that LLMs have become able to find so many vulnerabilities so quickly, fuzzing and LLMs are still complementary approaches. We recommend vendors perform both fuzzing and agentic analysis of their code to maximize the number of vulnerabilities that get reported before they impact users.

Introduction

Fuzzing has long been successfully applied to software projects such as Chrome or V8 (Chrome’s JavaScript engine) to detect bugs. Recently, the arrival of powerful LLMs able to read, reason about, and write code is causing a transformative change to software development and software security. With hundreds, sometimes thousands of new bugs and vulnerabilities having been discovered in well-fuzzed targets, a natural question arises: how do new LLMs compare to “traditional” fuzzers? Are they starting to replace them or do the two approaches complement each other? To help answer this, we look at three key questions:

  1. How many bugs does each approach find in the real world?
  2. What is the overlap between the two approaches?
  3. What is the “shape” of bugs discovered by one system but not the other?

For this blog post, we will limit ourselves to two concrete systems: Fuzzilli, a JavaScript engine fuzzer, and BigSleep, an agentic bug finding system. Both of these systems run in production and automatically look for bugs in newly added code, which makes them fairly directly comparable. Specifically, Fuzzilli runs continuously across many instances, with the first ones usually picking up new changes within a few hours, while BigSleep currently runs once per day and is prompted to review new commits and write proof-of-concept testcases for any bugs it discovers. Both systems report bugs through Clusterfuzz, which will do basic crash triaging, then file bug reports in Chrome’s issue tracker.

We also limit ourselves to V8 as a target, a high-performance JavaScript and WebAssembly engine with a sophisticated and mature testing and fuzzing setup. In particular, Fuzzilli is a special-purpose fuzzing tool designed to find bugs in these engines. The usually complicated nature of these bugs makes for an interesting “playground” to compare the two approaches’ ability to find deep and complex bugs in well-tested software. We leave it to future work to perform similar analysis on a broader set of targets, fuzzers, and agentic bug finding systems.

We will now look into each of the three main questions in a dedicated section. Each section will first summarize the general approach, then describe the methodology in detail (often as bullet points, so you can skip those blocks if you don’t care too much about the details) before exploring the results. We conclude with an attempt at interpreting the results to get a picture of the current state of fuzzing vs. AI.

Part 1: Production Statistics

This is the simplest part to analyze, although with some caveats we’ll see in a bit. The basic idea is to query issues from Chromium’s bugtracker (where V8 bugs are tracked) and extract various statistics from there. This is especially easy since there’s a dedicated hotlist for each of the two systems. We limit our analysis to the six months from April to September 2026 as the automatic BigSleep pipeline only became active in April this year. As such, our raw data comes from the following two bug tracker queries:

From there, the raw numbers of bug reports are:

Small win for BigSleep. However, this includes all types of bugs, and BigSleep initially also found a lot of configuration and integration issues in V8, for example crbug.com/503869331 where the unsafe flag blocking mechanism didn’t work properly yet. These are legitimate issues, but less relevant for our comparison since there’s little real-world security impact. One easy way to get rid of these is to look specifically at vulnerabilities by limiting the results to type:vulnerability:

Suddenly a clear win for the fuzzers? Not quite. There are still some problems with this data, notably duplicates and unactionable reports (e.g. bugs closed as WAI). At first, this seems like a simple thing to fix: just include another constraint such as status:(fixed | verified). Unfortunately, things are a bit more complicated. Take for example Fuzzilli bug crbug.com/511935486. The events on this bug are basically:

As such, this bug received zero developer interactions. Most likely what happened here is that this bug was a duplicate of another bug, crbug.com/511210507, but Clusterfuzz didn’t realize it as the crash signature (derived from the stack trace) was slightly different. When the other bug received a fix, the testcase stopped reproducing on Clusterfuzz, which closed the issue.

Maybe the best thing we can do is to look at “actionable” bugs, defined as those that are not marked as duplicates or WAI and that received at least one fix commit referencing them. Then, looking at only Type-Vuln bugs filters out a lot of the less-relevant BigSleep bugs, even though these were “actionable”. With that, the final picture looks like this:

So in terms of vulnerabilities, it’s roughly a tie (small advantage BigSleep) at the moment: 45 vs. 37.

This naturally raises the question of cost: how expensive is each of the two approaches and what does that mean in terms of cost-per-bug? Unfortunately, it’s challenging to make a comparison at this stage in the research. While some effort has been dedicated to making Clusterfuzz more efficient, similar effort has not been expended for Big Sleep and neither system is particularly calibrated for an optimal cost-per-bug ratio.

Part 2: Overlap

The next question is how much these approaches overlap each other. For this, our approach is twofold: we first attempt to find an answer in the bugtracker, by trying to identify bugs discovered by both systems. Next, we run a dedicated experiment in which we let BigSleep try to rediscover old Fuzzilli bugs.

Bugtracker Overlap

The basic idea is to see which bugs were found by both systems, so we’re effectively looking for duplicate bug reports. Similar to part 1, there are a few steps we need to take to ensure we get a somewhat accurate picture:

With all the different types of deduplication, we get this final picture:

Again, looking at just the actionable vulnerability reports (likely the best measure), we see an overlap of roughly 25% (20 out of a total of 81), with a slight majority of these found first by Fuzzilli and later by BigSleep.

Overlap Experiment

The second way to measure the overlap is to run a dedicated experiment in which we try to re-discover bugs found by one system in the past using the other. Specifically, here we try to re-discover old Fuzzilli bugs using a BigSleep setup that’s as close to the production configuration as possible. For each bug we collect the commit that (likely) introduced the bug, and ask the agent to review those changes just as in the production CI environment. Note that the other direction, trying to rediscover past BigSleep bugs with Fuzzilli is more difficult to set up, since fuzzers cannot easily be given a similar “starting point”. Moreover, for many of these bugs we already know the outcome of such an experiment: the majority of “historic” BigSleep bugs (e.g. those from 2025) were found after fuzzers like Fuzzilli had already been fuzzing vulnerable builds for months, sometimes years. So they would most likely also not rediscover them in an experiment (except perhaps in a few cases where Fuzzilli was since updated to support new language features). Similarly, in production, BigSleep only reports crashes with a delay of 1-2 days (as there’s currently only one run per day). As such, fuzzers also get a head-start there.

In more detail, here is how the experiment is set up:

Here are the results from the experiment:

Starting from just the commits that (likely) introduced the bugs, BigSleep was able to re-discover \~62% of the original bugs, while finding a different (but by now fixed) bug for another \~12% of commits. On around 26% of the commits it failed to find any bugs.

Memorization

One thing to worry about with experiments such as this one is memorization: the LLM remembering specific bugs from its (pre-)training data and as such having an easier time “re-discovering” them during evaluation. We can try to get some idea of the effect of this by looking at the success rates for bugs from different time spans, e.g. every half-year period for which we have bugs in the data set. If there was memorization, we’d expect higher success rates (both for a single attempt and for the combined run) on bugs prior to the learning data cutoff of the model. Here are the results for our experiment:

For the model used in this experiment (Gemini 3.6 flash), the knowledge cutoff date is March 2026. Combining that with the fact that in early 2026, Chrome bugs would only become publicly visible after \~3 months, the effective knowledge cutoff for Fuzzilli bugs should be around Jan 2026. Looking at the data, there seems to be no statistically significant amount of memorization pre-2026. We note, however, that this may be different for other models or future model generations and as such recommend that this be re-evaluated for similar experiments in the future.

Difficulty

A follow-up question we might ask is how the overlap compares with “difficulty”. For example, in the best case a system like BigSleep would easily find issues that fuzzers struggle with and vice versa. In that case, budget that was previously spent on long fuzzing sessions may now be re-allocated and put to different uses. Here are the results of analysing this, where “bug difficulty” is measured by the time it took Fuzzilli to discover the bug:

While we do get a statistically significant result here (although not of particularly great magnitude), it actually points in the other direction! Bugs that are hard to find for a fuzzer are likely also hard to find for an AI agent. As such, those fuzzers should probably be kept running for now.

Components

Another thing we can look at is a per-component breakdown of the bugs and their rediscovery rates:

Interestingly, the overlap is significantly smaller for compiler bugs. This is the area where Fuzzilli probably specializes most in, which may explain the results. Later on (in part 3), we’ll also develop another hypothesis for why (some) bugs in the compilers may be more difficult to rediscover for BigSleep.

Prompting

A final question is how much the specific prompting played a role in the performance of the AI agent. For that, we run two quick follow-up experiments, only on the bugs that weren’t previously re-discovered and with different prompts:

  1. Instead of starting from commits, we start from the vulnerable file or function. This may be an easier starting point but is still a fairly realistic setup since both can for example be obtained by extracting modified files and functions from a given commit.
  2. Instead of asking the agent to find memory corruption vulnerabilities (what we care most about), we specifically tell it that assertion failures and similar crashes are also desirable.

Running these two experiments with only 5 attempts per bug (instead of the 25 used in the main experiment), the agent re-discovered 10 additional bugs in total: 8 from the different starting points and 7 with the rephrased prompt (with 5 found in both runs).

In total, combining these results with our original experiment, we get the following picture:

So the share of “correct” bugs found climbs to \~70%, and we find some kind of bug for \~84% of starting points. However, for fairness it should be noted that equivalent tweaks to a fuzzer could likely also result in higher discovery rates for certain bugs, so the results of this follow-up experiment should be treated with some caution.

Summary

The main results from our two angles are:

  1. In production, the overlap is noticeable but not substantial at around 25%.
  2. In our experiment, overlap was around 60% of historic Fuzzilli bugs (70% with additional runs that used different prompts).
  3. The overlap is not uniformly distributed:
  4. Bugs that are difficult for a fuzzer to find are more likely to also be difficult to find for an agent, so there’s more overlap on “easy” bugs.
  5. The overlap seems to be smaller in the JIT Compilers and higher in the rest of the engine.

From just the experiment, we might expect a higher overlap in production. However, there are probably a few factors that prevent(ed) this:

Part 3: Shape of Bugs

In this section, we will first look at three case studies of bugs found by one system but not the other, then attempt to draw conclusions from these and other bugs.

BigSleep: Chrome Bug 449760249 (October 2025)

This bug is a nice example of BigSleep benefiting from the underlying model’s reasoning capabilities to discover a variant that was (likely) too complex to trigger by a fuzzer in a reasonable amount of time. It’s also a good example to show the general working of an agentic bug finding system.

Background: TDZ and Hole Check Elision

Before diving into the bug specifics, we first need to look at some JavaScript internals, specifically variable initialization. In JavaScript, variables declared with the let or const keywords cannot be used before they are initialized, and attempting to do so will lead to an exception (”ReferenceError: Cannot access 'foobar' before initialization”). In V8’s terminology, these variables reside in a “temporal dead zone” (TDZ) until they are initialized. Internally, V8 sets variables in the TDZ to a special sentinel value, the_hole, and then checks for this value when accessing the variable’s content.

To limit the performance impact of these hole-checks, V8’s bytecode compiler implements an optimization (behind the default-enabled --ignition-elide-redundant-tdz-checks flag) to identify redundant checks and subsequently remove them. The idea here is to identify “dominating” checks where it can then be guaranteed that a hole-check has already been performed for a given variable on all possible paths leading to the current position.

This is a security-relevant optimization as an incorrectly removed hole-check would result in the internal the_hole value leaking to JavaScript code, a condition that has historically been exploitable (although new mitigations for this bug class have since shipped).

The Original Vulnerability

The original bug (issue 40071027 - discovered two years earlier by Fuzzilli!) involved do-while loops containing a break statement. In a do-while loop, the loop condition evaluates after the body. If the condition contains an assignment to a variable, the bytecode compiler will mark the variable as initialized. However, if a break statement executes in the loop body in the first iteration, then the condition will never execute. The bytecode compiler originally failed to account for this, leading to an incorrectly omitted hole check and subsequent leak of the_hole value.

To fix this, the implementation of BytecodeGenerator::VisitDoWhileStatement was modified in the following way:

- VisitForTest(stmt->cond(), ...);
+ if (!loop_builder.break_labels()->empty()) {
+   HoleCheckElisionScope elider(this);
+   VisitForTest(stmt->cond(), ...);
+ } else {
+   VisitForTest(stmt->cond(), ...);
+ }

Specifically, the patch works by creating a HoleCheckElisionScope for processing the condition code in case the loop contains break statements. This means that any updates to the TDZ state of variables will be discarded once the scope object is destroyed. As such, code after the loop will see the previous TDZ state (which is the correct behavior in this case).

Discovery of the Variant

The BigSleep agent began its analysis by examining the patch to VisitDoWhileStatement, realizing that the fix relied on loop_builder.break_labels()->empty() to decide if a protective HoleCheckElisionScope was required. It then began creating and testing various hypotheses for scenarios in which no break statement was registered with a given loop but the condition would still be skipped at runtime. For example, it hypothesized that a try-catch in combination with a throw inside the loop might confuse the hole-check elision logic. It then investigated how try-catch was implemented in the bytecode compiler and concluded that this was handled correctly.

Eventually, the agent started investigating labeled breaks, which allow directly breaking out of inner loops. It hypothesized that these might not be handled correctly and continued to investigate how these were being processed, for example by studying the logic in BytecodeGenerator::VisitBreakStatement. This time, it seemed to have identified a real flaw: in the case of nested breaks, the inner loop was apparently not being informed of the existence of a break statement! It therefore formulated the following vulnerability hypothesis:

  1. If a labeled break inside a do-while loop targets an outer scope, the inner do-while loop’s LoopBuilder is not notified.
  2. Consequently, loop_builder.break_labels()->empty() remains true.
  3. Therefore, when generating the code for the loop condition, the HoleCheckElisionScope introduced by the patch is not used.
  4. The condition is bypassed at runtime, but the bytecode compiler erroneously removed a later hole-check, leading to a leak of the_hole to JavaScript.

The agent then began writing a testcase to prove the existence of this bug. The initial attempt with simply two nested loops and a label failed, however, as the bytecode compiler would add another protective scope object for the labeled block. To work around this, the agent decided to use a switch statement. When handling a switch statement, the bytecode compiler uses a HoleCheckElisionMergeScope to compute the intersection of initialization states across all branches. By ensuring that the variable was initialized in a default case, the merge logic then concluded that the variable was correctly initialized on all paths.

Proof of Concept

With that, the agent had everything in place to produce a testcase to demonstrate the vulnerability. Below is the agent’s PoC:

// Flags: --ignition-elide-redundant-tdz-checks
function trigger(cond) {
  {
    target: switch(1) {
      case 1:
        do {
          // Labeled break targeting the switch. The do-while LoopBuilder remains unaware.
          if (cond) break target;
        } while((x=1) && false); // Condition performs a check/assignment on x.
        break;
      default:
        x=1; // Ensure x is considered initialized across all switch branches for merging.
    }
    // Generator incorrectly believes x is checked/initialized. Elides TDZ check.
    x + 1;
    let x;
  }
 }
 trigger(true);

Here is how this testcase works:

  1. trigger(true) executes, entering case 1 of the switch.
  2. The do-while loop executes break target.
  3. Control flow jumps to the end of the switch, bypassing the do-while condition where x is assigned.
  4. At runtime, x remains uninitialized (holding the_hole).
  5. Because the TDZ check was elided at compile time, the engine leaks the internal the_hole value to JavaScript code in the x + 1 expression, leading to a potentially exploitable condition.

We can see how this bug would be difficult for a fuzzer to find since the testcase is quite complex. While a coverage-guided fuzzer would have received rewards for triggering individual code paths (such as the handling of labeled break, the switch statement, the do-while loop, etc. in the parser), there is no additional reward for putting these together “in the right way” and so ending up with a testcase like this is quite difficult.

BigSleep: Chrome Bug 443765373 (September 2025)

This bug is an example of a relatively simple bug type that is challenging to discover with standard fuzzing approaches.

Background: Exception handling in V8

To understand this bug, it is necessary to briefly look into exception handling in V8. Take the following example JavaScript code:

function simple() {
  try {
    throw 42;
  } catch (e) {
    return e;
  }
}

Here, we have a function consisting of a try-catch block in which an exception is raised. Running this in d8 with --print-bytecode, we can see the internal bytecode that the engine generates to execute the JavaScript code:

[generated bytecode for function: simple]
         0x2ad5001000e4 @    0 : 1b ff f9  Mov <context>, r0
         0x2ad5001000e7 @    3 : 0d 2a     LdaSmi [42]
         0x2ad5001000e9 @    5 : b5        Throw
         0x2ad5001000ea @    6 : d1        Star1
         0x2ad5001000eb @    7 : 8d f8 00  CreateCatchContext r1, [0]
         0x2ad5001000ee @   10 : d2        ...
         ...
         0x2ad5001000f7 @   19 : b7        Return
Handler Table (size = 16)
   from   to       hdlr (prediction,   data)
  (   3,   6)  ->     6 (prediction=1, data=0)

For the purpose of this bug, the relevant part is mostly the last line of the output. Here, the bytecode essentially says “if there is an exception while the PC is between offset 3 and 6 in this bytecode, then execution should continue at offset 6 (the start of the exception handler)”.

The Bug

The agent spotted the bug while looking at the following function, responsible for writing the handler table data structure in memory:

void HandlerTable::SetRangeHandler(int index, int handler_offset,
                                   CatchPrediction prediction) {
  int value = HandlerOffsetField::encode(handler_offset) |
              HandlerWasUsedField::encode(false) |
              HandlerPredictionField::encode(prediction);
  int offset = index * kRangeEntrySize + kRangeHandlerIndex;
  Memory<int32_t>(raw_encoded_data_ + offset * sizeof(int32_t)) = value;
}

Here, the int offset is being combined with a few other bits of data into a new int value. As such, the handler_offset cannot have the full integer range in the new field, possibly leading to integer truncation issues. The agent then investigated the specific bounds of these values and realized that the HandlerOffsetField had only 28 bits (allowing for 256MB) while the maximum bytecode size was 512MB! From there it was relatively simple to craft a testcase that crashes, simply by creating a huge function (resulting in >256MB of bytecode) with an exception handler at the end, then triggering an exception inside of it. The result is a jump to a wrong bytecode offset, a primitive that’s easily exploited. Here is the relevant part of the testcase:

// yield statements turn into a large chunk of bytecode
let kNumYields = 500000;
let body = `
  if ("foo" === "bar") {
    ${"yield* 42;".repeat(kNumYields)}
  }
  try {
    throw 42;
  } catch (e) {
     // Will never get here
  }
`;

Such a testcase would again be difficult to craft by a fuzzer as the fuzzer isn’t rewarded for creating large code (at least not with generic coverage feedback). In fact, the fuzzer is likely even actively discouraged from finding this bug since generating and executing large bytecode takes some time (in debug builds, the testcase generated by the agent needs a few seconds to trigger) and would therefore probably have led to timeouts during fuzzing.

Fuzzilli: Chrome Bug 40076141 (November 2023)

This bug is a good example of a type of bug that is likely difficult to find just by looking at (one part) of the source code. It was originally discovered by Fuzzilli and not rediscovered by any of the experiments from part 2.

Background: Value Representation in V8

Consider the following piece of JavaScript code:

let i = 42;
let f = 13.37;
let s = "foobar"
i = s;

Here we create multiple variables holding values of different types. Note, however, that we never declare the type of a variable, and that variables can hold values of different types throughout their lifetime (i first contains an integer, later a string). What we see here is that JavaScript is a dynamically typed language, and so variables (and also object fields, etc.) have no fixed type. However, the actual runtime values still need to have a type: the engine somehow needs to know that foobar is a string value and that certain operations (e.g. substring search) can therefore be performed on it. Conceptually, this is possible by representing runtime values with a struct such as

enum JSType {
    JS_TYPE_INT,
    JS_TYPE_FLOAT,
    JS_TYPE_STRING,
    ...
};
struct JSValue {
    JSType type;
    // Either the raw value or a pointer to another object, e.g a JSString.
    uintptr_t value;  
};

However, this is somewhat inefficient since we now need something like 16 bytes to represent a simple 4-byte integer at runtime. As such, modern JS engines use clever tagging schemas, which rely on unused pointer bits to encode type information. Specifically, in V8 the JSValue type is a tagged pointer (or just tagged for short) which stores one bit of type information in the least-significant bit:

As such, to figure out the type of a value at runtime, the engine first looks at the least-significant bit. If that is zero, the value can be “untagged” to obtain the integer value by right-shifting by one. On the other hand, if the bit is one, the value is a valid pointer to an object that contains further type information. For more details on V8’s pointer tagging scheme see also this blog post.

The Testcase

Let’s start by looking at the testcase produced by the fuzzer, which isn’t overly long:

function f0() {
    let v2 = 1;
    for (let i3 = v2; i3 >= v2; i3 += v2) {
        v2 = v2 && i3;
    }
}
f0();

The bug is caused during optimization of the loop by the Maglev compiler, specifically due to a conflict between two stages of the compiler pipeline: the initial graph builder and the later MaglevPhiRepresentationSelector.

The Graph Builder

The graph builder is responsible for building the initial Maglev intermediate representation (IR) of the code. For that, it consumes both the input code (in the form of V8 bytecode) as well as runtime feedback that was collected while the code executed in lower tiers such as the interpreter. When processing the loop in this testcase, the following happens:

  1. Both i3 and v2 are modified during the loop. Since the IR uses static single-assignment (SSA) form, these turn into Phi nodes, which make the code look like every variable only has a single assignment :)
  2. For the i3 >= v2 comparison, the runtime feedback states that i3 was so far always a Smi and therefore speculates that this will be the case in the future. It guards this assumption by emitting a CheckedSmiUntag, which will extract the raw int from the tagged value and will bail out to the interpreter if the input is ever not a Smi.
  3. For the following operations on v2, the compiler can then rely on the existing check and emit UnsafeSmiUntag nodes, which avoid the runtime checks but are only safe to use if the input is guaranteed to be a Smi.

The resulting code for the loop now looks roughly like this (in Maglev pseudocode):

loop:
  i3 <- Phi(1, i3_next)                   // loop variable, tagged
  v2 <- Phi(1, v2_next)                   // loop variable, tagged
  i3_int <- CheckedSmiUntag(i3)           // deopt if i3 is not a Smi
  v2_int <- CheckedSmiUntag(v2)           // deopt if v2 is not a Smi
  if i3_int < v2_int: break
  v2_next <- Phi(i3, v2)                  // result of `v2 && i3`: "known Smi"
  v2_next_int <- UnsafeSmiUntag(v2_next)  // no (re-)check needed
  i3_next_int = i3_int + v2_next_int
  i3_next <- Int32ToNumber(i3_next_int)   // tag the int32 result for the phi

The MaglevPhiRepresentationSelector

The purpose of this optimizer is to identify Phi nodes where all inputs have the same raw representation (e.g. int32), and if so convert the Phi to that representation to avoid future untagging operations. For the code above, the code sees that both inputs to i3 are int32 and therefore decides to convert the Phi to int32 representation. Following CheckedSmiUntag operations then become a no-op and can be removed, while consumers that need a tagged value now require a conversion step. With that, the final code now looks like this:

loop:
  i3 <- Phi(1, i3_next)                   // all inputs int32, so i3 becomes a raw int32
  v2 <- Phi(1, v2_next)                   // stays tagged
  v2_int <- CheckedSmiUntag(v2)
  if i3 < v2_int: break                   // CheckedSmiUntag(i3) removed as a no-op
  i3_tagged = Int32ToNumber(i3)           // HeapNumber if i3 >= 2^30!
  v2_next <- Phi(i3_tagged, v2)           // stays tagged, needs tagged inputs
  v2_next_int <- UnsafeSmiUntag(v2_next)  // HeapNumber pointer treated as a Smi
  i3_next <- i3 + v2_next_int             // tagging no longer needed here

The Conflict

The bug now occurs because of two optimizations interacting badly with each other:

  1. The graph builder uses UnsafeSmiUntag as it believes that the value must be a Smi at this point
  2. The representation selector removes a CheckedSmiUntag as it believes it is no longer required

Specifically, what can happen now is that eventually, i3 becomes larger than the maximum Smi value. At that point, Int32ToNumber can no longer fit the integer into a Smi and instead allocates a HeapNumber object (a container holding a larger number, basically). At this point, however, the UnsafeSmiUntag is no longer safe as the input is now a pointer. This leads to a corrupted pointer value being extracted and subsequently to memory corruption. Note that in the original code, this could not have happened as the HeapNumber would’ve been rejected by the CheckedSmiUntag(i3).

What may have made this bug difficult to find for BigSleep is the fact that both optimizations are correct in isolation. As such, looking at just one of them would’ve made it difficult to spot the issue. The agent would somehow have needed to hypothesize about the other optimization or stumbled upon the relevant code by chance (or exhaustive search).

Overview

Looking at the examples above as well as other bugs found by LLMs, we can see a few clear examples of bugs where an LLM has the advantage:

On the other hand, here are some advantages that fuzzers enjoy (for now):

There is probably also a category of bugs that are hard to find for both approaches: bugs that are very non-local (bad for the agent) and require complex testcases (bad for the fuzzer) to trigger. In this case, a fuzzer would probably discover the individual pieces of the testcase (through coverage feedback) but won’t be able to find the right combination to trigger the bug (as in our first example). A code-reviewing agent, on the other hand, may struggle with the large search space of code fragments that may interact badly with each other (as in our last example). This is only a hypothesis, however, and would require further experiments and data to validate.

Part 4: Conclusion

In this blog post, we’ve compared coverage-guided fuzzers with vulnerability-discovery agents that read and reason about the source code. Specifically, we’ve looked at Fuzzilli and BigSleep, which both run as automated pipelines on V8 to discover newly introduced bugs. Looking at the data from those pipelines, we saw a roughly equal share of vulnerabilities found in the last 6 months (around 40 each), and a moderate overlap in discovered bugs of around 25%. However, in a dedicated experiment, overlap was closer to 60%. For now, this means that the two approaches remain complementary, and specialized fuzzing continues to provide value by finding bugs quickly and identifying certain kinds of issues that are otherwise hard to spot. At the same time, LLMs only very recently became capable of finding bugs in software like V8 at all, and have made tremendous progress since. The model used for our experiment is by now already outdated and outperformed by newer generations. Looking to the future, we expect further model improvements, potentially coupled with better prompting and more flexible harnesses (that e.g. allow the model to build, improve, and run fuzzers itself instead of being limited to pure code auditing), to lead to a growing share of bugs being discovered by agentic systems.

原文链接:https://projectzero.google/2026/10/fuzzing-v-AI.html