24 popular npm packages, each pinned to a version with a known bug, in 3 suites. Every tool gets the same seed inputs, token dictionary, input size limit and time budget, and one shared oracle decides when the bug is found. darwin arm64, 11 cores, Node v26.13.0; Termo 0.1.0, Jazzer.js 4.0.0, fast-check 4.10.2.
Ranking
Each tool fuzzes each library 5 times: 5 separate runs of 60 seconds, each with a different random seed. On every library, the tools are ranked from 1 (best) to 4: first by how many of their 5 runs found the bug, then, to break ties, by the median time those runs took to find it (a run that never found it counts as the full 60 seconds). Tools with exactly the same result share the rank. A suite's score is a tool's average rank over that suite's libraries, and the overall score is the average of the suite scores, so a suite with many libraries counts the same as a small one. 1.00 means best on every library; 4.00 means last on every one.
#
Tool
Score
Slow inputs
Security
Robustness
Runs that found the bug
Best on
1
Termo
1.79
1.11
2.25
2.00
118 of 120
13 of 24 libraries
2
Random baseline
2.11
2.94
1.60
1.80
80 of 120
10 of 24 libraries
3
fast-check
2.84
3.17
2.55
2.80
75 of 120
1 of 24 libraries
4
Jazzer.js
3.26
2.78
3.60
3.40
99 of 120
0 of 24 libraries
Time to find the bugs
For each kind of bug, the share of runs that had found it by each moment, over all the libraries of that kind. A line that rises early and ends high finds more bugs, sooner. Hover for the numbers.
Slow inputs (performance bugs)Inputs that make a call take far longer than normal: share of runs that had found the bug, by time (9 libraries, 5 runs each). Higher and further left is better.
TermoJazzer.jsfast-checkRandom baseline
Table
Tool
Runs
Found by 1 s
by 10 s
by 60 s
Termo
45
71%
91%
98%
Jazzer.js
45
11%
42%
56%
fast-check
45
11%
22%
22%
Random baseline
45
18%
33%
33%
SecurityInputs that pollute Object.prototype: share of runs that had found the bug, by time (10 libraries, 5 runs each). Higher and further left is better.
TermoJazzer.jsfast-checkRandom baseline
Table
Tool
Runs
Found by 1 s
by 10 s
by 60 s
Termo
50
98%
98%
98%
Jazzer.js
50
84%
94%
98%
fast-check
50
78%
80%
80%
Random baseline
50
80%
80%
80%
Robustness (crashes)Parser exceptions other than the documented one for bad input: share of runs that had found the bug, by time (5 libraries, 5 runs each). Higher and further left is better.
TermoJazzer.jsfast-checkRandom baseline
Table
Tool
Runs
Found by 1 s
by 10 s
by 60 s
Termo
25
88%
100%
100%
Jazzer.js
25
76%
100%
100%
fast-check
25
72%
100%
100%
Random baseline
25
80%
100%
100%
Against Jazzer.js, Termo was significantly faster to find the bug on 19 libraries and significantly slower on 0 (exact permutation test over the runs, p ≤ 0.05).
Against fast-check, Termo was significantly faster to find the bug on 12 libraries and significantly slower on 7 (exact permutation test over the runs, p ≤ 0.05).
Against Random baseline, Termo was significantly faster to find the bug on 10 libraries and significantly slower on 9 (exact permutation test over the runs, p ≤ 0.05).
Termo was improved after looking at the results of rounds 1 and 3 (marked "tuned on"). The other rounds, 10 libraries, ran on a Termo that had never seen them; it ranked first on 5 of those. Termo's other setups (time and coverage on every library; a grammar of the format, which is more knowledge than the others get) are shown but not ranked.
Slow inputs
Regular expression backtracking and unbounded work: an input that makes a call take far longer than normal.
marked@4.0.9 · CVE-2022-21681 · round 1, tuned on
Regular expression denial of service in link reference definitions.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
4 of 5
26 s
277
Termo with a grammar
2 of 5
> 60 s
216
Jazzer.js
0 of 5
> 60 s
19k
fast-check
0 of 5
> 60 s
2.1k
Random baseline
0 of 5
> 60 s
38k
ms@0.7.0 · CVE-2015-8315 · round 1, tuned on
Regular expression denial of service in duration parsing.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
1.2 s
2.7k
Termo with a grammar
5 of 5
15 s
169
Jazzer.js
0 of 5
> 60 s
709k
fast-check
0 of 5
> 60 s
3.0k
Random baseline
0 of 5
> 60 s
226k
semver@5.7.1 · CVE-2022-25883 · round 1, tuned on
Regular expression denial of service in range parsing.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.2 s
203
Termo with a grammar
5 of 5
0.2 s
149
Jazzer.js
5 of 5
1.5 s
38k
fast-check
0 of 5
> 60 s
2.9k
Random baseline
0 of 5
> 60 s
101k
jpeg-js@0.3.7 · CVE-2020-8175 · round 1, tuned on
Unbounded memory use for images that claim huge dimensions.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.3 s
220
Jazzer.js
5 of 5
0.6 s
221
fast-check
5 of 5
3.3 s
7.7k
Random baseline
5 of 5
0.5 s
5.0k
ansi-regex@5.0.0 · CVE-2021-3807 · round 2
Regular expression denial of service when matching escape sequences.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
1.0 s
2.2k
Jazzer.js
0 of 5
> 60 s
665k
fast-check
0 of 5
> 60 s
2.9k
Random baseline
0 of 5
> 60 s
204k
color-string@1.5.4 · CVE-2021-29060 · round 2
Regular expression denial of service in hwb() parsing.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
1.0 s
14k
Jazzer.js
5 of 5
15 s
468k
fast-check
0 of 5
> 60 s
3.0k
Random baseline
5 of 5
7.5 s
187k
glob-parent@5.1.1 · CVE-2020-28469 · round 3, tuned on
Regular expression denial of service on enclosure patterns.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.2 s
25
Jazzer.js
4 of 5
16 s
25k
fast-check
5 of 5
0.1 s
3
Random baseline
5 of 5
0.7 s
780
trim-newlines@3.0.0 · CVE-2021-33623 · round 3, tuned on
Regular expression denial of service when trimming trailing newlines.
Regular expression denial of service in Link header parsing.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.2 s
180
Jazzer.js
5 of 5
1.4 s
35k
fast-check
0 of 5
> 60 s
2.5k
Random baseline
0 of 5
> 60 s
58k
Security
Prototype pollution: an input that makes the library write to Object.prototype.
ini@1.3.5 · CVE-2020-7788 · round 1, tuned on
Prototype pollution through section names.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.4 s
15k
Termo with a grammar
5 of 5
0.4 s
91
Jazzer.js
5 of 5
0.7 s
31k
fast-check
5 of 5
0.5 s
1.1k
Random baseline
5 of 5
0.5 s
24k
minimist@1.2.0 · CVE-2020-7598 · round 1, tuned on
Prototype pollution through option names.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
715
Termo with a grammar
5 of 5
0.1 s
71
Jazzer.js
5 of 5
0.7 s
36k
fast-check
5 of 5
0.1 s
33
Random baseline
5 of 5
0.1 s
1.3k
set-value@2.0.0 · CVE-2019-10747 · round 2
Prototype pollution through property paths.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
38
Jazzer.js
5 of 5
0.3 s
1.7k
fast-check
0 of 5 (5 blocked by a hang)
> 60 s
0
Random baseline
0 of 5 (5 blocked by a hang)
> 60 s
0
dot-prop@4.2.0 · CVE-2020-8116 · round 2
Prototype pollution through property paths.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
25
Jazzer.js
5 of 5
0.3 s
1.5k
fast-check
5 of 5
0.1 s
9
Random baseline
5 of 5
0.1 s
37
lodash@4.17.15 · CVE-2020-8203 · round 3, tuned on
Prototype pollution in zipObjectDeep.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
35
Jazzer.js
5 of 5
0.5 s
817
fast-check
5 of 5
0.1 s
3
Random baseline
5 of 5
0.1 s
13
deep-extend@0.5.0 · CVE-2018-3750 · round 3, tuned on
Prototype pollution when merging parsed JSON.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.3 s
14k
Jazzer.js
5 of 5
3.8 s
158k
fast-check
0 of 5
> 60 s
3.0k
Random baseline
0 of 5
> 60 s
118k
y18n@4.0.0 · CVE-2020-7774 · round 3, tuned on
Prototype pollution through the locale name.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
203
Jazzer.js
5 of 5
0.6 s
11k
fast-check
5 of 5
0.1 s
158
Random baseline
5 of 5
0.1 s
199
jsonpointer@4.0.1 · CVE-2021-23807 · round 4
Prototype pollution through JSON pointers.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
472
Jazzer.js
5 of 5
0.3 s
4.7k
fast-check
5 of 5
0.1 s
36
Random baseline
5 of 5
0.1 s
689
flat@5.0.0 · CVE-2020-36632 · round 4
Prototype pollution when unflattening keys.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
4 of 5 (1 blocked by a hang)
0.1 s
63
Jazzer.js
4 of 5 (1 blocked by a hang)
0.3 s
784
fast-check
5 of 5
0.1 s
7
Random baseline
5 of 5
0.1 s
34
dot-object@2.1.2 · CVE-2021-23383 · round 4
Prototype pollution through dotted paths.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
18
Jazzer.js
5 of 5
0.3 s
2.5k
fast-check
5 of 5
0.1 s
6
Random baseline
5 of 5
0.1 s
6
Robustness
Parser crashes: an exception other than the one the parser documents for bad input.
yaml@1.10.0 · crash · round 3, tuned on
Exceptions other than YAMLError on malformed YAML.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.8 s
751
Jazzer.js
5 of 5
1.0 s
4.1k
fast-check
5 of 5
0.2 s
221
Random baseline
5 of 5
0.1 s
124
esprima@4.0.1 · crash · round 3, tuned on
Exceptions other than its parse errors on malformed JavaScript.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
433
Jazzer.js
5 of 5
1.1 s
6.4k
fast-check
5 of 5
0.4 s
899
Random baseline
5 of 5
0.1 s
410
json5@1.0.1 · crash · round 3, tuned on
Exceptions other than SyntaxError on malformed JSON5.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.4 s
123
Jazzer.js
5 of 5
0.5 s
4.7k
fast-check
5 of 5
2.5 s
1.2k
Random baseline
5 of 5
4.3 s
145
@iarna/toml@2.2.5 · crash · round 4
Exceptions other than TomlError on malformed TOML.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
1.1k
Jazzer.js
5 of 5
0.7 s
12k
fast-check
5 of 5
2.3 s
2.9k
Random baseline
5 of 5
0.4 s
21k
fast-xml-parser@3.21.0 · crash · round 4
Exceptions other than its plain Errors for malformed XML.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
5 of 5
0.1 s
6
Jazzer.js
5 of 5
0.3 s
10
fast-check
5 of 5
0.1 s
1
Random baseline
5 of 5
0.1 s
2
Method
Bugs are the published ones (CVEs) for the slow-input and security libraries. A slow input must be slow twice in a row, and a watchdog catches inputs that hang.
Robustness libraries were kept only if Jazzer.js or the random baseline crashed them while vetting, so their choice does not favor Termo.
Each run is a separate process, and its time to find the bug includes startup. A run stopped by a different bug (a call that never returns) counts as not found.
Five short runs per tool and library tell large differences apart, not small ones. Code, libraries and raw results are in bench/ in the Termo repository.