Fuzzer benchmark

24 popular npm packages, each pinned to a version with a known bug, in 3 suites. Every tool gets the same seed inputs, token dictionary, input size limit and time budget, and one shared oracle decides when the bug is found. darwin arm64, 11 cores, Node v26.13.0; Termo 0.1.0, Jazzer.js 4.0.0, fast-check 4.10.2.

Ranking

Each tool fuzzes each library 5 times: 5 separate runs of 60 seconds, each with a different random seed. On every library, the tools are ranked from 1 (best) to 4: first by how many of their 5 runs found the bug, then, to break ties, by the median time those runs took to find it (a run that never found it counts as the full 60 seconds). Tools with exactly the same result share the rank. A suite's score is a tool's average rank over that suite's libraries, and the overall score is the average of the suite scores, so a suite with many libraries counts the same as a small one. 1.00 means best on every library; 4.00 means last on every one.

#ToolScoreSlow inputsSecurityRobustnessRuns that found the bugBest on
1Termo1.791.112.252.00118 of 12013 of 24 libraries
2Random baseline2.112.941.601.8080 of 12010 of 24 libraries
3fast-check2.843.172.552.8075 of 1201 of 24 libraries
4Jazzer.js3.262.783.603.4099 of 1200 of 24 libraries

Time to find the bugs

For each kind of bug, the share of runs that had found it by each moment, over all the libraries of that kind. A line that rises early and ends high finds more bugs, sooner. Hover for the numbers.

Slow inputs (performance bugs)Inputs that make a call take far longer than normal: share of runs that had found the bug, by time (9 libraries, 5 runs each). Higher and further left is better.
TermoJazzer.jsfast-checkRandom baseline
0%25%50%75%100%0.1 s1 s10 s60 sTermo 98%Jazzer.js 56%Random baseline 33%fast-check 22%
Table
ToolRunsFound by 1 sby 10 sby 60 s
Termo4571%91%98%
Jazzer.js4511%42%56%
fast-check4511%22%22%
Random baseline4518%33%33%
SecurityInputs that pollute Object.prototype: share of runs that had found the bug, by time (10 libraries, 5 runs each). Higher and further left is better.
TermoJazzer.jsfast-checkRandom baseline
0%25%50%75%100%0.1 s1 s10 s60 sTermo 98%Jazzer.js 98%fast-check 80%Random baseline 80%
Table
ToolRunsFound by 1 sby 10 sby 60 s
Termo5098%98%98%
Jazzer.js5084%94%98%
fast-check5078%80%80%
Random baseline5080%80%80%
Robustness (crashes)Parser exceptions other than the documented one for bad input: share of runs that had found the bug, by time (5 libraries, 5 runs each). Higher and further left is better.
TermoJazzer.jsfast-checkRandom baseline
0%25%50%75%100%0.1 s1 s10 s60 sTermo 100%Jazzer.js 100%fast-check 100%Random baseline 100%
Table
ToolRunsFound by 1 sby 10 sby 60 s
Termo2588%100%100%
Jazzer.js2576%100%100%
fast-check2572%100%100%
Random baseline2580%100%100%

Termo was improved after looking at the results of rounds 1 and 3 (marked "tuned on"). The other rounds, 10 libraries, ran on a Termo that had never seen them; it ranked first on 5 of those. Termo's other setups (time and coverage on every library; a grammar of the format, which is more knowledge than the others get) are shown but not ranked.

Slow inputs

Regular expression backtracking and unbounded work: an input that makes a call take far longer than normal.

marked@4.0.9 · CVE-2022-21681 · round 1, tuned on

Regular expression denial of service in link reference definitions.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 26 s, 9.3k inputsTermo, run 2: 26 s, 7.2k inputsTermo, run 3: not found, 13k inputsTermo, run 4: 2.4 s, 1.5k inputsTermo, run 5: 42 s, 6.9k inputsTermo with a grammarTermo with a grammar, run 1: not found, 14k inputsTermo with a grammar, run 2: not found, 13k inputsTermo with a grammar, run 3: 39 s, 6.8k inputsTermo with a grammar, run 4: 42 s, 7.8k inputsTermo with a grammar, run 5: not found, 14k inputsJazzer.jsJazzer.js, run 1: not found, 1.1M inputsJazzer.js, run 2: not found, 1.2M inputsJazzer.js, run 3: not found, 1.1M inputsJazzer.js, run 4: not found, 1.1M inputsJazzer.js, run 5: not found, 1.1M inputsfast-checkfast-check, run 1: not found, 129k inputsfast-check, run 2: not found, 128k inputsfast-check, run 3: not found, 125k inputsfast-check, run 4: not found, 128k inputsfast-check, run 5: not found, 128k inputsRandom baselineRandom baseline, run 1: not found, 2.3M inputsRandom baseline, run 2: not found, 2.3M inputsRandom baseline, run 3: not found, 2.3M inputsRandom baseline, run 4: not found, 2.3M inputsRandom baseline, run 5: not found, 2.3M inputs
ToolRuns that found itMedian time to find itInputs per second
Termo4 of 526 s277
Termo with a grammar2 of 5> 60 s216
Jazzer.js0 of 5> 60 s19k
fast-check0 of 5> 60 s2.1k
Random baseline0 of 5> 60 s38k

ms@0.7.0 · CVE-2015-8315 · round 1, tuned on

Regular expression denial of service in duration parsing.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.5 s, 1.8k inputsTermo, run 2: 3.2 s, 4.2k inputsTermo, run 3: 2.8 s, 8.7k inputsTermo, run 4: 0.8 s, 3.7k inputsTermo, run 5: 1.2 s, 3.2k inputsTermo with a grammarTermo with a grammar, run 1: 13 s, 4.8k inputsTermo with a grammar, run 2: 27 s, 1.6k inputsTermo with a grammar, run 3: 15 s, 1.2k inputsTermo with a grammar, run 4: 7.5 s, 3.3k inputsTermo with a grammar, run 5: 34 s, 5.7k inputsJazzer.jsJazzer.js, run 1: not found, 42.4M inputsJazzer.js, run 2: not found, 42.2M inputsJazzer.js, run 3: not found, 42.9M inputsJazzer.js, run 4: not found, 43.4M inputsJazzer.js, run 5: not found, 42.6M inputsfast-checkfast-check, run 1: not found, 178k inputsfast-check, run 2: not found, 180k inputsfast-check, run 3: not found, 174k inputsfast-check, run 4: not found, 177k inputsfast-check, run 5: not found, 184k inputsRandom baselineRandom baseline, run 1: not found, 13.5M inputsRandom baseline, run 2: not found, 13.4M inputsRandom baseline, run 3: not found, 13.6M inputsRandom baseline, run 4: not found, 13.6M inputsRandom baseline, run 5: not found, 13.6M inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 51.2 s2.7k
Termo with a grammar5 of 515 s169
Jazzer.js0 of 5> 60 s709k
fast-check0 of 5> 60 s3.0k
Random baseline0 of 5> 60 s226k

semver@5.7.1 · CVE-2022-25883 · round 1, tuned on

Regular expression denial of service in range parsing.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.2 s, 145 inputsTermo, run 2: 0.2 s, 203 inputsTermo, run 3: 0.2 s, 572 inputsTermo, run 4: 0.2 s, 115 inputsTermo, run 5: 0.2 s, 462 inputsTermo with a grammarTermo with a grammar, run 1: 0.1 s, 28 inputsTermo with a grammar, run 2: 0.3 s, 546 inputsTermo with a grammar, run 3: 0.1 s, 149 inputsTermo with a grammar, run 4: 0.2 s, 116 inputsTermo with a grammar, run 5: 0.2 s, 156 inputsJazzer.jsJazzer.js, run 1: 1.7 s, 65k inputsJazzer.js, run 2: 1.5 s, 60k inputsJazzer.js, run 3: 2.7 s, 96k inputsJazzer.js, run 4: 1.3 s, 49k inputsJazzer.js, run 5: 1.5 s, 56k inputsfast-checkfast-check, run 1: not found, 176k inputsfast-check, run 2: not found, 177k inputsfast-check, run 3: not found, 170k inputsfast-check, run 4: not found, 172k inputsfast-check, run 5: not found, 179k inputsRandom baselineRandom baseline, run 1: not found, 6.1M inputsRandom baseline, run 2: not found, 6.1M inputsRandom baseline, run 3: not found, 6.1M inputsRandom baseline, run 4: not found, 5.5M inputsRandom baseline, run 5: not found, 6.0M inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.2 s203
Termo with a grammar5 of 50.2 s149
Jazzer.js5 of 51.5 s38k
fast-check0 of 5> 60 s2.9k
Random baseline0 of 5> 60 s101k

jpeg-js@0.3.7 · CVE-2020-8175 · round 1, tuned on

Unbounded memory use for images that claim huge dimensions.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.3 s, 220 inputsTermo, run 2: 0.4 s, 139 inputsTermo, run 3: 0.3 s, 54 inputsTermo, run 4: 0.7 s, 1.0k inputsTermo, run 5: 0.3 s, 261 inputsJazzer.jsJazzer.js, run 1: 0.8 s, 1.1k inputsJazzer.js, run 2: 0.5 s, 63 inputsJazzer.js, run 3: 0.6 s, 1.3k inputsJazzer.js, run 4: 0.6 s, 14 inputsJazzer.js, run 5: 0.6 s, 221 inputsfast-checkfast-check, run 1: 3.3 s, 28k inputsfast-check, run 2: 3.4 s, 26k inputsfast-check, run 3: 1.1 s, 7.0k inputsfast-check, run 4: 4.0 s, 34k inputsfast-check, run 5: 1.4 s, 10k inputsRandom baselineRandom baseline, run 1: 0.3 s, 1.4k inputsRandom baseline, run 2: 0.5 s, 10k inputsRandom baseline, run 3: 0.5 s, 5.0k inputsRandom baseline, run 4: 0.5 s, 11k inputsRandom baseline, run 5: 0.3 s, 3.7k inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.3 s220
Jazzer.js5 of 50.6 s221
fast-check5 of 53.3 s7.7k
Random baseline5 of 50.5 s5.0k

ansi-regex@5.0.0 · CVE-2021-3807 · round 2

Regular expression denial of service when matching escape sequences.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 7.7 s, 7.6k inputsTermo, run 2: 1.0 s, 2.4k inputsTermo, run 3: 4.9 s, 11k inputsTermo, run 4: 0.3 s, 2.5k inputsTermo, run 5: 1.0 s, 1.9k inputsJazzer.jsJazzer.js, run 1: not found, 40.4M inputsJazzer.js, run 2: not found, 39.9M inputsJazzer.js, run 3: not found, 39.3M inputsJazzer.js, run 4: not found, 39.4M inputsJazzer.js, run 5: not found, 40.0M inputsfast-checkfast-check, run 1: not found, 176k inputsfast-check, run 2: not found, 181k inputsfast-check, run 3: not found, 176k inputsfast-check, run 4: not found, 176k inputsfast-check, run 5: not found, 183k inputsRandom baselineRandom baseline, run 1: not found, 12.2M inputsRandom baseline, run 2: not found, 12.2M inputsRandom baseline, run 3: not found, 12.2M inputsRandom baseline, run 4: not found, 12.2M inputsRandom baseline, run 5: not found, 12.3M inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 51.0 s2.2k
Jazzer.js0 of 5> 60 s665k
fast-check0 of 5> 60 s2.9k
Random baseline0 of 5> 60 s204k

color-string@1.5.4 · CVE-2021-29060 · round 2

Regular expression denial of service in hwb() parsing.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 1.0 s, 40k inputsTermo, run 2: 2.4 s, 30k inputsTermo, run 3: 0.4 s, 11k inputsTermo, run 4: 2.2 s, 31k inputsTermo, run 5: 0.8 s, 34k inputsJazzer.jsJazzer.js, run 1: 3.3 s, 1.4M inputsJazzer.js, run 2: 3.0 s, 1.3M inputsJazzer.js, run 3: 15 s, 7.0M inputsJazzer.js, run 4: 27 s, 12.9M inputsJazzer.js, run 5: 21 s, 10.0M inputsfast-checkfast-check, run 1: not found, 182k inputsfast-check, run 2: not found, 177k inputsfast-check, run 3: not found, 177k inputsfast-check, run 4: not found, 178k inputsfast-check, run 5: not found, 184k inputsRandom baselineRandom baseline, run 1: 7.5 s, 1.4M inputsRandom baseline, run 2: 7.5 s, 1.4M inputsRandom baseline, run 3: 7.7 s, 1.4M inputsRandom baseline, run 4: 8.1 s, 1.4M inputsRandom baseline, run 5: 7.4 s, 1.4M inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 51.0 s14k
Jazzer.js5 of 515 s468k
fast-check0 of 5> 60 s3.0k
Random baseline5 of 57.5 s187k

glob-parent@5.1.1 · CVE-2020-28469 · round 3, tuned on

Regular expression denial of service on enclosure patterns.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.2 s, 12 inputsTermo, run 2: 0.2 s, 10 inputsTermo, run 3: 0.5 s, 147 inputsTermo, run 4: 0.1 s, 25 inputsTermo, run 5: 0.3 s, 139 inputsJazzer.jsJazzer.js, run 1: 16 s, 406k inputsJazzer.js, run 2: 36 s, 320k inputsJazzer.js, run 3: 7.6 s, 269k inputsJazzer.js, run 4: not found, 517k inputsJazzer.js, run 5: 6.5 s, 161k inputsfast-checkfast-check, run 1: 0.1 s, 7 inputsfast-check, run 2: 0.1 s, 6 inputsfast-check, run 3: 0.1 s, 3 inputsfast-check, run 4: 0.1 s, 1 inputsfast-check, run 5: 0.1 s, 1 inputsRandom baselineRandom baseline, run 1: 0.7 s, 780 inputsRandom baseline, run 2: 2.9 s, 3.7k inputsRandom baseline, run 3: 2.5 s, 3.3k inputsRandom baseline, run 4: 0.1 s, 39 inputsRandom baseline, run 5: 0.1 s, 16 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.2 s25
Jazzer.js4 of 516 s25k
fast-check5 of 50.1 s3
Random baseline5 of 50.7 s780

trim-newlines@3.0.0 · CVE-2021-33623 · round 3, tuned on

Regular expression denial of service when trimming trailing newlines.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.3 s, 479 inputsTermo, run 2: 0.2 s, 91 inputsTermo, run 3: 0.2 s, 136 inputsTermo, run 4: 0.1 s, 26 inputsTermo, run 5: 0.3 s, 492 inputsJazzer.jsJazzer.js, run 1: not found, 51.1M inputsJazzer.js, run 2: not found, 50.2M inputsJazzer.js, run 3: not found, 51.0M inputsJazzer.js, run 4: not found, 50.9M inputsJazzer.js, run 5: 29 s, 23.0M inputsfast-checkfast-check, run 1: not found, 184k inputsfast-check, run 2: not found, 179k inputsfast-check, run 3: not found, 176k inputsfast-check, run 4: not found, 176k inputsfast-check, run 5: not found, 183k inputsRandom baselineRandom baseline, run 1: not found, 10.6M inputsRandom baseline, run 2: not found, 10.5M inputsRandom baseline, run 3: not found, 10.5M inputsRandom baseline, run 4: not found, 10.5M inputsRandom baseline, run 5: not found, 10.5M inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.2 s136
Jazzer.js1 of 5> 60 s848k
fast-check0 of 5> 60 s3.0k
Random baseline0 of 5> 60 s175k

parse-link-header@1.0.1 · CVE-2021-23490 · round 4

Regular expression denial of service in Link header parsing.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.3 s, 537 inputsTermo, run 2: 0.2 s, 81 inputsTermo, run 3: 0.4 s, 245 inputsTermo, run 4: 0.1 s, 89 inputsTermo, run 5: 0.2 s, 180 inputsJazzer.jsJazzer.js, run 1: 1.4 s, 51k inputsJazzer.js, run 2: 2.5 s, 76k inputsJazzer.js, run 3: 2.4 s, 70k inputsJazzer.js, run 4: 1.3 s, 52k inputsJazzer.js, run 5: 1.2 s, 55k inputsfast-checkfast-check, run 1: not found, 153k inputsfast-check, run 2: not found, 149k inputsfast-check, run 3: not found, 150k inputsfast-check, run 4: not found, 149k inputsfast-check, run 5: not found, 174k inputsRandom baselineRandom baseline, run 1: not found, 3.5M inputsRandom baseline, run 2: not found, 3.4M inputsRandom baseline, run 3: not found, 3.4M inputsRandom baseline, run 4: not found, 3.5M inputsRandom baseline, run 5: not found, 3.8M inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.2 s180
Jazzer.js5 of 51.4 s35k
fast-check0 of 5> 60 s2.5k
Random baseline0 of 5> 60 s58k

Security

Prototype pollution: an input that makes the library write to Object.prototype.

ini@1.3.5 · CVE-2020-7788 · round 1, tuned on

Prototype pollution through section names.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.5 s, 24k inputsTermo, run 2: 0.4 s, 15k inputsTermo, run 3: 0.7 s, 35k inputsTermo, run 4: 0.2 s, 3.3k inputsTermo, run 5: 0.2 s, 6.3k inputsTermo with a grammarTermo with a grammar, run 1: 0.9 s, 366 inputsTermo with a grammar, run 2: 0.2 s, 91 inputsTermo with a grammar, run 3: 0.4 s, 453 inputsTermo with a grammar, run 4: 0.1 s, 53 inputsTermo with a grammar, run 5: 1.6 s, 78 inputsJazzer.jsJazzer.js, run 1: 1.0 s, 53k inputsJazzer.js, run 2: 0.7 s, 38k inputsJazzer.js, run 3: 0.6 s, 25k inputsJazzer.js, run 4: 0.5 s, 15k inputsJazzer.js, run 5: 0.7 s, 31k inputsfast-checkfast-check, run 1: 0.4 s, 794 inputsfast-check, run 2: 0.2 s, 292 inputsfast-check, run 3: 0.5 s, 1.1k inputsfast-check, run 4: 0.7 s, 1.6k inputsfast-check, run 5: 1.9 s, 5.0k inputsRandom baselineRandom baseline, run 1: 0.5 s, 32k inputsRandom baseline, run 2: 0.1 s, 80 inputsRandom baseline, run 3: 0.5 s, 24k inputsRandom baseline, run 4: 0.7 s, 38k inputsRandom baseline, run 5: 0.1 s, 2.7k inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.4 s15k
Termo with a grammar5 of 50.4 s91
Jazzer.js5 of 50.7 s31k
fast-check5 of 50.5 s1.1k
Random baseline5 of 50.5 s24k

minimist@1.2.0 · CVE-2020-7598 · round 1, tuned on

Prototype pollution through option names.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 700 inputsTermo, run 2: 0.1 s, 2.0k inputsTermo, run 3: 0.1 s, 340 inputsTermo, run 4: 0.1 s, 715 inputsTermo, run 5: 0.1 s, 772 inputsTermo with a grammarTermo with a grammar, run 1: 0.2 s, 117 inputsTermo with a grammar, run 2: 0.1 s, 71 inputsTermo with a grammar, run 3: 0.1 s, 6 inputsTermo with a grammar, run 4: 0.5 s, 163 inputsTermo with a grammar, run 5: 0.1 s, 30 inputsJazzer.jsJazzer.js, run 1: 1.0 s, 63k inputsJazzer.js, run 2: 1.1 s, 80k inputsJazzer.js, run 3: 0.7 s, 36k inputsJazzer.js, run 4: 0.6 s, 27k inputsJazzer.js, run 5: 0.6 s, 26k inputsfast-checkfast-check, run 1: 0.1 s, 20 inputsfast-check, run 2: 0.1 s, 33 inputsfast-check, run 3: 0.2 s, 304 inputsfast-check, run 4: 0.1 s, 17 inputsfast-check, run 5: 0.3 s, 444 inputsRandom baselineRandom baseline, run 1: 0.1 s, 1.3k inputsRandom baseline, run 2: 0.1 s, 2.3k inputsRandom baseline, run 3: 0.1 s, 285 inputsRandom baseline, run 4: 0.1 s, 2.6k inputsRandom baseline, run 5: 0.1 s, 213 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s715
Termo with a grammar5 of 50.1 s71
Jazzer.js5 of 50.7 s36k
fast-check5 of 50.1 s33
Random baseline5 of 50.1 s1.3k

set-value@2.0.0 · CVE-2019-10747 · round 2

Prototype pollution through property paths.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 152 inputsTermo, run 2: 0.1 s, 25 inputsTermo, run 3: 0.1 s, 19 inputsTermo, run 4: 0.1 s, 118 inputsTermo, run 5: 0.1 s, 38 inputsJazzer.jsJazzer.js, run 1: 0.3 s, 1.7k inputsJazzer.js, run 2: 0.3 s, 1.5k inputsJazzer.js, run 3: 0.3 s, 1.7k inputsJazzer.js, run 4: 0.4 s, 1.8k inputsJazzer.js, run 5: 0.3 s, 2.3k inputsfast-checkfast-check, run 1: blocked by a hang, 7 inputsfast-check, run 2: blocked by a hang, 6 inputsfast-check, run 3: blocked by a hang, 9 inputsfast-check, run 4: blocked by a hang, 2 inputsfast-check, run 5: blocked by a hang, 3 inputsRandom baselineRandom baseline, run 1: blocked by a hang, 14 inputsRandom baseline, run 2: blocked by a hang, 2 inputsRandom baseline, run 3: blocked by a hang, 7 inputsRandom baseline, run 4: blocked by a hang, 6 inputsRandom baseline, run 5: blocked by a hang, 16 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s38
Jazzer.js5 of 50.3 s1.7k
fast-check0 of 5 (5 blocked by a hang)> 60 s0
Random baseline0 of 5 (5 blocked by a hang)> 60 s0

dot-prop@4.2.0 · CVE-2020-8116 · round 2

Prototype pollution through property paths.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 125 inputsTermo, run 2: 0.1 s, 25 inputsTermo, run 3: 0.1 s, 19 inputsTermo, run 4: 0.1 s, 112 inputsTermo, run 5: 0.1 s, 14 inputsJazzer.jsJazzer.js, run 1: 0.3 s, 1.5k inputsJazzer.js, run 2: 0.3 s, 1.5k inputsJazzer.js, run 3: 0.3 s, 714 inputsJazzer.js, run 4: 0.3 s, 2.2k inputsJazzer.js, run 5: 0.3 s, 2.9k inputsfast-checkfast-check, run 1: 0.1 s, 11 inputsfast-check, run 2: 0.1 s, 9 inputsfast-check, run 3: 0.1 s, 14 inputsfast-check, run 4: 0.1 s, 1 inputsfast-check, run 5: 0.1 s, 1 inputsRandom baselineRandom baseline, run 1: 0.1 s, 37 inputsRandom baseline, run 2: 0.1 s, 11 inputsRandom baseline, run 3: 0.1 s, 83 inputsRandom baseline, run 4: 0.1 s, 63 inputsRandom baseline, run 5: 0.1 s, 27 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s25
Jazzer.js5 of 50.3 s1.5k
fast-check5 of 50.1 s9
Random baseline5 of 50.1 s37

lodash@4.17.15 · CVE-2020-8203 · round 3, tuned on

Prototype pollution in zipObjectDeep.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 51 inputsTermo, run 2: 0.1 s, 35 inputsTermo, run 3: 0.1 s, 21 inputsTermo, run 4: 0.1 s, 64 inputsTermo, run 5: 0.1 s, 14 inputsJazzer.jsJazzer.js, run 1: 0.5 s, 380 inputsJazzer.js, run 2: 0.5 s, 817 inputsJazzer.js, run 3: 0.4 s, 1.1k inputsJazzer.js, run 4: 0.5 s, 2.2k inputsJazzer.js, run 5: 0.5 s, 390 inputsfast-checkfast-check, run 1: 0.1 s, 7 inputsfast-check, run 2: 0.1 s, 6 inputsfast-check, run 3: 0.1 s, 3 inputsfast-check, run 4: 0.1 s, 1 inputsfast-check, run 5: 0.1 s, 1 inputsRandom baselineRandom baseline, run 1: 0.1 s, 14 inputsRandom baseline, run 2: 0.1 s, 11 inputsRandom baseline, run 3: 0.1 s, 13 inputsRandom baseline, run 4: 0.1 s, 15 inputsRandom baseline, run 5: 0.1 s, 4 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s35
Jazzer.js5 of 50.5 s817
fast-check5 of 50.1 s3
Random baseline5 of 50.1 s13

deep-extend@0.5.0 · CVE-2018-3750 · round 3, tuned on

Prototype pollution when merging parsed JSON.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.4 s, 28k inputsTermo, run 2: 0.3 s, 14k inputsTermo, run 3: 0.4 s, 25k inputsTermo, run 4: 0.2 s, 12k inputsTermo, run 5: 0.2 s, 2.7k inputsJazzer.jsJazzer.js, run 1: 2.8 s, 435k inputsJazzer.js, run 2: 10 s, 1.4M inputsJazzer.js, run 3: 3.8 s, 666k inputsJazzer.js, run 4: 21 s, 2.9M inputsJazzer.js, run 5: 3.0 s, 506k inputsfast-checkfast-check, run 1: not found, 186k inputsfast-check, run 2: not found, 181k inputsfast-check, run 3: not found, 177k inputsfast-check, run 4: not found, 176k inputsfast-check, run 5: not found, 182k inputsRandom baselineRandom baseline, run 1: not found, 7.1M inputsRandom baseline, run 2: not found, 7.1M inputsRandom baseline, run 3: not found, 7.0M inputsRandom baseline, run 4: not found, 7.0M inputsRandom baseline, run 5: not found, 7.2M inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.3 s14k
Jazzer.js5 of 53.8 s158k
fast-check0 of 5> 60 s3.0k
Random baseline0 of 5> 60 s118k

y18n@4.0.0 · CVE-2020-7774 · round 3, tuned on

Prototype pollution through the locale name.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 132 inputsTermo, run 2: 0.1 s, 254 inputsTermo, run 3: 0.1 s, 406 inputsTermo, run 4: 0.1 s, 8 inputsTermo, run 5: 0.4 s, 203 inputsJazzer.jsJazzer.js, run 1: 0.8 s, 22k inputsJazzer.js, run 2: 0.3 s, 769 inputsJazzer.js, run 3: 0.5 s, 11k inputsJazzer.js, run 4: 0.7 s, 17k inputsJazzer.js, run 5: 0.6 s, 5.1k inputsfast-checkfast-check, run 1: 0.2 s, 217 inputsfast-check, run 2: 0.1 s, 49 inputsfast-check, run 3: 0.1 s, 158 inputsfast-check, run 4: 0.2 s, 238 inputsfast-check, run 5: 0.1 s, 82 inputsRandom baselineRandom baseline, run 1: 0.1 s, 63 inputsRandom baseline, run 2: 0.1 s, 199 inputsRandom baseline, run 3: 0.1 s, 60 inputsRandom baseline, run 4: 0.1 s, 670 inputsRandom baseline, run 5: 0.1 s, 721 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s203
Jazzer.js5 of 50.6 s11k
fast-check5 of 50.1 s158
Random baseline5 of 50.1 s199

jsonpointer@4.0.1 · CVE-2021-23807 · round 4

Prototype pollution through JSON pointers.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 617 inputsTermo, run 2: 0.1 s, 227 inputsTermo, run 3: 0.1 s, 472 inputsTermo, run 4: 0.1 s, 1.5k inputsTermo, run 5: 0.1 s, 121 inputsJazzer.jsJazzer.js, run 1: 0.3 s, 2.5k inputsJazzer.js, run 2: 0.3 s, 5.2k inputsJazzer.js, run 3: 0.3 s, 765 inputsJazzer.js, run 4: 0.3 s, 4.7k inputsJazzer.js, run 5: 0.3 s, 5.7k inputsfast-checkfast-check, run 1: 0.2 s, 202 inputsfast-check, run 2: 0.1 s, 15 inputsfast-check, run 3: 0.1 s, 33 inputsfast-check, run 4: 0.2 s, 250 inputsfast-check, run 5: 0.1 s, 36 inputsRandom baselineRandom baseline, run 1: 0.1 s, 689 inputsRandom baseline, run 2: 0.1 s, 720 inputsRandom baseline, run 3: 0.1 s, 1.2k inputsRandom baseline, run 4: 0.1 s, 190 inputsRandom baseline, run 5: 0.1 s, 34 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s472
Jazzer.js5 of 50.3 s4.7k
fast-check5 of 50.1 s36
Random baseline5 of 50.1 s689

flat@5.0.0 · CVE-2020-36632 · round 4

Prototype pollution when unflattening keys.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 11 inputsTermo, run 2: 0.1 s, 183 inputsTermo, run 3: 0.1 s, 63 inputsTermo, run 4: 0.1 s, 247 inputsTermo, run 5: blocked by a hang, 34 inputsJazzer.jsJazzer.js, run 1: blocked by a hang, 1.4k inputsJazzer.js, run 2: 0.3 s, 784 inputsJazzer.js, run 3: 0.3 s, 2.1k inputsJazzer.js, run 4: 0.3 s, 203 inputsJazzer.js, run 5: 0.3 s, 2.6k inputsfast-checkfast-check, run 1: 0.1 s, 7 inputsfast-check, run 2: 0.1 s, 9 inputsfast-check, run 3: 0.1 s, 9 inputsfast-check, run 4: 0.1 s, 1 inputsfast-check, run 5: 0.1 s, 1 inputsRandom baselineRandom baseline, run 1: 0.1 s, 37 inputsRandom baseline, run 2: 0.1 s, 5 inputsRandom baseline, run 3: 0.1 s, 17 inputsRandom baseline, run 4: 0.1 s, 38 inputsRandom baseline, run 5: 0.1 s, 34 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo4 of 5 (1 blocked by a hang)0.1 s63
Jazzer.js4 of 5 (1 blocked by a hang)0.3 s784
fast-check5 of 50.1 s7
Random baseline5 of 50.1 s34

dot-object@2.1.2 · CVE-2021-23383 · round 4

Prototype pollution through dotted paths.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 12 inputsTermo, run 2: 0.1 s, 35 inputsTermo, run 3: 0.1 s, 50 inputsTermo, run 4: 0.1 s, 18 inputsTermo, run 5: 0.1 s, 14 inputsJazzer.jsJazzer.js, run 1: 0.3 s, 1.7k inputsJazzer.js, run 2: 0.3 s, 2.3k inputsJazzer.js, run 3: 0.3 s, 2.9k inputsJazzer.js, run 4: 0.3 s, 2.5k inputsJazzer.js, run 5: 0.3 s, 3.2k inputsfast-checkfast-check, run 1: 0.1 s, 7 inputsfast-check, run 2: 0.1 s, 6 inputsfast-check, run 3: 0.1 s, 9 inputsfast-check, run 4: 0.1 s, 1 inputsfast-check, run 5: 0.1 s, 1 inputsRandom baselineRandom baseline, run 1: 0.1 s, 14 inputsRandom baseline, run 2: 0.1 s, 4 inputsRandom baseline, run 3: 0.1 s, 7 inputsRandom baseline, run 4: 0.1 s, 6 inputsRandom baseline, run 5: 0.1 s, 4 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s18
Jazzer.js5 of 50.3 s2.5k
fast-check5 of 50.1 s6
Random baseline5 of 50.1 s6

Robustness

Parser crashes: an exception other than the one the parser documents for bad input.

yaml@1.10.0 · crash · round 3, tuned on

Exceptions other than YAMLError on malformed YAML.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.4 s, 609 inputsTermo, run 2: 0.8 s, 834 inputsTermo, run 3: 0.8 s, 825 inputsTermo, run 4: 1.5 s, 1.1k inputsTermo, run 5: 0.3 s, 354 inputsJazzer.jsJazzer.js, run 1: 0.8 s, 1.8k inputsJazzer.js, run 2: 0.7 s, 1.7k inputsJazzer.js, run 3: 1.1 s, 6.1k inputsJazzer.js, run 4: 1.2 s, 5.4k inputsJazzer.js, run 5: 1.0 s, 4.1k inputsfast-checkfast-check, run 1: 0.1 s, 60 inputsfast-check, run 2: 0.4 s, 507 inputsfast-check, run 3: 0.2 s, 221 inputsfast-check, run 4: 0.3 s, 297 inputsfast-check, run 5: 0.1 s, 45 inputsRandom baselineRandom baseline, run 1: 0.1 s, 49 inputsRandom baseline, run 2: 0.1 s, 124 inputsRandom baseline, run 3: 0.1 s, 108 inputsRandom baseline, run 4: 0.1 s, 646 inputsRandom baseline, run 5: 0.1 s, 310 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.8 s751
Jazzer.js5 of 51.0 s4.1k
fast-check5 of 50.2 s221
Random baseline5 of 50.1 s124

esprima@4.0.1 · crash · round 3, tuned on

Exceptions other than its parse errors on malformed JavaScript.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 433 inputsTermo, run 2: 0.1 s, 316 inputsTermo, run 3: 0.1 s, 532 inputsTermo, run 4: 0.1 s, 282 inputsTermo, run 5: 0.1 s, 522 inputsJazzer.jsJazzer.js, run 1: 1.1 s, 6.6k inputsJazzer.js, run 2: 1.7 s, 17k inputsJazzer.js, run 3: 1.0 s, 6.5k inputsJazzer.js, run 4: 2.6 s, 33k inputsJazzer.js, run 5: 0.6 s, 829 inputsfast-checkfast-check, run 1: 0.5 s, 1.1k inputsfast-check, run 2: 0.4 s, 712 inputsfast-check, run 3: 0.8 s, 1.9k inputsfast-check, run 4: 0.2 s, 153 inputsfast-check, run 5: 0.4 s, 899 inputsRandom baselineRandom baseline, run 1: 0.1 s, 383 inputsRandom baseline, run 2: 0.1 s, 410 inputsRandom baseline, run 3: 0.1 s, 1.9k inputsRandom baseline, run 4: 0.2 s, 3.7k inputsRandom baseline, run 5: 0.1 s, 19 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s433
Jazzer.js5 of 51.1 s6.4k
fast-check5 of 50.4 s899
Random baseline5 of 50.1 s410

json5@1.0.1 · crash · round 3, tuned on

Exceptions other than SyntaxError on malformed JSON5.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 4.4 s, 7.7k inputsTermo, run 2: 0.1 s, 67 inputsTermo, run 3: 0.1 s, 102 inputsTermo, run 4: 0.4 s, 6.4k inputsTermo, run 5: 5.0 s, 611 inputsJazzer.jsJazzer.js, run 1: 0.5 s, 6.4k inputsJazzer.js, run 2: 0.4 s, 1.5k inputsJazzer.js, run 3: 0.5 s, 6.8k inputsJazzer.js, run 4: 0.5 s, 4.7k inputsJazzer.js, run 5: 0.4 s, 1.8k inputsfast-checkfast-check, run 1: 9.2 s, 28k inputsfast-check, run 2: 0.3 s, 368 inputsfast-check, run 3: 2.5 s, 7.2k inputsfast-check, run 4: 0.3 s, 655 inputsfast-check, run 5: 7.1 s, 8.5k inputsRandom baselineRandom baseline, run 1: 0.1 s, 1.0k inputsRandom baseline, run 2: 4.1 s, 1.8k inputsRandom baseline, run 3: 4.3 s, 279 inputsRandom baseline, run 4: 4.6 s, 4 inputsRandom baseline, run 5: 4.4 s, 632 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.4 s123
Jazzer.js5 of 50.5 s4.7k
fast-check5 of 52.5 s1.2k
Random baseline5 of 54.3 s145

@iarna/toml@2.2.5 · crash · round 4

Exceptions other than TomlError on malformed TOML.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.2 s, 2.1k inputsTermo, run 2: 0.1 s, 142 inputsTermo, run 3: 0.1 s, 1.1k inputsTermo, run 4: 0.2 s, 1.0k inputsTermo, run 5: 0.1 s, 1.3k inputsJazzer.jsJazzer.js, run 1: 0.9 s, 20k inputsJazzer.js, run 2: 0.4 s, 481 inputsJazzer.js, run 3: 0.8 s, 17k inputsJazzer.js, run 4: 0.7 s, 12k inputsJazzer.js, run 5: 0.4 s, 8 inputsfast-checkfast-check, run 1: 2.3 s, 6.7k inputsfast-check, run 2: 2.2 s, 6.2k inputsfast-check, run 3: 0.4 s, 826 inputsfast-check, run 4: 6.5 s, 20k inputsfast-check, run 5: 7.2 s, 23k inputsRandom baselineRandom baseline, run 1: 0.9 s, 63k inputsRandom baseline, run 2: 0.4 s, 21k inputsRandom baseline, run 3: 0.2 s, 12k inputsRandom baseline, run 4: 0.2 s, 13k inputsRandom baseline, run 5: 1.1 s, 86k inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s1.1k
Jazzer.js5 of 50.7 s12k
fast-check5 of 52.3 s2.9k
Random baseline5 of 50.4 s21k

fast-xml-parser@3.21.0 · crash · round 4

Exceptions other than its plain Errors for malformed XML.

0.1 s1 s10 s60 snot foundTermoTermo, run 1: 0.1 s, 3 inputsTermo, run 2: 0.1 s, 7 inputsTermo, run 3: 0.1 s, 9 inputsTermo, run 4: 0.1 s, 4 inputsTermo, run 5: 0.1 s, 6 inputsJazzer.jsJazzer.js, run 1: 0.3 s, 4 inputsJazzer.js, run 2: 0.4 s, 11 inputsJazzer.js, run 3: 0.3 s, 7 inputsJazzer.js, run 4: 0.3 s, 10 inputsJazzer.js, run 5: 0.3 s, 32 inputsfast-checkfast-check, run 1: 0.1 s, 3 inputsfast-check, run 2: 0.1 s, 1 inputsfast-check, run 3: 0.1 s, 3 inputsfast-check, run 4: 0.1 s, 1 inputsfast-check, run 5: 0.1 s, 1 inputsRandom baselineRandom baseline, run 1: 0.1 s, 1 inputsRandom baseline, run 2: 0.1 s, 2 inputsRandom baseline, run 3: 0.1 s, 1 inputsRandom baseline, run 4: 0.1 s, 3 inputsRandom baseline, run 5: 0.1 s, 8 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo5 of 50.1 s6
Jazzer.js5 of 50.3 s10
fast-check5 of 50.1 s1
Random baseline5 of 50.1 s2

Method