Fuzzer benchmark

10 Python libraries in 1 suite. 10 parsers (popular libraries pinned to older versions, and standard-library modules), kept because at least one tool found an unexpected exception within five minutes while vetting. Termo runs each harness through its persistent Python driver, which keeps it loaded and reports branch coverage; Hypothesis, HypoFuzz and the random baseline run inputs in-process. Atheris, Google's coverage-guided Python fuzzer, is not included: it does not build on macOS. Every bug is replayed in a fresh Python process, and only reproduced bugs count. Every tool gets the same seed inputs, token dictionary, input size limit and time budget, and one shared oracle decides when the bug is found. darwin arm64, 11 cores, Node v26.13.0; Hypothesis 6.170.15, HypoFuzz 25.11.01.

Ranking

Each tool fuzzes each library 10 times: 10 separate runs of 300 seconds, each with a different random seed. On every library, the tools are ranked from 1 (best) to 4: first by how many of their 10 runs found the bug, then, to break ties, by the median time those runs took to find it (a run that never found it counts as the full 300 seconds). Tools with exactly the same result share the rank. The score is a tool's average rank over all the libraries. 1.00 means best on every library; 4.00 means last on every one.

#ToolScoreRuns that found the bugBest on
1Termo1.6098 of 1004 of 10 libraries
2Random baseline1.8075 of 1006 of 10 libraries
3Hypothesis2.8580 of 1000 of 10 libraries
4HypoFuzz3.7552 of 1000 of 10 libraries

Time to find the bugs

For each kind of bug, the share of runs that had found it by each moment, over all the libraries of that kind. A line that rises early and ends high finds more bugs, sooner. Hover for the numbers.

Robustness (crashes)Exceptions other than the documented one, in Python parsers: share of runs that had found the bug, by time (10 libraries, 10 runs each). Higher and further left is better.
TermoHypothesisHypoFuzzRandom baseline
0%25%50%75%100%0.1 s1 s10 s300 sTermo 98%Hypothesis 80%Random baseline 75%HypoFuzz 52%
Table
ToolRunsFound by 1 sby 10 sby 300 s
Termo10062%83%98%
Hypothesis10047%61%80%
HypoFuzz1000%33%52%
Random baseline10059%62%75%

Robustness

Exceptions other than the documented one for bad input, in popular Python libraries (older pinned versions) and standard-library parsers.

toml 0.10.2 · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: 0.5 s, 97 inputsTermo, run 2: 0.4 s, 2.4k inputsTermo, run 3: 0.3 s, 897 inputsTermo, run 4: 0.4 s, 2.0k inputsTermo, run 5: 0.2 s, 234 inputsTermo, run 6: 0.2 s, 648 inputsTermo, run 7: 0.4 s, 1.8k inputsTermo, run 8: 0.2 s, 402 inputsTermo, run 9: 0.2 s, 314 inputsTermo, run 10: 0.1 s, 28 inputsHypothesisHypothesis, run 1: 0.8 s, 540 inputsHypothesis, run 2: 0.2 s, 105 inputsHypothesis, run 3: 0.4 s, 723 inputsHypothesis, run 4: 0.2 s, 14 inputsHypothesis, run 5: 0.5 s, 1.1k inputsHypothesis, run 6: 0.3 s, 336 inputsHypothesis, run 7: 0.3 s, 141 inputsHypothesis, run 8: 0.3 s, 343 inputsHypothesis, run 9: 0.9 s, 2.3k inputsHypothesis, run 10: 0.3 s, 281 inputsHypoFuzzHypoFuzz, run 1: 129 s, 0 inputsHypoFuzz, run 2: 9.5 s, 0 inputsHypoFuzz, run 3: not found, 0 inputsHypoFuzz, run 4: 146 s, 0 inputsHypoFuzz, run 5: not found, 0 inputsHypoFuzz, run 6: 6.2 s, 0 inputsHypoFuzz, run 7: 44 s, 0 inputsHypoFuzz, run 8: 43 s, 0 inputsHypoFuzz, run 9: 282 s, 0 inputsHypoFuzz, run 10: 65 s, 0 inputsRandom baselineRandom baseline, run 1: 0.2 s, 1.4k inputsRandom baseline, run 2: 0.0 s, 152 inputsRandom baseline, run 3: 0.1 s, 783 inputsRandom baseline, run 4: 0.0 s, 55 inputsRandom baseline, run 5: 0.1 s, 1.1k inputsRandom baseline, run 6: 0.1 s, 769 inputsRandom baseline, run 7: 0.0 s, 454 inputsRandom baseline, run 8: 0.1 s, 341 inputsRandom baseline, run 9: 0.0 s, 73 inputsRandom baseline, run 10: 0.0 s, 228 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo10 of 100.3 s525
Hypothesis10 of 100.3 s340
HypoFuzz8 of 1097 s0
Random baseline10 of 100.0 s398

PyYAML 6.0.1 · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: 35 s, 132k inputsTermo, run 2: 42 s, 184k inputsTermo, run 3: 51 s, 225k inputsTermo, run 4: 49 s, 156k inputsTermo, run 5: 6.3 s, 26k inputsTermo, run 6: 3.8 s, 15k inputsTermo, run 7: 12 s, 44k inputsTermo, run 8: 7.4 s, 26k inputsTermo, run 9: 15 s, 60k inputsTermo, run 10: 11 s, 44k inputsHypothesisHypothesis, run 1: 96 s, 216k inputsHypothesis, run 2: 42 s, 106k inputsHypothesis, run 3: 81 s, 211k inputsHypothesis, run 4: 72 s, 186k inputsHypothesis, run 5: 33 s, 85k inputsHypothesis, run 6: 44 s, 113k inputsHypothesis, run 7: 37 s, 89k inputsHypothesis, run 8: 25 s, 48k inputsHypothesis, run 9: 57 s, 137k inputsHypothesis, run 10: 7.8 s, 19k inputsHypoFuzzHypoFuzz, run 1: not found, 0 inputsHypoFuzz, run 2: not found, 0 inputsHypoFuzz, run 3: not found, 0 inputsHypoFuzz, run 4: not found, 0 inputsHypoFuzz, run 5: not found, 0 inputsHypoFuzz, run 6: not found, 0 inputsHypoFuzz, run 7: not found, 0 inputsHypoFuzz, run 8: not found, 0 inputsHypoFuzz, run 9: not found, 0 inputsHypoFuzz, run 10: not found, 0 inputsRandom baselineRandom baseline, run 1: 0.5 s, 5.8k inputsRandom baseline, run 2: 0.1 s, 931 inputsRandom baseline, run 3: 0.5 s, 6.7k inputsRandom baseline, run 4: 0.8 s, 12k inputsRandom baseline, run 5: 0.5 s, 7.0k inputsRandom baseline, run 6: 0.6 s, 8.6k inputsRandom baseline, run 7: 0.7 s, 8.9k inputsRandom baseline, run 8: 0.2 s, 2.6k inputsRandom baseline, run 9: 0.4 s, 5.4k inputsRandom baseline, run 10: 2.1 s, 25k inputs
ToolRuns that found itMedian time to find itInputs per second
Termo10 of 1013 s4.0k
Hypothesis10 of 1043 s2.5k
HypoFuzz0 of 10> 300 s0
Random baseline10 of 100.5 s6.9k

python-dateutil 2.8.2 · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: 1.3 s, 6.6k inputsTermo, run 2: 1.3 s, 7.1k inputsTermo, run 3: 0.8 s, 4.4k inputsTermo, run 4: 1.3 s, 7.4k inputsTermo, run 5: 0.5 s, 2.4k inputsTermo, run 6: 0.3 s, 733 inputsTermo, run 7: 0.3 s, 450 inputsTermo, run 8: 0.4 s, 1.3k inputsTermo, run 9: 0.6 s, 2.5k inputsTermo, run 10: 1.0 s, 4.8k inputsHypothesisHypothesis, run 1: not found, 892k inputsHypothesis, run 2: not found, 913k inputsHypothesis, run 3: not found, 941k inputsHypothesis, run 4: not found, 945k inputsHypothesis, run 5: not found, 828k inputsHypothesis, run 6: not found, 922k inputsHypothesis, run 7: not found, 832k inputsHypothesis, run 8: not found, 873k inputsHypothesis, run 9: not found, 885k inputsHypothesis, run 10: not found, 804k inputsHypoFuzzHypoFuzz, run 1: not found, 0 inputsHypoFuzz, run 2: not found, 0 inputsHypoFuzz, run 3: not found, 0 inputsHypoFuzz, run 4: not found, 0 inputsHypoFuzz, run 5: not found, 0 inputsHypoFuzz, run 6: not found, 0 inputsHypoFuzz, run 7: not found, 0 inputsHypoFuzz, run 8: not found, 0 inputsHypoFuzz, run 9: not found, 0 inputsHypoFuzz, run 10: not found, 0 inputsRandom baselineRandom baseline, run 1: not found, 18.0M inputsRandom baseline, run 2: not found, 17.8M inputsRandom baseline, run 3: not found, 18.5M inputsRandom baseline, run 4: not found, 18.5M inputsRandom baseline, run 5: not found, 16.3M inputsRandom baseline, run 6: not found, 18.6M inputsRandom baseline, run 7: not found, 16.4M inputsRandom baseline, run 8: not found, 18.1M inputsRandom baseline, run 9: not found, 18.1M inputsRandom baseline, run 10: not found, 16.3M inputs
ToolRuns that found itMedian time to find itInputs per second
Termo10 of 100.7 s3.4k
Hypothesis0 of 10> 300 s3.0k
HypoFuzz0 of 10> 300 s0
Random baseline0 of 10> 300 s60k

sqlparse 0.4.2 · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: 0.3 s, 92 inputsTermo, run 2: 7.1 s, 5.7k inputsTermo, run 3: 2.3 s, 1.5k inputsTermo, run 4: 9.8 s, 5.6k inputsTermo, run 5: 2.1 s, 866 inputsTermo, run 6: 3.1 s, 2.4k inputsTermo, run 7: 7.4 s, 4.8k inputsTermo, run 8: 2.6 s, 1.2k inputsTermo, run 9: 0.8 s, 348 inputsTermo, run 10: 6.8 s, 4.2k inputsHypothesisHypothesis, run 1: 0.9 s, 620 inputsHypothesis, run 2: 7.7 s, 6.9k inputsHypothesis, run 3: 8.5 s, 7.8k inputsHypothesis, run 4: 1.3 s, 1.1k inputsHypothesis, run 5: 15 s, 12k inputsHypothesis, run 6: 19 s, 18k inputsHypothesis, run 7: 6.9 s, 5.4k inputsHypothesis, run 8: 2.2 s, 1.4k inputsHypothesis, run 9: 6.6 s, 5.8k inputsHypothesis, run 10: 18 s, 13k inputsHypoFuzzHypoFuzz, run 1: not found, 0 inputsHypoFuzz, run 2: not found, 0 inputsHypoFuzz, run 3: 270 s, 0 inputsHypoFuzz, run 4: not found, 0 inputsHypoFuzz, run 5: not found, 0 inputsHypoFuzz, run 6: not found, 0 inputsHypoFuzz, run 7: 198 s, 0 inputsHypoFuzz, run 8: not found, 0 inputsHypoFuzz, run 9: 189 s, 0 inputsHypoFuzz, run 10: not found, 0 inputsRandom baselineRandom baseline, run 1: not found, 164k inputsRandom baseline, run 2: not found, 159k inputsRandom baseline, run 3: not found, 168k inputsRandom baseline, run 4: not found, 167k inputsRandom baseline, run 5: 195 s, 94k inputsRandom baseline, run 6: not found, 165k inputsRandom baseline, run 7: not found, 152k inputsRandom baseline, run 8: not found, 166k inputsRandom baseline, run 9: not found, 163k inputsRandom baseline, run 10: 203 s, 101k inputs
ToolRuns that found itMedian time to find itInputs per second
Termo10 of 102.8 s592
Hypothesis10 of 107.3 s795
HypoFuzz3 of 10> 300 s0
Random baseline2 of 10> 300 s544

Markdown 3.3.7 · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: 1.3 s, 3.0k inputsTermo, run 2: 22 s, 25k inputsTermo, run 3: 0.5 s, 1.1k inputsTermo, run 4: 5.0 s, 9.2k inputsTermo, run 5: 2.2 s, 3.8k inputsTermo, run 6: 0.7 s, 1.5k inputsTermo, run 7: 0.7 s, 1.1k inputsTermo, run 8: 2.5 s, 6.3k inputsTermo, run 9: 1.9 s, 5.1k inputsTermo, run 10: 0.7 s, 1.3k inputsHypothesisHypothesis, run 1: 15 s, 31k inputsHypothesis, run 2: 9.1 s, 16k inputsHypothesis, run 3: 75 s, 166k inputsHypothesis, run 4: 32 s, 70k inputsHypothesis, run 5: 33 s, 63k inputsHypothesis, run 6: 4.0 s, 8.5k inputsHypothesis, run 7: 7.7 s, 12k inputsHypothesis, run 8: 81 s, 183k inputsHypothesis, run 9: 39 s, 82k inputsHypothesis, run 10: 32 s, 68k inputsHypoFuzzHypoFuzz, run 1: not found, 0 inputsHypoFuzz, run 2: not found, 0 inputsHypoFuzz, run 3: not found, 0 inputsHypoFuzz, run 4: not found, 0 inputsHypoFuzz, run 5: not found, 0 inputsHypoFuzz, run 6: not found, 0 inputsHypoFuzz, run 7: not found, 0 inputsHypoFuzz, run 8: 262 s, 0 inputsHypoFuzz, run 9: not found, 0 inputsHypoFuzz, run 10: not found, 0 inputsRandom baselineRandom baseline, run 1: not found, 1.4M inputsRandom baseline, run 2: not found, 1.4M inputsRandom baseline, run 3: 34 s, 166k inputsRandom baseline, run 4: 112 s, 545k inputsRandom baseline, run 5: not found, 1.4M inputsRandom baseline, run 6: not found, 1.4M inputsRandom baseline, run 7: 132 s, 558k inputsRandom baseline, run 8: 61 s, 293k inputsRandom baseline, run 9: 229 s, 1.1M inputsRandom baseline, run 10: 82 s, 382k inputs
ToolRuns that found itMedian time to find itInputs per second
Termo10 of 101.6 s1.6k
Hypothesis10 of 1032 s2.1k
HypoFuzz1 of 10> 300 s0
Random baseline6 of 10181 s4.7k

html5lib 1.1 · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: not found, 666k inputsTermo, run 2: 40 s, 87k inputsTermo, run 3: not found, 739k inputsTermo, run 4: 4.4 s, 12k inputsTermo, run 5: 11 s, 33k inputsTermo, run 6: 52 s, 127k inputsTermo, run 7: 50 s, 112k inputsTermo, run 8: 134 s, 283k inputsTermo, run 9: 28 s, 82k inputsTermo, run 10: 92 s, 211k inputsHypothesisHypothesis, run 1: not found, 706k inputsHypothesis, run 2: not found, 693k inputsHypothesis, run 3: not found, 722k inputsHypothesis, run 4: not found, 690k inputsHypothesis, run 5: not found, 711k inputsHypothesis, run 6: not found, 656k inputsHypothesis, run 7: not found, 620k inputsHypothesis, run 8: not found, 681k inputsHypothesis, run 9: not found, 678k inputsHypothesis, run 10: not found, 705k inputsHypoFuzzHypoFuzz, run 1: not found, 0 inputsHypoFuzz, run 2: not found, 0 inputsHypoFuzz, run 3: not found, 0 inputsHypoFuzz, run 4: not found, 0 inputsHypoFuzz, run 5: not found, 0 inputsHypoFuzz, run 6: not found, 0 inputsHypoFuzz, run 7: not found, 0 inputsHypoFuzz, run 8: not found, 0 inputsHypoFuzz, run 9: not found, 0 inputsHypoFuzz, run 10: not found, 0 inputsRandom baselineRandom baseline, run 1: 3.4 s, 20k inputsRandom baseline, run 2: 24 s, 127k inputsRandom baseline, run 3: not found, 1.8M inputsRandom baseline, run 4: 38 s, 230k inputsRandom baseline, run 5: 4.7 s, 28k inputsRandom baseline, run 6: not found, 1.6M inputsRandom baseline, run 7: not found, 1.6M inputsRandom baseline, run 8: 50 s, 302k inputsRandom baseline, run 9: 198 s, 1.1M inputsRandom baseline, run 10: 18 s, 106k inputs
ToolRuns that found itMedian time to find itInputs per second
Termo8 of 1051 s2.4k
Hypothesis0 of 10> 300 s2.3k
HypoFuzz0 of 10> 300 s0
Random baseline7 of 1044 s5.9k

Python 3.12 email · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: 0.4 s, 166 inputsTermo, run 2: 0.2 s, 15 inputsTermo, run 3: 0.5 s, 511 inputsTermo, run 4: 0.2 s, 62 inputsTermo, run 5: 0.3 s, 151 inputsTermo, run 6: 0.3 s, 20 inputsTermo, run 7: 0.2 s, 2 inputsTermo, run 8: 0.2 s, 16 inputsTermo, run 9: 0.2 s, 25 inputsTermo, run 10: 0.3 s, 126 inputsHypothesisHypothesis, run 1: 0.9 s, 1.2k inputsHypothesis, run 2: 2.2 s, 2.7k inputsHypothesis, run 3: 1.2 s, 1.7k inputsHypothesis, run 4: 0.4 s, 317 inputsHypothesis, run 5: 0.6 s, 511 inputsHypothesis, run 6: 0.8 s, 663 inputsHypothesis, run 7: 0.5 s, 425 inputsHypothesis, run 8: 0.4 s, 285 inputsHypothesis, run 9: 0.3 s, 167 inputsHypothesis, run 10: 1.0 s, 1.3k inputsHypoFuzzHypoFuzz, run 1: 5.5 s, 0 inputsHypoFuzz, run 2: 4.6 s, 0 inputsHypoFuzz, run 3: 6.8 s, 0 inputsHypoFuzz, run 4: 3.1 s, 0 inputsHypoFuzz, run 5: 8.1 s, 0 inputsHypoFuzz, run 6: 10 s, 0 inputsHypoFuzz, run 7: 13 s, 0 inputsHypoFuzz, run 8: 2.2 s, 0 inputsHypoFuzz, run 9: 7.9 s, 0 inputsHypoFuzz, run 10: 4.1 s, 0 inputsRandom baselineRandom baseline, run 1: 0.1 s, 2 inputsRandom baseline, run 2: 0.1 s, 65 inputsRandom baseline, run 3: 0.1 s, 23 inputsRandom baseline, run 4: 0.1 s, 27 inputsRandom baseline, run 5: 0.1 s, 40 inputsRandom baseline, run 6: 0.1 s, 38 inputsRandom baseline, run 7: 0.1 s, 31 inputsRandom baseline, run 8: 0.1 s, 30 inputsRandom baseline, run 9: 0.1 s, 31 inputsRandom baseline, run 10: 0.1 s, 6 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo10 of 100.2 s44
Hypothesis10 of 100.7 s587
HypoFuzz10 of 106.2 s0
Random baseline10 of 100.1 s31

Python 3.12 html.parser · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: 0.1 s, 20 inputsTermo, run 2: 0.1 s, 3 inputsTermo, run 3: 0.1 s, 5 inputsTermo, run 4: 0.1 s, 23 inputsTermo, run 5: 0.1 s, 25 inputsTermo, run 6: 0.2 s, 21 inputsTermo, run 7: 0.1 s, 7 inputsTermo, run 8: 0.1 s, 14 inputsTermo, run 9: 0.1 s, 5 inputsTermo, run 10: 0.1 s, 18 inputsHypothesisHypothesis, run 1: 0.2 s, 60 inputsHypothesis, run 2: 0.3 s, 85 inputsHypothesis, run 3: 0.2 s, 7 inputsHypothesis, run 4: 0.2 s, 80 inputsHypothesis, run 5: 0.2 s, 51 inputsHypothesis, run 6: 0.2 s, 53 inputsHypothesis, run 7: 0.3 s, 115 inputsHypothesis, run 8: 0.2 s, 130 inputsHypothesis, run 9: 0.2 s, 61 inputsHypothesis, run 10: 0.3 s, 119 inputsHypoFuzzHypoFuzz, run 1: 1.1 s, 0 inputsHypoFuzz, run 2: 1.7 s, 0 inputsHypoFuzz, run 3: 1.1 s, 0 inputsHypoFuzz, run 4: 1.2 s, 0 inputsHypoFuzz, run 5: 1.1 s, 0 inputsHypoFuzz, run 6: 1.5 s, 0 inputsHypoFuzz, run 7: 1.3 s, 0 inputsHypoFuzz, run 8: 1.2 s, 0 inputsHypoFuzz, run 9: 1.7 s, 0 inputsHypoFuzz, run 10: 1.5 s, 0 inputsRandom baselineRandom baseline, run 1: 0.0 s, 46 inputsRandom baseline, run 2: 0.0 s, 14 inputsRandom baseline, run 3: 0.0 s, 4 inputsRandom baseline, run 4: 0.0 s, 55 inputsRandom baseline, run 5: 0.0 s, 5 inputsRandom baseline, run 6: 0.0 s, 8 inputsRandom baseline, run 7: 0.0 s, 44 inputsRandom baseline, run 8: 0.0 s, 8 inputsRandom baseline, run 9: 0.1 s, 7 inputsRandom baseline, run 10: 0.0 s, 7 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo10 of 100.1 s16
Hypothesis10 of 100.2 s71
HypoFuzz10 of 101.2 s0
Random baseline10 of 100.0 s8

packaging 21.3 · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: 0.3 s, 375 inputsTermo, run 2: 0.4 s, 470 inputsTermo, run 3: 0.3 s, 308 inputsTermo, run 4: 0.2 s, 117 inputsTermo, run 5: 0.2 s, 64 inputsTermo, run 6: 0.4 s, 575 inputsTermo, run 7: 0.3 s, 496 inputsTermo, run 8: 0.3 s, 177 inputsTermo, run 9: 0.4 s, 138 inputsTermo, run 10: 0.6 s, 1.5k inputsHypothesisHypothesis, run 1: 0.2 s, 21 inputsHypothesis, run 2: 1.5 s, 2.7k inputsHypothesis, run 3: 0.8 s, 1.3k inputsHypothesis, run 4: 0.4 s, 337 inputsHypothesis, run 5: 0.4 s, 255 inputsHypothesis, run 6: 0.4 s, 145 inputsHypothesis, run 7: 0.7 s, 954 inputsHypothesis, run 8: 0.5 s, 659 inputsHypothesis, run 9: 0.4 s, 336 inputsHypothesis, run 10: 3.2 s, 7.4k inputsHypoFuzzHypoFuzz, run 1: 55 s, 0 inputsHypoFuzz, run 2: 97 s, 0 inputsHypoFuzz, run 3: 21 s, 0 inputsHypoFuzz, run 4: 3.4 s, 0 inputsHypoFuzz, run 5: 5.8 s, 0 inputsHypoFuzz, run 6: 84 s, 0 inputsHypoFuzz, run 7: 6.4 s, 0 inputsHypoFuzz, run 8: 283 s, 0 inputsHypoFuzz, run 9: 14 s, 0 inputsHypoFuzz, run 10: 64 s, 0 inputsRandom baselineRandom baseline, run 1: 0.1 s, 145 inputsRandom baseline, run 2: 0.1 s, 36 inputsRandom baseline, run 3: 0.1 s, 87 inputsRandom baseline, run 4: 0.1 s, 759 inputsRandom baseline, run 5: 0.1 s, 654 inputsRandom baseline, run 6: 0.1 s, 4 inputsRandom baseline, run 7: 0.1 s, 427 inputsRandom baseline, run 8: 0.1 s, 24 inputsRandom baseline, run 9: 0.1 s, 164 inputsRandom baseline, run 10: 0.1 s, 170 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo10 of 100.3 s342
Hypothesis10 of 100.5 s498
HypoFuzz10 of 1038 s0
Random baseline10 of 100.1 s155

Python 3.12 plistlib · crash · vetted: a tool found it within five minutes

Exceptions other than the documented ones.

0.1 s1 s10 s300 snot foundTermoTermo, run 1: 0.1 s, 27 inputsTermo, run 2: 0.1 s, 19 inputsTermo, run 3: 0.1 s, 54 inputsTermo, run 4: 0.2 s, 250 inputsTermo, run 5: 0.1 s, 27 inputsTermo, run 6: 0.2 s, 154 inputsTermo, run 7: 0.1 s, 29 inputsTermo, run 8: 0.1 s, 18 inputsTermo, run 9: 0.2 s, 950 inputsTermo, run 10: 0.2 s, 642 inputsHypothesisHypothesis, run 1: 0.2 s, 49 inputsHypothesis, run 2: 0.3 s, 284 inputsHypothesis, run 3: 0.2 s, 203 inputsHypothesis, run 4: 0.2 s, 17 inputsHypothesis, run 5: 0.2 s, 77 inputsHypothesis, run 6: 0.3 s, 250 inputsHypothesis, run 7: 0.2 s, 112 inputsHypothesis, run 8: 0.2 s, 139 inputsHypothesis, run 9: 0.2 s, 88 inputsHypothesis, run 10: 0.2 s, 46 inputsHypoFuzzHypoFuzz, run 1: 1.3 s, 0 inputsHypoFuzz, run 2: 1.6 s, 0 inputsHypoFuzz, run 3: 1.4 s, 0 inputsHypoFuzz, run 4: 1.4 s, 0 inputsHypoFuzz, run 5: 1.2 s, 0 inputsHypoFuzz, run 6: 2.3 s, 0 inputsHypoFuzz, run 7: 1.5 s, 0 inputsHypoFuzz, run 8: 1.5 s, 0 inputsHypoFuzz, run 9: 2.2 s, 0 inputsHypoFuzz, run 10: 1.7 s, 0 inputsRandom baselineRandom baseline, run 1: 0.0 s, 26 inputsRandom baseline, run 2: 0.0 s, 35 inputsRandom baseline, run 3: 0.0 s, 2 inputsRandom baseline, run 4: 0.0 s, 15 inputsRandom baseline, run 5: 0.0 s, 70 inputsRandom baseline, run 6: 0.0 s, 45 inputsRandom baseline, run 7: 0.0 s, 126 inputsRandom baseline, run 8: 0.0 s, 77 inputsRandom baseline, run 9: 0.0 s, 308 inputsRandom baseline, run 10: 0.0 s, 26 inputs
ToolRuns that found itMedian time to find itInputs per second
Termo10 of 100.1 s42
Hypothesis10 of 100.2 s100
HypoFuzz10 of 101.5 s0
Random baseline10 of 100.0 s40