10 Python libraries in 1 suite. 10 parsers (popular libraries pinned to older versions, and standard-library modules), kept because at least one tool found an unexpected exception within five minutes while vetting. Termo runs each harness through its persistent Python driver, which keeps it loaded and reports branch coverage; Hypothesis, HypoFuzz and the random baseline run inputs in-process. Atheris, Google's coverage-guided Python fuzzer, is not included: it does not build on macOS. Every bug is replayed in a fresh Python process, and only reproduced bugs count. Every tool gets the same seed inputs, token dictionary, input size limit and time budget, and one shared oracle decides when the bug is found. darwin arm64, 11 cores, Node v26.13.0; Hypothesis 6.170.15, HypoFuzz 25.11.01.
Ranking
Each tool fuzzes each library 10 times: 10 separate runs of 300 seconds, each with a different random seed. On every library, the tools are ranked from 1 (best) to 4: first by how many of their 10 runs found the bug, then, to break ties, by the median time those runs took to find it (a run that never found it counts as the full 300 seconds). Tools with exactly the same result share the rank. The score is a tool's average rank over all the libraries. 1.00 means best on every library; 4.00 means last on every one.
#
Tool
Score
Runs that found the bug
Best on
1
Termo
1.60
98 of 100
4 of 10 libraries
2
Random baseline
1.80
75 of 100
6 of 10 libraries
3
Hypothesis
2.85
80 of 100
0 of 10 libraries
4
HypoFuzz
3.75
52 of 100
0 of 10 libraries
Time to find the bugs
For each kind of bug, the share of runs that had found it by each moment, over all the libraries of that kind. A line that rises early and ends high finds more bugs, sooner. Hover for the numbers.
Robustness (crashes)Exceptions other than the documented one, in Python parsers: share of runs that had found the bug, by time (10 libraries, 10 runs each). Higher and further left is better.
TermoHypothesisHypoFuzzRandom baseline
Table
Tool
Runs
Found by 1 s
by 10 s
by 300 s
Termo
100
62%
83%
98%
Hypothesis
100
47%
61%
80%
HypoFuzz
100
0%
33%
52%
Random baseline
100
59%
62%
75%
Against Hypothesis, Termo was significantly faster to find the bug on 8 libraries and significantly slower on 0 (exact permutation test over the runs, p ≤ 0.05).
Against HypoFuzz, Termo was significantly faster to find the bug on 10 libraries and significantly slower on 0 (exact permutation test over the runs, p ≤ 0.05).
Against Random baseline, Termo was significantly faster to find the bug on 3 libraries and significantly slower on 6 (exact permutation test over the runs, p ≤ 0.05).
Robustness
Exceptions other than the documented one for bad input, in popular Python libraries (older pinned versions) and standard-library parsers.
toml 0.10.2 · crash · vetted: a tool found it within five minutes
Exceptions other than the documented ones.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
10 of 10
0.3 s
525
Hypothesis
10 of 10
0.3 s
340
HypoFuzz
8 of 10
97 s
0
Random baseline
10 of 10
0.0 s
398
PyYAML 6.0.1 · crash · vetted: a tool found it within five minutes
Exceptions other than the documented ones.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
10 of 10
13 s
4.0k
Hypothesis
10 of 10
43 s
2.5k
HypoFuzz
0 of 10
> 300 s
0
Random baseline
10 of 10
0.5 s
6.9k
python-dateutil 2.8.2 · crash · vetted: a tool found it within five minutes
Exceptions other than the documented ones.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
10 of 10
0.7 s
3.4k
Hypothesis
0 of 10
> 300 s
3.0k
HypoFuzz
0 of 10
> 300 s
0
Random baseline
0 of 10
> 300 s
60k
sqlparse 0.4.2 · crash · vetted: a tool found it within five minutes
Exceptions other than the documented ones.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
10 of 10
2.8 s
592
Hypothesis
10 of 10
7.3 s
795
HypoFuzz
3 of 10
> 300 s
0
Random baseline
2 of 10
> 300 s
544
Markdown 3.3.7 · crash · vetted: a tool found it within five minutes
Exceptions other than the documented ones.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
10 of 10
1.6 s
1.6k
Hypothesis
10 of 10
32 s
2.1k
HypoFuzz
1 of 10
> 300 s
0
Random baseline
6 of 10
181 s
4.7k
html5lib 1.1 · crash · vetted: a tool found it within five minutes
Exceptions other than the documented ones.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
8 of 10
51 s
2.4k
Hypothesis
0 of 10
> 300 s
2.3k
HypoFuzz
0 of 10
> 300 s
0
Random baseline
7 of 10
44 s
5.9k
Python 3.12 email · crash · vetted: a tool found it within five minutes
Exceptions other than the documented ones.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
10 of 10
0.2 s
44
Hypothesis
10 of 10
0.7 s
587
HypoFuzz
10 of 10
6.2 s
0
Random baseline
10 of 10
0.1 s
31
Python 3.12 html.parser · crash · vetted: a tool found it within five minutes
Exceptions other than the documented ones.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
10 of 10
0.1 s
16
Hypothesis
10 of 10
0.2 s
71
HypoFuzz
10 of 10
1.2 s
0
Random baseline
10 of 10
0.0 s
8
packaging 21.3 · crash · vetted: a tool found it within five minutes
Exceptions other than the documented ones.
Tool
Runs that found it
Median time to find it
Inputs per second
Termo
10 of 10
0.3 s
342
Hypothesis
10 of 10
0.5 s
498
HypoFuzz
10 of 10
38 s
0
Random baseline
10 of 10
0.1 s
155
Python 3.12 plistlib · crash · vetted: a tool found it within five minutes