email-redos

Sample campaign: an e-mail validator with a planted bug · guided by executionTime · seed 1

Executions36517.79/s
Duration20.52 s
Distinct bugs1first at #211
Median time688 nsp95 8.80 ms
Worst slowdown306,818×worst / median
Outliers32iqr

Bugs

Distinct problems, grouped by kind and by where they happen.

KindBugOccurrencesFirst seenSuspects
timeoutexecution exceeded the timeout37#211

Execution time per iteration

Every collected execution. Emphasized points are statistical outliers; colored ones are bugs.

executionoutliertimeoutlog scale
1 µs10 µs100 µs1 ms10 ms100 ms1 s10 s0100200300400 iteration execution time
Data (365 rows, first 50)
iterationstatusexecution timeinput
1success2 µsana@acme.io
3success917 nsana@acmacme.iome.io
4success875 nsaan..a@a.cmac.me.ia.ome.i.o
5success2 µsanaeacm@.io
6success792 nsana@a@a@acmacmacme.i@a@acmamacmacme.i@a@acma@acm
7success666 nsana@ecmacae.iomm.io
8success375 nsana@a@a@acmacmacme.i@a@acmamacmacme.i@a@acma@acm
9success250 nsana}@acme.ido
10success417 nsaaaaa!naa@acme.@io
11success334 nsana@a@a@acm@aaaaaaai@a@acmamacm.cme.i@a@acma@acm
12success208 nsaaa@a@cmmamiaacmmcmace.i@@cm@acm
13success333 nsY]a,a'm@acmacc]cmB.i@?@ac%wmac4Vc1e.i@a@acma@Tcm
14success333 ns
15success375 nsana@a@ac@ame.io
16success417 nsaa@a@a@acmacmacme.@a@acmamacmacme.i@a@acma@acm
17success292 nsana@a@a@acmacmacme.i@a@acmamacmacme.i@a@acma@acm
18success250 nsa.a@a@a@acmacmacme@i@a@aam.macmac!e.i@a@acaa@acm
19success250 ns
20success334 nsn.e@acmiaao
21success167 ns!.!a
22success375 nsana@a@aaacmacmacme.i@a@acmam@cmacme.i@a@acma@acm
23success875 nsanaoamce.i@
24success292 nsana@dacWmeZt1.Eijoj
25success375 ns@cmeaanoai.
26success333 ns!.!!.!.!.!!.!.!.!a!!.!.!..!!.!.!.!a!!.!.!.!a.!aa
27success417 nsme@naaoc.ia
28success375 nsana@acacmm@acacmme.ie.io
29success1 µsana@acmDe.ioW
30success291 nsanaa@a@acmacmacme.i@a@cmamamcme.i@a@acma@acm
31success709 nsanaace.
32success167 ns%9oF
33success250 ns[QT@s&i)
34success500 nsana@acmaacme.iome!.!aaaaio
35success459 ns@eanaomc.ia
36success250 ns
37success292 nsan@cme.io
2success875 nsa.b+c@shop.example.com
39success708 nsaia@am.acme.mocenio
38success167 ns^|
41success417 nsana@ac.macme.iaome.io
40success1 µsaaaaa@a.aa!@!.aaa@@aaaa@.aaa
42success6 µsaaaaaaaaa.an!aaaaaaa@acm@acm!ea..iomae.aaaaia@o
44success208 ns.aaaaa.@@a.aa!@!.aaa@@a.aaa@.aaa
45success2 µsaa!aaaaaa.an!aaaaaaa@acm@acmaea..iomae.aaaaia@o
43success1 µsa!aaa@a.aaa@aaaa!@@a.aa@.aaa
47success500 nsz^*VA_N.k4|gc@?'y^
46success292 nsaa@.a@aaaa!@!.aaaa@aaaa@.aaa
49success6 µsaaaaaaaaa.an!aaaaaaa@acm@1cm!eU.u$om1ej.aaaaij@L
50success500 nsa@acme.io
48success6 µsaoaaaaaaa.an!aaaaaaa@acm@@cm!ea..iomae.aaaaiaaa

How execution times are distributed

A long right tail means a few inputs are far slower than the rest.

020406080100 ns1 µs10 µs100 µs1 ms10 ms100 ms execution time (log scale) executions
Data (23 rows)
fromtoexecutions
100 ns178 ns23
178 ns316 ns60
316 ns562 ns67
562 ns1 µs47
1 µs2 µs29
2 µs3 µs15
3 µs6 µs7
6 µs10 µs12
10 µs18 µs4
18 µs32 µs2
32 µs56 µs3
56 µs100 µs7
100 µs178 µs8
316 µs562 µs6
562 µs1.00 ms6
1.78 ms3.16 ms5
3.16 ms5.62 ms6
5.62 ms10.00 ms7
17.78 ms31.62 ms2
31.62 ms56.23 ms4
56.23 ms100 ms2
100 ms178 ms5
178 ms316 ms1

Input size and execution time

Input size does not explain execution time here (best fit R² 0.30): the slow inputs are slow because of their structure, not their length.

100 ns1 µs10 µs100 µs1 ms10 ms100 ms1 s110100 input size (log scale) execution time (log scale)

Spearman correlation between size and time: 0.54.

Discovery over time

When the search found new behavior. Flat curves mean the campaign has stopped learning.

Top cases 44
02040600100200300400 iteration
Distinct bugs 1
010100200300400 iteration

Mutators

Share of each mutator's uses that improved the campaign.

addByte 22.0% of 50duplicateChunk 13.8% of 58byteFlip 11.1% of 99dictionary 8.6% of 70delByte 8.1% of 86

Performance outliers

CaseexecutionTime× medianInput
#263211 ms297933.8×aoaaaaaacaaaaacaaaaaaaaca.c..iaa
#239128 ms180776.7×aoaaaaaaaaaaca.aaaaaaaataaa@!@am@aaa...!Z.aae.!!
#234115 ms162631.6×aoaaaaaaaaaaca.aaaaaaaataaa@!@am@aaa...!Z.aae.!!
#189114 ms160406.6×aoaaaaaaaaaaca.aaaaaaaaaaaa@!@am@aaa...!@.aae.!!
#203113 ms159270.6×aoaaaaaaaaaaca.aaaaaaaaaaaa@!@am@aaa...!@.aae.!!
#219111 ms156329.4×aoaaaaaaaaaaca.aaaaaaaaaaaa@!@am@aaa...!@.aae.!!
#26067.39 ms95187.3×aoaaaaaacaaaaaacaaaaacac..imaaae
#25465.76 ms92882.8×aaaaaaaacaaaaacaaaaaacaa.@aaaeae
#19634.81 ms49162.5×aoaaaaaaaaaacaaaaa.aaaaca..im@am@aaa..im@aaaeeia
#16434.33 ms48490.3×aoaaaaaaaaaacaaaaa.aaaaca..im@am@aaa..im@aaaeeia
#19034.32 ms48476.2×aoaaaaOaaaaacaaaaa.aaaaca..Gm@am@a.a..im@faaeeia
#19233.24 ms46946.5×aaaaaaaaaaaaaaaaaa.aaaaca..imaaaaaaaaaam@!aaaaia
#20628.20 ms39835.8×aaaaaaaaaaaaca.aa.aaaaoaaaa@!@am@aaa.a.!@.aae.!!
#19418.17 ms25660.8×aoaaaaaaaaaacaaaa.aaca..i@amaaa..i@aaaia
#2439.25 ms13058.6×xaaaaaaaaaaaaaaa.aa.!aaaaaaaaaa.aa^.~|
#1268.86 ms12508.2×aoaaaaaaaaaaca.aaa.aaaaca..im@am@aaa..im@aaaeeia
#1558.85 ms12504.3×aoaaaaaaaaaaca.aaa.aaaaca..im@am@aaa..im@aaaeeia
#1798.71 ms12300.1×aoaaaaaaaaaaca.aaa.aaaaca..am@am@aaa..im@aiaeeia
#1668.64 ms12206.9×aoaaaaaaaaaaca.aaa.aaaaca..im@am@aaa..im@aaaeeia
#1878.51 ms12023.4×aoaaaaaaaaaaca.aaa.aaaaca..im@am@aaa..im@aaaeeia

executionTime · method iqr · bound 12 µs · median 708 ns over 328 executions.

Regression tests

TestKindMinimized inputReduction
terminates in time (case 211)timeoutaaaaaaaaacaaaaaaaacaaaaaaaaaaca.c31% (ddmin)
runs within its time budget (case 263)performanceaoaaaaaacaaaaacaaaaaaaaca.c..iaa0% (ddmin)
runs within its time budget (case 239)performanceoaaaaaaaaaaaaaaaaaataaam@aaa42% (ddmin)
runs within its time budget (case 234)performanceoaaaaaaaaaaaaaaaaaataaaamaaa42% (ddmin)

Statistics

Metricnminp50p90p95p99maxmeansd
executionTime328125 ns688 ns852 µs8.80 ms113 ms211 ms3.69 ms19.35 ms
inputSize3650274848484824.8818.79