email-redos
Sample campaign: an e-mail validator with a planted bug · guided by executionTime · seed 1
Executions36517.79/s
Duration20.52 s
Distinct bugs1first at #211
Median time688 nsp95 8.80 ms
Worst slowdown306,818×worst / median
Outliers32iqr
Bugs
Distinct problems, grouped by kind and by where they happen.
| Kind | Bug | Occurrences | First seen | Suspects |
|---|---|---|---|---|
| timeout | execution exceeded the timeout | 37 | #211 |
Execution time per iteration
Every collected execution. Emphasized points are statistical outliers; colored ones are bugs.
executionoutliertimeoutlog scale
Data (365 rows, first 50)
| iteration | status | execution time | input |
|---|---|---|---|
| 1 | success | 2 µs | ana@acme.io |
| 3 | success | 917 ns | ana@acmacme.iome.io |
| 4 | success | 875 ns | aan..a@a.cmac.me.ia.ome.i.o |
| 5 | success | 2 µs | anaeacm@.io |
| 6 | success | 792 ns | ana@a@a@acmacmacme.i@a@acmamacmacme.i@a@acma@acm |
| 7 | success | 666 ns | ana@ecmacae.iomm.io |
| 8 | success | 375 ns | ana@a@a@acmacmacme.i@a@acmamacmacme.i@a@acma@acm |
| 9 | success | 250 ns | ana}@acme.ido |
| 10 | success | 417 ns | aaaaa!naa@acme.@io |
| 11 | success | 334 ns | ana@a@a@acm@aaaaaaai@a@acmamacm.cme.i@a@acma@acm |
| 12 | success | 208 ns | aaa@a@cmmamiaacmmcmace.i@@cm@acm |
| 13 | success | 333 ns | Y]a,a'm@acmacc]cmB.i@?@ac%wmac4Vc1e.i@a@acma@Tcm |
| 14 | success | 333 ns | |
| 15 | success | 375 ns | ana@a@ac@ame.io |
| 16 | success | 417 ns | aa@a@a@acmacmacme.@a@acmamacmacme.i@a@acma@acm |
| 17 | success | 292 ns | ana@a@a@acmacmacme.i@a@acmamacmacme.i@a@acma@acm |
| 18 | success | 250 ns | a.a@a@a@acmacmacme@i@a@aam.macmac!e.i@a@acaa@acm |
| 19 | success | 250 ns | |
| 20 | success | 334 ns | n.e@acmiaao |
| 21 | success | 167 ns | !.!a |
| 22 | success | 375 ns | ana@a@aaacmacmacme.i@a@acmam@cmacme.i@a@acma@acm |
| 23 | success | 875 ns | anaoamce.i@ |
| 24 | success | 292 ns | ana@dacWmeZt1.Eijoj |
| 25 | success | 375 ns | @cmeaanoai. |
| 26 | success | 333 ns | !.!!.!.!.!!.!.!.!a!!.!.!..!!.!.!.!a!!.!.!.!a.!aa |
| 27 | success | 417 ns | me@naaoc.ia |
| 28 | success | 375 ns | ana@acacmm@acacmme.ie.io |
| 29 | success | 1 µs | ana@acmDe.ioW |
| 30 | success | 291 ns | anaa@a@acmacmacme.i@a@cmamamcme.i@a@acma@acm |
| 31 | success | 709 ns | anaace. |
| 32 | success | 167 ns | %9oF |
| 33 | success | 250 ns | [QT@s&i) |
| 34 | success | 500 ns | ana@acmaacme.iome!.!aaaaio |
| 35 | success | 459 ns | @eanaomc.ia |
| 36 | success | 250 ns | |
| 37 | success | 292 ns | an@cme.io |
| 2 | success | 875 ns | a.b+c@shop.example.com |
| 39 | success | 708 ns | aia@am.acme.mocenio |
| 38 | success | 167 ns | ^| |
| 41 | success | 417 ns | ana@ac.macme.iaome.io |
| 40 | success | 1 µs | aaaaa@a.aa!@!.aaa@@aaaa@.aaa |
| 42 | success | 6 µs | aaaaaaaaa.an!aaaaaaa@acm@acm!ea..iomae.aaaaia@o |
| 44 | success | 208 ns | .aaaaa.@@a.aa!@!.aaa@@a.aaa@.aaa |
| 45 | success | 2 µs | aa!aaaaaa.an!aaaaaaa@acm@acmaea..iomae.aaaaia@o |
| 43 | success | 1 µs | a!aaa@a.aaa@aaaa!@@a.aa@.aaa |
| 47 | success | 500 ns | z^*VA_N.k4|gc@?'y^ |
| 46 | success | 292 ns | aa@.a@aaaa!@!.aaaa@aaaa@.aaa |
| 49 | success | 6 µs | aaaaaaaaa.an!aaaaaaa@acm@1cm!eU.u$om1ej.aaaaij@L |
| 50 | success | 500 ns | a@acme.io |
| 48 | success | 6 µs | aoaaaaaaa.an!aaaaaaa@acm@@cm!ea..iomae.aaaaiaaa |
How execution times are distributed
A long right tail means a few inputs are far slower than the rest.
Data (23 rows)
| from | to | executions |
|---|---|---|
| 100 ns | 178 ns | 23 |
| 178 ns | 316 ns | 60 |
| 316 ns | 562 ns | 67 |
| 562 ns | 1 µs | 47 |
| 1 µs | 2 µs | 29 |
| 2 µs | 3 µs | 15 |
| 3 µs | 6 µs | 7 |
| 6 µs | 10 µs | 12 |
| 10 µs | 18 µs | 4 |
| 18 µs | 32 µs | 2 |
| 32 µs | 56 µs | 3 |
| 56 µs | 100 µs | 7 |
| 100 µs | 178 µs | 8 |
| 316 µs | 562 µs | 6 |
| 562 µs | 1.00 ms | 6 |
| 1.78 ms | 3.16 ms | 5 |
| 3.16 ms | 5.62 ms | 6 |
| 5.62 ms | 10.00 ms | 7 |
| 17.78 ms | 31.62 ms | 2 |
| 31.62 ms | 56.23 ms | 4 |
| 56.23 ms | 100 ms | 2 |
| 100 ms | 178 ms | 5 |
| 178 ms | 316 ms | 1 |
Input size and execution time
Input size does not explain execution time here (best fit R² 0.30): the slow inputs are slow because of their structure, not their length.
Spearman correlation between size and time: 0.54.
Discovery over time
When the search found new behavior. Flat curves mean the campaign has stopped learning.
Mutators
Share of each mutator's uses that improved the campaign.
Performance outliers
| Case | executionTime | × median | Input |
|---|---|---|---|
| #263 | 211 ms | 297933.8× | aoaaaaaacaaaaacaaaaaaaaca.c..iaa |
| #239 | 128 ms | 180776.7× | aoaaaaaaaaaaca.aaaaaaaataaa@!@am@aaa...!Z.aae.!! |
| #234 | 115 ms | 162631.6× | aoaaaaaaaaaaca.aaaaaaaataaa@!@am@aaa...!Z.aae.!! |
| #189 | 114 ms | 160406.6× | aoaaaaaaaaaaca.aaaaaaaaaaaa@!@am@aaa...!@.aae.!! |
| #203 | 113 ms | 159270.6× | aoaaaaaaaaaaca.aaaaaaaaaaaa@!@am@aaa...!@.aae.!! |
| #219 | 111 ms | 156329.4× | aoaaaaaaaaaaca.aaaaaaaaaaaa@!@am@aaa...!@.aae.!! |
| #260 | 67.39 ms | 95187.3× | aoaaaaaacaaaaaacaaaaacac..imaaae |
| #254 | 65.76 ms | 92882.8× | aaaaaaaacaaaaacaaaaaacaa.@aaaeae |
| #196 | 34.81 ms | 49162.5× | aoaaaaaaaaaacaaaaa.aaaaca..im@am@aaa..im@aaaeeia |
| #164 | 34.33 ms | 48490.3× | aoaaaaaaaaaacaaaaa.aaaaca..im@am@aaa..im@aaaeeia |
| #190 | 34.32 ms | 48476.2× | aoaaaaOaaaaacaaaaa.aaaaca..Gm@am@a.a..im@faaeeia |
| #192 | 33.24 ms | 46946.5× | aaaaaaaaaaaaaaaaaa.aaaaca..imaaaaaaaaaam@!aaaaia |
| #206 | 28.20 ms | 39835.8× | aaaaaaaaaaaaca.aa.aaaaoaaaa@!@am@aaa.a.!@.aae.!! |
| #194 | 18.17 ms | 25660.8× | aoaaaaaaaaaacaaaa.aaca..i@amaaa..i@aaaia |
| #243 | 9.25 ms | 13058.6× | xaaaaaaaaaaaaaaa.aa.!aaaaaaaaaa.aa^.~| |
| #126 | 8.86 ms | 12508.2× | aoaaaaaaaaaaca.aaa.aaaaca..im@am@aaa..im@aaaeeia |
| #155 | 8.85 ms | 12504.3× | aoaaaaaaaaaaca.aaa.aaaaca..im@am@aaa..im@aaaeeia |
| #179 | 8.71 ms | 12300.1× | aoaaaaaaaaaaca.aaa.aaaaca..am@am@aaa..im@aiaeeia |
| #166 | 8.64 ms | 12206.9× | aoaaaaaaaaaaca.aaa.aaaaca..im@am@aaa..im@aaaeeia |
| #187 | 8.51 ms | 12023.4× | aoaaaaaaaaaaca.aaa.aaaaca..im@am@aaa..im@aaaeeia |
executionTime · method iqr · bound 12 µs · median 708 ns over 328 executions.
Regression tests
| Test | Kind | Minimized input | Reduction |
|---|---|---|---|
| terminates in time (case 211) | timeout | aaaaaaaaacaaaaaaaacaaaaaaaaaaca.c | 31% (ddmin) |
| runs within its time budget (case 263) | performance | aoaaaaaacaaaaacaaaaaaaaca.c..iaa | 0% (ddmin) |
| runs within its time budget (case 239) | performance | oaaaaaaaaaaaaaaaaaataaam@aaa | 42% (ddmin) |
| runs within its time budget (case 234) | performance | oaaaaaaaaaaaaaaaaaataaaamaaa | 42% (ddmin) |
Statistics
| Metric | n | min | p50 | p90 | p95 | p99 | max | mean | sd |
|---|---|---|---|---|---|---|---|---|---|
| executionTime | 328 | 125 ns | 688 ns | 852 µs | 8.80 ms | 113 ms | 211 ms | 3.69 ms | 19.35 ms |
| inputSize | 365 | 0 | 27 | 48 | 48 | 48 | 48 | 24.88 | 18.79 |