ledgerline-orders

Sample campaign: an order-processing service with planted bugs · guided by coverage, errors · seed 11

Executions10,0003,568/s
Duration2.80 s
Distinct bugs5first at #18
Median time1 µsp95 546 µs
Function calls8median · worst 30.1k (3,760×)
Worst slowdown10,467×worst / median
Outliers0

Bugs

Distinct problems, grouped by kind and by where they happen. Inputs says whether valid inputs reach each bug, or only inputs that break a rule of the grammar on purpose.

KindBugOccurrencesFirst seenInputsSuspects
errorTypeError: Cannot destructure property 'country' of 'order.address' as it is null.267#18 22 of 267 invalid: range element oneOf size nil
range: Product.weight 13 -> 30001 (outside 0..30000)
taxRate (site/samples/orders/tax.js:16:11)
tax (site/samples/orders/tax.js:22:30)
handleOrder (site/samples/orders/service.js:24:20)
errorRangeError: Maximum call stack size exceeded1205#29 126 of 1205 invalid: oneOf nil size range element type
oneOf: Part.sku "giftcard" -> "mateInvalidOption" (not one of lamp, de…
expand (site/samples/orders/catalog.js:18:23)
expand (site/samples/orders/catalog.js:25:57)
expand (site/samples/orders/catalog.js:25:57)
errorTypeError: Reduce of empty array with no initial value456#45 34 of 456 invalid: type oneOf size element range nil
type: Part.quantity 4 -> "mate invalid" (not a Integer)
Array.reduce (<anonymous>)
quote (site/samples/orders/shipping.js:35:16)
handleOrder (site/samples/orders/service.js:25:65)
errorTypeError: Cannot read properties of undefined (reading 'regions')5#139 only invalid inputs: oneOf nil
oneOf: Address.country "GB" -> "mateInvalidOption" (not one of US, DE,…
taxRate (site/samples/orders/tax.js:18:16)
tax (site/samples/orders/tax.js:22:30)
handleOrder (site/samples/orders/service.js:24:20)
errorRangeError: The number NaN cannot be converted to a BigInt because it is not an integer2#5921 only invalid inputs: range
range: Line.quantity 57 -> 0 (outside 1..60)
BigInt (<anonymous>)
site/samples/orders/invoice.js:11:16
Array.map (<anonymous>)

Execution time per iteration

Every collected execution. Emphasized points are statistical outliers; colored ones are bugs.

executionerrorlog scale
1 µs10 µs100 µs1 ms10 ms100 ms05,00010,000 iteration execution time
Data (10000 rows, first 50)
iterationstatusexecution timeinput
1success126 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
2success10.47 ms{"$type":"Order","currency":"GBP","delivery":"ship","address…
3success16 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
4success22 µs{"$type":"Order","currency":"EUR","delivery":"ship","address…
5success72 µs{"$type":"Order","currency":"GBP","delivery":"ship","address…
6success8 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
7success5 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
8success7 µs{"$type":"Order","currency":"EUR","delivery":"ship","address…
9success81 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
10success53 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
11success15 µs{"$type":"Order","currency":"EUR","delivery":"ship","address…
12success4 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
13success7 µs{"$type":"Order","currency":"EUR","delivery":"ship","address…
14success6 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
15success4 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
16success66 µs{"$type":"Order","currency":"GBP","delivery":"ship","address…
17success3 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
18error30 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
19success3 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
20success46 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
21success67 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
22success5 µs{"$type":"Order","currency":"GBP","delivery":"ship","address…
23success4 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
24success5 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
25error32 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
26success3 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
27error22 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
28success3 µs{"$type":"Order","currency":"GBP","delivery":"ship","address…
29error3.68 ms{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
30error3.39 ms{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
31success3 µs{"$type":"Order","currency":"GBP","delivery":"ship","address…
32success7 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
33success3 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
34success3 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
35success2 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
36error3.78 ms{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
37success6 µs{"$type":"Order","currency":"EUR","delivery":"ship","address…
38success2 µs{"$type":"Order","currency":"EUR","delivery":"ship","address…
39success3 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
40success2 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
41success114 µs{"$type":"Order","currency":"EUR","delivery":"ship","address…
42success4 µs{"$type":"Order","currency":"EUR","delivery":"ship","address…
43error33 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
44error15 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
45error13 µs{"$type":"Order","currency":"EUR","delivery":"ship","address…
46success2 µs{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
47success49 µs{"$type":"Order","currency":"GBP","delivery":"ship","address…
48success2 µs{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
49success2 µs{"$type":"Order","currency":"GBP","delivery":"ship","address…
50success2 µs{"$type":"Order","currency":"USD","delivery":"pickup","addre…

How execution times are distributed

A long right tail means a few inputs are far slower than the rest.

01,0002,0003,0004,0001 µs10 µs100 µs1 ms10 ms execution time (log scale) executions
Data (15 rows)
fromtoexecutions
178 ns316 ns4
316 ns562 ns1486
562 ns1 µs3417
1 µs2 µs897
2 µs3 µs563
3 µs6 µs65
6 µs10 µs22
10 µs18 µs350
18 µs32 µs852
32 µs56 µs338
56 µs100 µs64
100 µs178 µs4
178 µs316 µs1
316 µs562 µs1
10.00 ms17.78 ms1

Function calls per iteration

How much code each input made the target run. Unlike time, call counts are exact and do not depend on the machine, so they expose algorithmic problems before they are slow enough to time.

executionerrorlog scale
1101001k10k100k05,00010,000 iteration function calls
Data (10000 rows, first 50)
iterationstatusfunction callsinput
1success8{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
2success31{"$type":"Order","currency":"GBP","delivery":"ship","address…
3success8{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
4success5{"$type":"Order","currency":"EUR","delivery":"ship","address…
5success31{"$type":"Order","currency":"GBP","delivery":"ship","address…
6success5{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
7success8{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
8success9{"$type":"Order","currency":"EUR","delivery":"ship","address…
9success20{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
10success20{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
11success5{"$type":"Order","currency":"EUR","delivery":"ship","address…
12success7{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
13success9{"$type":"Order","currency":"EUR","delivery":"ship","address…
14success5{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
15success5{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
16success31{"$type":"Order","currency":"GBP","delivery":"ship","address…
17success8{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
18error14{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
19success3{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
20success24{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
21success40{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
22success7{"$type":"Order","currency":"GBP","delivery":"ship","address…
23success10{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
24success5{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
25error14{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
26success5{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
27error16{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
28success8{"$type":"Order","currency":"GBP","delivery":"ship","address…
29error3,347{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
30error3,347{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
31success5{"$type":"Order","currency":"GBP","delivery":"ship","address…
32success3{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
33success5{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
34success8{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
35success3{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
36error3,347{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
37success9{"$type":"Order","currency":"EUR","delivery":"ship","address…
38success4{"$type":"Order","currency":"EUR","delivery":"ship","address…
39success7{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
40success3{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
41success56{"$type":"Order","currency":"EUR","delivery":"ship","address…
42success8{"$type":"Order","currency":"EUR","delivery":"ship","address…
43error14{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
44error14{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
45error18{"$type":"Order","currency":"EUR","delivery":"ship","address…
46success5{"$type":"Order","currency":"EUR","delivery":"pickup","addre…
47success31{"$type":"Order","currency":"GBP","delivery":"ship","address…
48success3{"$type":"Order","currency":"GBP","delivery":"pickup","addre…
49success5{"$type":"Order","currency":"GBP","delivery":"ship","address…
50success5{"$type":"Order","currency":"USD","delivery":"pickup","addre…

Where the calls go

The functions the most expensive input calls most, against a typical input. The function that grows most is where the work explodes.

Case #2927, 862 calls, against a median case (#7474, 7 calls). Log scale.

shipping.js:<anonymous> 402 not called in a typical caseshipping.js:<anonymous> 201 not called in a typical caseshipping.js:<anonymous> 200 not called in a typical casecatalog.js:expand 12 not called in a typical casevalidate.js:<anonymous> 10 2× typicalpricing.js:<anonymous> 7 not called in a typical caseservice.js:<anonymous> 3 not called in a typical caseservice.js:<anonymous> 3 not called in a typical case
Data (8 rows)
functioncalls (worst case)calls (typical case)
orders/shipping.js:<anonymous>@10774020
orders/shipping.js:<anonymous>@10452010
orders/shipping.js:<anonymous>@11552000
orders/catalog.js:expand@793120
orders/validate.js:<anonymous>@1571105
orders/pricing.js:<anonymous>@39670
orders/service.js:<anonymous>@94430
orders/service.js:<anonymous>@139530

Input size and execution time

Input size does not explain execution time here (best fit R² 0.13): the slow inputs are slow because of their structure, not their length.

100 ns1 µs10 µs100 µs1 ms10 ms100 ms1001,00010,000 input size (log scale) execution time (log scale)

Spearman correlation between size and time: 0.39.

Discovery over time

When the search found new behavior. Flat curves mean the campaign has stopped learning.

Top cases 11
05101505,00010,000 iteration
Distinct bugs 5
024605,00010,000 iteration
Covered blocks 79
02040608005,00010,000 iteration

Mutators

Share of each mutator's uses that improved the campaign.

mateDerivation 0.1% of 9999

Performance outliers

No significant outliers.

Regression tests

TestKindMinimized inputReduction
does not throw TypeError: Cannot destructure property 'country' of 'order.address' as (case 18)error{"$type":"Order","currency":"USD","delivery":"pickup","address":null,"products":[{"$type":"Product","sku":"officeset","p…73% (mate)
does not throw RangeError: Maximum call stack size exceeded (case 29)error{"$type":"Order","currency":"USD","delivery":"pickup","address":null,"products":[{"$type":"Product","sku":"officeset","p…75% (mate)
does not throw TypeError: Reduce of empty array with no initial value (case 45)error{"$type":"Order","currency":"USD","delivery":"ship","address":{"$type":"Address","country":"US","region":"CA"},"products…78% (mate)
does not throw TypeError: Cannot read properties of undefined (reading 'regions') (case 139)error{"$type":"Order","currency":"GBP","delivery":"pickup","address":{"$type":"Address","country":"mateInvalidOption","region…0% (mate)
does not throw RangeError: The number NaN cannot be converted to a BigInt because it is (case 5921)error{"$type":"Order","currency":"GBP","delivery":"pickup","address":{"$type":"Address","country":"GB","region":"NY"},"produc…0% (mate)

Statistics

Metricnminp50p90p95p99maxmeansd
executionTime10000291 ns1 µs515 µs546 µs667 µs10.47 ms81 µs266 µs
functionCalls10000286,0216,02512,03830,077923.432,690
blocks10000134512,06712,07930,11872,2092,0316,091
coverage1000016.0%29.6%63.0%66.7%69.1%70.4%35.8%12.9%
inputSize10000144456589611645779444.40109.75