ledgerline-orders
Sample campaign: an order-processing service with planted bugs · guided by coverage, errors · seed 11
Bugs
Distinct problems, grouped by kind and by where they happen. Inputs says whether valid inputs reach each bug, or only inputs that break a rule of the grammar on purpose.
| Kind | Bug | Occurrences | First seen | Inputs | Suspects |
|---|---|---|---|---|---|
| error | TypeError: Cannot destructure property 'country' of 'order.address' as it is null. | 267 | #18 | 22 of 267 invalid: range element oneOf size nilrange: Product.weight 13 -> 30001 (outside 0..30000) | taxRate (site/samples/orders/tax.js:16:11)tax (site/samples/orders/tax.js:22:30)handleOrder (site/samples/orders/service.js:24:20) |
| error | RangeError: Maximum call stack size exceeded | 1205 | #29 | 126 of 1205 invalid: oneOf nil size range element typeoneOf: Part.sku "giftcard" -> "mateInvalidOption" (not one of lamp, de… | expand (site/samples/orders/catalog.js:18:23)expand (site/samples/orders/catalog.js:25:57)expand (site/samples/orders/catalog.js:25:57) |
| error | TypeError: Reduce of empty array with no initial value | 456 | #45 | 34 of 456 invalid: type oneOf size element range niltype: Part.quantity 4 -> "mate invalid" (not a Integer) | Array.reduce (<anonymous>)quote (site/samples/orders/shipping.js:35:16)handleOrder (site/samples/orders/service.js:25:65) |
| error | TypeError: Cannot read properties of undefined (reading 'regions') | 5 | #139 | only invalid inputs: oneOf niloneOf: Address.country "GB" -> "mateInvalidOption" (not one of US, DE,… | taxRate (site/samples/orders/tax.js:18:16)tax (site/samples/orders/tax.js:22:30)handleOrder (site/samples/orders/service.js:24:20) |
| error | RangeError: The number NaN cannot be converted to a BigInt because it is not an integer | 2 | #5921 | only invalid inputs: rangerange: Line.quantity 57 -> 0 (outside 1..60) | BigInt (<anonymous>)site/samples/orders/invoice.js:11:16Array.map (<anonymous>) |
Execution time per iteration
Every collected execution. Emphasized points are statistical outliers; colored ones are bugs.
Data (10000 rows, first 50)
| iteration | status | execution time | input |
|---|---|---|---|
| 1 | success | 126 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 2 | success | 10.47 ms | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 3 | success | 16 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 4 | success | 22 µs | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 5 | success | 72 µs | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 6 | success | 8 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 7 | success | 5 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 8 | success | 7 µs | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 9 | success | 81 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 10 | success | 53 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 11 | success | 15 µs | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 12 | success | 4 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 13 | success | 7 µs | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 14 | success | 6 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 15 | success | 4 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 16 | success | 66 µs | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 17 | success | 3 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 18 | error | 30 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 19 | success | 3 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 20 | success | 46 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 21 | success | 67 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 22 | success | 5 µs | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 23 | success | 4 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 24 | success | 5 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 25 | error | 32 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 26 | success | 3 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 27 | error | 22 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 28 | success | 3 µs | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 29 | error | 3.68 ms | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 30 | error | 3.39 ms | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 31 | success | 3 µs | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 32 | success | 7 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 33 | success | 3 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 34 | success | 3 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 35 | success | 2 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 36 | error | 3.78 ms | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 37 | success | 6 µs | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 38 | success | 2 µs | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 39 | success | 3 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 40 | success | 2 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 41 | success | 114 µs | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 42 | success | 4 µs | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 43 | error | 33 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 44 | error | 15 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 45 | error | 13 µs | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 46 | success | 2 µs | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 47 | success | 49 µs | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 48 | success | 2 µs | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 49 | success | 2 µs | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 50 | success | 2 µs | {"$type":"Order","currency":"USD","delivery":"pickup","addre… |
How execution times are distributed
A long right tail means a few inputs are far slower than the rest.
Data (15 rows)
| from | to | executions |
|---|---|---|
| 178 ns | 316 ns | 4 |
| 316 ns | 562 ns | 1486 |
| 562 ns | 1 µs | 3417 |
| 1 µs | 2 µs | 897 |
| 2 µs | 3 µs | 563 |
| 3 µs | 6 µs | 65 |
| 6 µs | 10 µs | 22 |
| 10 µs | 18 µs | 350 |
| 18 µs | 32 µs | 852 |
| 32 µs | 56 µs | 338 |
| 56 µs | 100 µs | 64 |
| 100 µs | 178 µs | 4 |
| 178 µs | 316 µs | 1 |
| 316 µs | 562 µs | 1 |
| 10.00 ms | 17.78 ms | 1 |
Function calls per iteration
How much code each input made the target run. Unlike time, call counts are exact and do not depend on the machine, so they expose algorithmic problems before they are slow enough to time.
Data (10000 rows, first 50)
| iteration | status | function calls | input |
|---|---|---|---|
| 1 | success | 8 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 2 | success | 31 | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 3 | success | 8 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 4 | success | 5 | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 5 | success | 31 | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 6 | success | 5 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 7 | success | 8 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 8 | success | 9 | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 9 | success | 20 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 10 | success | 20 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 11 | success | 5 | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 12 | success | 7 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 13 | success | 9 | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 14 | success | 5 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 15 | success | 5 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 16 | success | 31 | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 17 | success | 8 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 18 | error | 14 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 19 | success | 3 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 20 | success | 24 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 21 | success | 40 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 22 | success | 7 | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 23 | success | 10 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 24 | success | 5 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 25 | error | 14 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 26 | success | 5 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 27 | error | 16 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 28 | success | 8 | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 29 | error | 3,347 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 30 | error | 3,347 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 31 | success | 5 | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 32 | success | 3 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 33 | success | 5 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 34 | success | 8 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 35 | success | 3 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 36 | error | 3,347 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 37 | success | 9 | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 38 | success | 4 | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 39 | success | 7 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 40 | success | 3 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 41 | success | 56 | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 42 | success | 8 | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 43 | error | 14 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 44 | error | 14 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 45 | error | 18 | {"$type":"Order","currency":"EUR","delivery":"ship","address… |
| 46 | success | 5 | {"$type":"Order","currency":"EUR","delivery":"pickup","addre… |
| 47 | success | 31 | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 48 | success | 3 | {"$type":"Order","currency":"GBP","delivery":"pickup","addre… |
| 49 | success | 5 | {"$type":"Order","currency":"GBP","delivery":"ship","address… |
| 50 | success | 5 | {"$type":"Order","currency":"USD","delivery":"pickup","addre… |
Where the calls go
The functions the most expensive input calls most, against a typical input. The function that grows most is where the work explodes.
Case #2927, 862 calls, against a median case (#7474, 7 calls). Log scale.
Data (8 rows)
| function | calls (worst case) | calls (typical case) |
|---|---|---|
| orders/shipping.js:<anonymous>@1077 | 402 | 0 |
| orders/shipping.js:<anonymous>@1045 | 201 | 0 |
| orders/shipping.js:<anonymous>@1155 | 200 | 0 |
| orders/catalog.js:expand@793 | 12 | 0 |
| orders/validate.js:<anonymous>@1571 | 10 | 5 |
| orders/pricing.js:<anonymous>@396 | 7 | 0 |
| orders/service.js:<anonymous>@944 | 3 | 0 |
| orders/service.js:<anonymous>@1395 | 3 | 0 |
Input size and execution time
Input size does not explain execution time here (best fit R² 0.13): the slow inputs are slow because of their structure, not their length.
Spearman correlation between size and time: 0.39.
Discovery over time
When the search found new behavior. Flat curves mean the campaign has stopped learning.
Mutators
Share of each mutator's uses that improved the campaign.
Performance outliers
No significant outliers.
Regression tests
| Test | Kind | Minimized input | Reduction |
|---|---|---|---|
| does not throw TypeError: Cannot destructure property 'country' of 'order.address' as (case 18) | error | {"$type":"Order","currency":"USD","delivery":"pickup","address":null,"products":[{"$type":"Product","sku":"officeset","p… | 73% (mate) |
| does not throw RangeError: Maximum call stack size exceeded (case 29) | error | {"$type":"Order","currency":"USD","delivery":"pickup","address":null,"products":[{"$type":"Product","sku":"officeset","p… | 75% (mate) |
| does not throw TypeError: Reduce of empty array with no initial value (case 45) | error | {"$type":"Order","currency":"USD","delivery":"ship","address":{"$type":"Address","country":"US","region":"CA"},"products… | 78% (mate) |
| does not throw TypeError: Cannot read properties of undefined (reading 'regions') (case 139) | error | {"$type":"Order","currency":"GBP","delivery":"pickup","address":{"$type":"Address","country":"mateInvalidOption","region… | 0% (mate) |
| does not throw RangeError: The number NaN cannot be converted to a BigInt because it is (case 5921) | error | {"$type":"Order","currency":"GBP","delivery":"pickup","address":{"$type":"Address","country":"GB","region":"NY"},"produc… | 0% (mate) |
Statistics
| Metric | n | min | p50 | p90 | p95 | p99 | max | mean | sd |
|---|---|---|---|---|---|---|---|---|---|
| executionTime | 10000 | 291 ns | 1 µs | 515 µs | 546 µs | 667 µs | 10.47 ms | 81 µs | 266 µs |
| functionCalls | 10000 | 2 | 8 | 6,021 | 6,025 | 12,038 | 30,077 | 923.43 | 2,690 |
| blocks | 10000 | 13 | 45 | 12,067 | 12,079 | 30,118 | 72,209 | 2,031 | 6,091 |
| coverage | 10000 | 16.0% | 29.6% | 63.0% | 66.7% | 69.1% | 70.4% | 35.8% | 12.9% |
| inputSize | 10000 | 144 | 456 | 589 | 611 | 645 | 779 | 444.40 | 109.75 |