quill-templates

Sample campaign: an HTML template engine with a planted bug · guided by methodCalls · seed 7

Executions4,000930.00/s
Duration4.30 s
Distinct bugs0
Median time30 µsp95 401 µs
Function calls1,725median · worst 26.6M (15,399×)
Worst slowdown14,330×worst / median
Outliers524iqr

Bugs

Distinct problems, grouped by kind and by where they happen.

No errors, property failures or timeouts.

Function calls per iteration

How much code each input made the target run. Unlike time, call counts are exact and do not depend on the machine, so they expose algorithmic problems before they are slow enough to time.

executionoutlierlog scale
1001k10k100k1M10M100M01,0002,0003,0004,000 iteration function calls
Data (4000 rows, first 50)
iterationstatusfunction callsinput
1success311<ul>{% for item in items %}{{ card(item | title) }}{% endfor…
2success319<p>Hello, {{ user.name }} {{ badge(user.tier) }}</p>{{ card(…
3success139</b>{{ item }}
4success278{% for item in items %}{% for item in empty %}{{ price(title…
5success250{% if user.name %}{% if user.name %}<li>{% for item in empty…
6success273{% for item in empty %}<b>{{ card(price(card(user.tier) + ti…
7success138{{ "ok" | default }}
8success372{% for item in items %}{% for item in items %}{% if "ok" %}{…
9success260{% for item in items %}{% for item in empty %}{% for item in…
10success138{{ title }}
11success424{% for item in items %}{% for item in items %}{% if user.nam…
12success462{% if card("sale" | upper) %}{{ card(user.tier | lower) + us…
13success376{% if badge(title | default) %}{{ "new" + "ok" + item + user…
14success335{% if "new" %}{{ badge("sale" | title) + user.tier + "new" }…
15success288{% for item in items %}{% for item in empty %}{{ badge(user.…
16success280{% for item in items %}{% for item in empty %}{{ price(title…
17success701{% if user.tier %}{{ price(card(user.name + title) | trim) }…
18success810{% if user.tier %}{{ price(card(user.name + title + card(ite…
19success771{% if user.tier %}{{ card(card("sale" + user.name + user.nam…
20success220{% for item in items %}{% for item in items %}</li>{% endfor…
21success731{% if user.tier %}{{ price(price("ok" + user.name + user.nam…
22success553{% if user.tier %}{{ price(card(user.name + title + user.nam…
23success185{% if item %}{{ user.tier }}Hello, {% else %}{{ "ok" + user.…
24success747{% if user.tier %}{{ item | title }}{% for item in items %}{…
25success500{% if card(card("new" | trim) + user.tier + user.tier + item…
26success124</li>
27success1,064{% if card("sale" | upper) %}{{ price(price(user.name + titl…
28success351{% if card("sale" | trim) %}{{ "sale" }}{% for item in items…
29success1,254{% if card(price(price(user.name + title + badge(user.tier +…
30success1,778{% if card(price(price(user.name + title + price(price(user.…
31success1,247{% if card(price(price(user.name + title + price(user.tier +…
32success1,971{% if card(price(price(user.name + title + price(price(title…
33success701{% if user.tier %}{{ price(card(user.name + title) | trim) }…
34success947{% if user.tier %}{{ price(card(user.name + item + card(user…
35success1,239{% if card(price(price(user.name + title + price(user.tier +…
36success1,062{% if card(card(price(user.name + title + price(item + item)…
37success701{% if user.tier %}{{ price(card(user.name + title) | trim) }…
38success272{% if price(price(title) | upper) %}</p>{% endif %}
39success460{% if "sale" %}Hello, {% for item in items %}{{ title + user…
40success189{{ card(title) | lower }}
41success866{% if title %}<p>{% for item in items %}{% for item in items…
42success330{% if user.name %}</b>{{ item | title }}<li>{% for item in i…
43success409{% if card(price("ok")) %}{{ price(user.name) | upper }}{% i…
44success801{% if user.tier %}{{ price(card(user.name + title + card(ite…
45success262{% for item in empty %}{% for item in empty %}<b>{% endfor %…
46success169{{ item + user.tier + title | upper }}
47success363{% for item in items %}{% for item in items %}{% if "ok" %}{…
48success257<b>{% for item in items %}{% for item in empty %}{{ "new" | …
49success660{% if user.tier %}{{ price(card(user.name + title) | trim) }…
50success479{% if card(item + user.tier + card(item | lower) + user.tier…

Where the calls go

The functions the most expensive input calls most, against a typical input. The function that grows most is where the work explodes.

Case #3978, 26,555,751 calls, against a median case (#523, 1,729 calls). Log scale.

render.js:evaluate 3,738,829 13,211× typicalfilters.js:<anonymous> 3,537,270 73,693× typicalrender.js:renderNode 2,359,411 15,835× typicalrender.js:lookup 2,164,342 16,032× typicalrender.js:resolve 2,164,342 16,032× typicalrender.js:toText 1,379,430 11,035× typicalfilters.js:escapeHtml 1,379,430 18,392× typicalrender.js:constructor 1,179,706 16,160× typical
Data (8 rows)
functioncalls (worst case)calls (typical case)
templates/render.js:evaluate@23453,738,829283
templates/filters.js:<anonymous>@3103,537,27048
templates/render.js:renderNode@13222,359,411149
templates/render.js:lookup@6142,164,342135
templates/render.js:resolve@38362,164,342135
templates/render.js:toText@40871,379,430125
templates/filters.js:escapeHtml@1491,379,43075
templates/render.js:constructor@5241,179,70673

Execution time per iteration

Every collected execution. Emphasized points are statistical outliers; colored ones are bugs.

executionlog scale
1 µs10 µs100 µs1 ms10 ms100 ms1 s01,0002,0003,0004,000 iteration execution time
Data (4000 rows, first 50)
iterationstatusexecution timeinput
1success812 µs<ul>{% for item in items %}{{ card(item | title) }}{% endfor…
2success129 µs<p>Hello, {{ user.name }} {{ badge(user.tier) }}</p>{{ card(…
3success79 µs</b>{{ item }}
4success103 µs{% for item in items %}{% for item in empty %}{{ price(title…
5success93 µs{% if user.name %}{% if user.name %}<li>{% for item in empty…
6success77 µs{% for item in empty %}<b>{{ card(price(card(user.tier) + ti…
7success50 µs{{ "ok" | default }}
8success55 µs{% for item in items %}{% for item in items %}{% if "ok" %}{…
9success42 µs{% for item in items %}{% for item in empty %}{% for item in…
10success28 µs{{ title }}
11success53 µs{% for item in items %}{% for item in items %}{% if user.nam…
12success143 µs{% if card("sale" | upper) %}{{ card(user.tier | lower) + us…
13success55 µs{% if badge(title | default) %}{{ "new" + "ok" + item + user…
14success46 µs{% if "new" %}{{ badge("sale" | title) + user.tier + "new" }…
15success35 µs{% for item in items %}{% for item in empty %}{{ badge(user.…
16success45 µs{% for item in items %}{% for item in empty %}{{ price(title…
17success104 µs{% if user.tier %}{{ price(card(user.name + title) | trim) }…
18success78 µs{% if user.tier %}{{ price(card(user.name + title + card(ite…
19success66 µs{% if user.tier %}{{ card(card("sale" + user.name + user.nam…
20success26 µs{% for item in items %}{% for item in items %}</li>{% endfor…
21success67 µs{% if user.tier %}{{ price(price("ok" + user.name + user.nam…
22success65 µs{% if user.tier %}{{ price(card(user.name + title + user.nam…
23success32 µs{% if item %}{{ user.tier }}Hello, {% else %}{{ "ok" + user.…
24success78 µs{% if user.tier %}{{ item | title }}{% for item in items %}{…
25success48 µs{% if card(card("new" | trim) + user.tier + user.tier + item…
26success30 µs</li>
27success97 µs{% if card("sale" | upper) %}{{ price(price(user.name + titl…
28success58 µs{% if card("sale" | trim) %}{{ "sale" }}{% for item in items…
29success84 µs{% if card(price(price(user.name + title + badge(user.tier +…
30success75 µs{% if card(price(price(user.name + title + price(price(user.…
31success68 µs{% if card(price(price(user.name + title + price(user.tier +…
32success84 µs{% if card(price(price(user.name + title + price(price(title…
33success38 µs{% if user.tier %}{{ price(card(user.name + title) | trim) }…
34success55 µs{% if user.tier %}{{ price(card(user.name + item + card(user…
35success48 µs{% if card(price(price(user.name + title + price(user.tier +…
36success71 µs{% if card(card(price(user.name + title + price(item + item)…
37success39 µs{% if user.tier %}{{ price(card(user.name + title) | trim) }…
38success20 µs{% if price(price(title) | upper) %}</p>{% endif %}
39success29 µs{% if "sale" %}Hello, {% for item in items %}{{ title + user…
40success14 µs{{ card(title) | lower }}
41success42 µs{% if title %}<p>{% for item in items %}{% for item in items…
42success24 µs{% if user.name %}</b>{{ item | title }}<li>{% for item in i…
43success25 µs{% if card(price("ok")) %}{{ price(user.name) | upper }}{% i…
44success38 µs{% if user.tier %}{{ price(card(user.name + title + card(ite…
45success19 µs{% for item in empty %}{% for item in empty %}<b>{% endfor %…
46success13 µs{{ item + user.tier + title | upper }}
47success24 µs{% for item in items %}{% for item in items %}{% if "ok" %}{…
48success33 µs<b>{% for item in items %}{% for item in empty %}{{ "new" | …
49success55 µs{% if user.tier %}{{ price(card(user.name + title) | trim) }…
50success40 µs{% if card(item + user.tier + card(item | lower) + user.tier…

How execution times are distributed

A long right tail means a few inputs are far slower than the rest.

05001,00010 µs100 µs1 ms10 ms100 ms1 s execution time (log scale) executions
Data (18 rows)
fromtoexecutions
6 µs10 µs505
10 µs18 µs780
18 µs32 µs902
32 µs56 µs533
56 µs100 µs310
100 µs178 µs370
178 µs316 µs93
316 µs562 µs473
562 µs1.00 ms9
1.00 ms1.78 ms3
1.78 ms3.16 ms1
3.16 ms5.62 ms4
5.62 ms10.00 ms2
10.00 ms17.78 ms2
17.78 ms31.62 ms3
31.62 ms56.23 ms2
100 ms178 ms2
316 ms562 ms6

Input size and execution time

Input size does not explain execution time here (best fit R² 0.36): the slow inputs are slow because of their structure, not their length.

1 µs10 µs100 µs1 ms10 ms100 ms1 s1101001,000 input size (log scale) execution time (log scale)

Spearman correlation between size and time: 0.81.

Discovery over time

When the search found new behavior. Flat curves mean the campaign has stopped learning.

Top cases 39
01020304001,0002,0003,0004,000 iteration
Distinct bugs 0
0101,0002,0003,0004,000 iteration
Covered blocks 136
05010015001,0002,0003,0004,000 iteration

Mutators

Share of each mutator's uses that improved the campaign.

grammarTree 0.9% of 3998

Performance outliers

CasefunctionCalls× medianInput
#237326,555,75115359.0×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#250126,555,75115359.0×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#337426,555,75115359.0×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#397826,555,75115359.0×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#349826,555,66015359.0×{% for item in items %}{% for item in items %}{{ price(title) }}{% if user.tier %}{{ title + card(price(user.tier) + bad…
#299726,371,43115252.4×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#34817,084,7274097.6×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#28286,639,0113839.8×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#26063,317,8861919.0×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#35713,317,8861919.0×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#34901,859,5101075.5×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#29521,709,792988.9×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#24401,660,621960.5×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#3058830,078480.1×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#2581829,754479.9×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#3854415,995240.6×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#2610388,650224.8× {% for item in items %}{% for item in items %}{{ title + badge(badge(title | trim) | upper) | upper }}{% if card(price(…
#2264207,862120.2×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#2668207,862120.2×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…
#3024207,862120.2×{% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie…

functionCalls · method iqr · bound 13,247 · median 1,729 over 4000 executions.

Regression tests

TestKindMinimized inputReduction
stays within its function-call budget (case 2373)performance{% for item in items %}{{ badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(item))))))…53% (grabr+vulcan)

Statistics

Metricnminp50p90p95p99maxmeansd
executionTime40006 µs30 µs357 µs401 µs523 µs432 ms844 µs16.76 ms
functionCalls40001241,72527,38427,92532,09926,555,75152,6871,041,412
blocks40008914,75058,12759,05168,65751,745,253104,6392,030,732
coverage400032.3%61.0%64.1%64.6%64.6%65.6%57.0%9.0%
inputSize40001192341348356360196.3892.34