quill-templates
Sample campaign: an HTML template engine with a planted bug · guided by methodCalls · seed 7
Bugs
Distinct problems, grouped by kind and by where they happen.
No errors, property failures or timeouts.
Function calls per iteration
How much code each input made the target run. Unlike time, call counts are exact and do not depend on the machine, so they expose algorithmic problems before they are slow enough to time.
Data (4000 rows, first 50)
| iteration | status | function calls | input |
|---|---|---|---|
| 1 | success | 311 | <ul>{% for item in items %}{{ card(item | title) }}{% endfor… |
| 2 | success | 319 | <p>Hello, {{ user.name }} {{ badge(user.tier) }}</p>{{ card(… |
| 3 | success | 139 | </b>{{ item }} |
| 4 | success | 278 | {% for item in items %}{% for item in empty %}{{ price(title… |
| 5 | success | 250 | {% if user.name %}{% if user.name %}<li>{% for item in empty… |
| 6 | success | 273 | {% for item in empty %}<b>{{ card(price(card(user.tier) + ti… |
| 7 | success | 138 | {{ "ok" | default }} |
| 8 | success | 372 | {% for item in items %}{% for item in items %}{% if "ok" %}{… |
| 9 | success | 260 | {% for item in items %}{% for item in empty %}{% for item in… |
| 10 | success | 138 | {{ title }} |
| 11 | success | 424 | {% for item in items %}{% for item in items %}{% if user.nam… |
| 12 | success | 462 | {% if card("sale" | upper) %}{{ card(user.tier | lower) + us… |
| 13 | success | 376 | {% if badge(title | default) %}{{ "new" + "ok" + item + user… |
| 14 | success | 335 | {% if "new" %}{{ badge("sale" | title) + user.tier + "new" }… |
| 15 | success | 288 | {% for item in items %}{% for item in empty %}{{ badge(user.… |
| 16 | success | 280 | {% for item in items %}{% for item in empty %}{{ price(title… |
| 17 | success | 701 | {% if user.tier %}{{ price(card(user.name + title) | trim) }… |
| 18 | success | 810 | {% if user.tier %}{{ price(card(user.name + title + card(ite… |
| 19 | success | 771 | {% if user.tier %}{{ card(card("sale" + user.name + user.nam… |
| 20 | success | 220 | {% for item in items %}{% for item in items %}</li>{% endfor… |
| 21 | success | 731 | {% if user.tier %}{{ price(price("ok" + user.name + user.nam… |
| 22 | success | 553 | {% if user.tier %}{{ price(card(user.name + title + user.nam… |
| 23 | success | 185 | {% if item %}{{ user.tier }}Hello, {% else %}{{ "ok" + user.… |
| 24 | success | 747 | {% if user.tier %}{{ item | title }}{% for item in items %}{… |
| 25 | success | 500 | {% if card(card("new" | trim) + user.tier + user.tier + item… |
| 26 | success | 124 | </li> |
| 27 | success | 1,064 | {% if card("sale" | upper) %}{{ price(price(user.name + titl… |
| 28 | success | 351 | {% if card("sale" | trim) %}{{ "sale" }}{% for item in items… |
| 29 | success | 1,254 | {% if card(price(price(user.name + title + badge(user.tier +… |
| 30 | success | 1,778 | {% if card(price(price(user.name + title + price(price(user.… |
| 31 | success | 1,247 | {% if card(price(price(user.name + title + price(user.tier +… |
| 32 | success | 1,971 | {% if card(price(price(user.name + title + price(price(title… |
| 33 | success | 701 | {% if user.tier %}{{ price(card(user.name + title) | trim) }… |
| 34 | success | 947 | {% if user.tier %}{{ price(card(user.name + item + card(user… |
| 35 | success | 1,239 | {% if card(price(price(user.name + title + price(user.tier +… |
| 36 | success | 1,062 | {% if card(card(price(user.name + title + price(item + item)… |
| 37 | success | 701 | {% if user.tier %}{{ price(card(user.name + title) | trim) }… |
| 38 | success | 272 | {% if price(price(title) | upper) %}</p>{% endif %} |
| 39 | success | 460 | {% if "sale" %}Hello, {% for item in items %}{{ title + user… |
| 40 | success | 189 | {{ card(title) | lower }} |
| 41 | success | 866 | {% if title %}<p>{% for item in items %}{% for item in items… |
| 42 | success | 330 | {% if user.name %}</b>{{ item | title }}<li>{% for item in i… |
| 43 | success | 409 | {% if card(price("ok")) %}{{ price(user.name) | upper }}{% i… |
| 44 | success | 801 | {% if user.tier %}{{ price(card(user.name + title + card(ite… |
| 45 | success | 262 | {% for item in empty %}{% for item in empty %}<b>{% endfor %… |
| 46 | success | 169 | {{ item + user.tier + title | upper }} |
| 47 | success | 363 | {% for item in items %}{% for item in items %}{% if "ok" %}{… |
| 48 | success | 257 | <b>{% for item in items %}{% for item in empty %}{{ "new" | … |
| 49 | success | 660 | {% if user.tier %}{{ price(card(user.name + title) | trim) }… |
| 50 | success | 479 | {% if card(item + user.tier + card(item | lower) + user.tier… |
Where the calls go
The functions the most expensive input calls most, against a typical input. The function that grows most is where the work explodes.
Case #3978, 26,555,751 calls, against a median case (#523, 1,729 calls). Log scale.
Data (8 rows)
| function | calls (worst case) | calls (typical case) |
|---|---|---|
| templates/render.js:evaluate@2345 | 3,738,829 | 283 |
| templates/filters.js:<anonymous>@310 | 3,537,270 | 48 |
| templates/render.js:renderNode@1322 | 2,359,411 | 149 |
| templates/render.js:lookup@614 | 2,164,342 | 135 |
| templates/render.js:resolve@3836 | 2,164,342 | 135 |
| templates/render.js:toText@4087 | 1,379,430 | 125 |
| templates/filters.js:escapeHtml@149 | 1,379,430 | 75 |
| templates/render.js:constructor@524 | 1,179,706 | 73 |
Execution time per iteration
Every collected execution. Emphasized points are statistical outliers; colored ones are bugs.
Data (4000 rows, first 50)
| iteration | status | execution time | input |
|---|---|---|---|
| 1 | success | 812 µs | <ul>{% for item in items %}{{ card(item | title) }}{% endfor… |
| 2 | success | 129 µs | <p>Hello, {{ user.name }} {{ badge(user.tier) }}</p>{{ card(… |
| 3 | success | 79 µs | </b>{{ item }} |
| 4 | success | 103 µs | {% for item in items %}{% for item in empty %}{{ price(title… |
| 5 | success | 93 µs | {% if user.name %}{% if user.name %}<li>{% for item in empty… |
| 6 | success | 77 µs | {% for item in empty %}<b>{{ card(price(card(user.tier) + ti… |
| 7 | success | 50 µs | {{ "ok" | default }} |
| 8 | success | 55 µs | {% for item in items %}{% for item in items %}{% if "ok" %}{… |
| 9 | success | 42 µs | {% for item in items %}{% for item in empty %}{% for item in… |
| 10 | success | 28 µs | {{ title }} |
| 11 | success | 53 µs | {% for item in items %}{% for item in items %}{% if user.nam… |
| 12 | success | 143 µs | {% if card("sale" | upper) %}{{ card(user.tier | lower) + us… |
| 13 | success | 55 µs | {% if badge(title | default) %}{{ "new" + "ok" + item + user… |
| 14 | success | 46 µs | {% if "new" %}{{ badge("sale" | title) + user.tier + "new" }… |
| 15 | success | 35 µs | {% for item in items %}{% for item in empty %}{{ badge(user.… |
| 16 | success | 45 µs | {% for item in items %}{% for item in empty %}{{ price(title… |
| 17 | success | 104 µs | {% if user.tier %}{{ price(card(user.name + title) | trim) }… |
| 18 | success | 78 µs | {% if user.tier %}{{ price(card(user.name + title + card(ite… |
| 19 | success | 66 µs | {% if user.tier %}{{ card(card("sale" + user.name + user.nam… |
| 20 | success | 26 µs | {% for item in items %}{% for item in items %}</li>{% endfor… |
| 21 | success | 67 µs | {% if user.tier %}{{ price(price("ok" + user.name + user.nam… |
| 22 | success | 65 µs | {% if user.tier %}{{ price(card(user.name + title + user.nam… |
| 23 | success | 32 µs | {% if item %}{{ user.tier }}Hello, {% else %}{{ "ok" + user.… |
| 24 | success | 78 µs | {% if user.tier %}{{ item | title }}{% for item in items %}{… |
| 25 | success | 48 µs | {% if card(card("new" | trim) + user.tier + user.tier + item… |
| 26 | success | 30 µs | </li> |
| 27 | success | 97 µs | {% if card("sale" | upper) %}{{ price(price(user.name + titl… |
| 28 | success | 58 µs | {% if card("sale" | trim) %}{{ "sale" }}{% for item in items… |
| 29 | success | 84 µs | {% if card(price(price(user.name + title + badge(user.tier +… |
| 30 | success | 75 µs | {% if card(price(price(user.name + title + price(price(user.… |
| 31 | success | 68 µs | {% if card(price(price(user.name + title + price(user.tier +… |
| 32 | success | 84 µs | {% if card(price(price(user.name + title + price(price(title… |
| 33 | success | 38 µs | {% if user.tier %}{{ price(card(user.name + title) | trim) }… |
| 34 | success | 55 µs | {% if user.tier %}{{ price(card(user.name + item + card(user… |
| 35 | success | 48 µs | {% if card(price(price(user.name + title + price(user.tier +… |
| 36 | success | 71 µs | {% if card(card(price(user.name + title + price(item + item)… |
| 37 | success | 39 µs | {% if user.tier %}{{ price(card(user.name + title) | trim) }… |
| 38 | success | 20 µs | {% if price(price(title) | upper) %}</p>{% endif %} |
| 39 | success | 29 µs | {% if "sale" %}Hello, {% for item in items %}{{ title + user… |
| 40 | success | 14 µs | {{ card(title) | lower }} |
| 41 | success | 42 µs | {% if title %}<p>{% for item in items %}{% for item in items… |
| 42 | success | 24 µs | {% if user.name %}</b>{{ item | title }}<li>{% for item in i… |
| 43 | success | 25 µs | {% if card(price("ok")) %}{{ price(user.name) | upper }}{% i… |
| 44 | success | 38 µs | {% if user.tier %}{{ price(card(user.name + title + card(ite… |
| 45 | success | 19 µs | {% for item in empty %}{% for item in empty %}<b>{% endfor %… |
| 46 | success | 13 µs | {{ item + user.tier + title | upper }} |
| 47 | success | 24 µs | {% for item in items %}{% for item in items %}{% if "ok" %}{… |
| 48 | success | 33 µs | <b>{% for item in items %}{% for item in empty %}{{ "new" | … |
| 49 | success | 55 µs | {% if user.tier %}{{ price(card(user.name + title) | trim) }… |
| 50 | success | 40 µs | {% if card(item + user.tier + card(item | lower) + user.tier… |
How execution times are distributed
A long right tail means a few inputs are far slower than the rest.
Data (18 rows)
| from | to | executions |
|---|---|---|
| 6 µs | 10 µs | 505 |
| 10 µs | 18 µs | 780 |
| 18 µs | 32 µs | 902 |
| 32 µs | 56 µs | 533 |
| 56 µs | 100 µs | 310 |
| 100 µs | 178 µs | 370 |
| 178 µs | 316 µs | 93 |
| 316 µs | 562 µs | 473 |
| 562 µs | 1.00 ms | 9 |
| 1.00 ms | 1.78 ms | 3 |
| 1.78 ms | 3.16 ms | 1 |
| 3.16 ms | 5.62 ms | 4 |
| 5.62 ms | 10.00 ms | 2 |
| 10.00 ms | 17.78 ms | 2 |
| 17.78 ms | 31.62 ms | 3 |
| 31.62 ms | 56.23 ms | 2 |
| 100 ms | 178 ms | 2 |
| 316 ms | 562 ms | 6 |
Input size and execution time
Input size does not explain execution time here (best fit R² 0.36): the slow inputs are slow because of their structure, not their length.
Spearman correlation between size and time: 0.81.
Discovery over time
When the search found new behavior. Flat curves mean the campaign has stopped learning.
Mutators
Share of each mutator's uses that improved the campaign.
Performance outliers
| Case | functionCalls | × median | Input |
|---|---|---|---|
| #2373 | 26,555,751 | 15359.0× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #2501 | 26,555,751 | 15359.0× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #3374 | 26,555,751 | 15359.0× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #3978 | 26,555,751 | 15359.0× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #3498 | 26,555,660 | 15359.0× | {% for item in items %}{% for item in items %}{{ price(title) }}{% if user.tier %}{{ title + card(price(user.tier) + bad… |
| #2997 | 26,371,431 | 15252.4× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #3481 | 7,084,727 | 4097.6× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #2828 | 6,639,011 | 3839.8× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #2606 | 3,317,886 | 1919.0× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #3571 | 3,317,886 | 1919.0× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #3490 | 1,859,510 | 1075.5× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #2952 | 1,709,792 | 988.9× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #2440 | 1,660,621 | 960.5× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #3058 | 830,078 | 480.1× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #2581 | 829,754 | 479.9× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #3854 | 415,995 | 240.6× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #2610 | 388,650 | 224.8× | {% for item in items %}{% for item in items %}{{ title + badge(badge(title | trim) | upper) | upper }}{% if card(price(… |
| #2264 | 207,862 | 120.2× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #2668 | 207,862 | 120.2× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
| #3024 | 207,862 | 120.2× | {% for item in items %}{% for item in items %}{{ price(title) | upper }}{% if user.tier %}{{ title + card(price(user.tie… |
functionCalls · method iqr · bound 13,247 · median 1,729 over 4000 executions.
Regression tests
| Test | Kind | Minimized input | Reduction |
|---|---|---|---|
| stays within its function-call budget (case 2373) | performance | {% for item in items %}{{ badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(badge(item))))))… | 53% (grabr+vulcan) |
Statistics
| Metric | n | min | p50 | p90 | p95 | p99 | max | mean | sd |
|---|---|---|---|---|---|---|---|---|---|
| executionTime | 4000 | 6 µs | 30 µs | 357 µs | 401 µs | 523 µs | 432 ms | 844 µs | 16.76 ms |
| functionCalls | 4000 | 124 | 1,725 | 27,384 | 27,925 | 32,099 | 26,555,751 | 52,687 | 1,041,412 |
| blocks | 4000 | 891 | 4,750 | 58,127 | 59,051 | 68,657 | 51,745,253 | 104,639 | 2,030,732 |
| coverage | 4000 | 32.3% | 61.0% | 64.1% | 64.6% | 64.6% | 65.6% | 57.0% | 9.0% |
| inputSize | 4000 | 1 | 192 | 341 | 348 | 356 | 360 | 196.38 | 92.34 |