Competition standings
ferroplan measures itself on 32 IPC boards — every deterministic track it
can enter from IPC-5 (2006) through IPC-2026: classical
satisficing, temporal, PDDL3 preferences and constraints, net-benefit,
numeric, the multi-core track under its wall-clock rule, and seven optimal
tracks where coverage is proof rate. Budgets are 60 s per instance (300 s
for the one official-budget agile entry), one thread, two instances at a
time on the sweep box, every reported plan externally validated by
VAL. The tables below are
GENERATED from the raw per-instance sweep logs (python3 benchmarks/standings.py, and the crucible's own renderer must agree with
it byte for byte) and are refreshed against the final binary at every
release cut — scoreboards defend themselves.
Three kinds of honesty markers appear throughout:
- Reference-scored means scored against the official competition
field. The IPC-5 preference boards are scored from the vendored official
results archive (
benchmarks/IPC5-results.tgz— seebenchmarks/ATTRIBUTION.mdfor provenance) by the IPC quality score against SGPlan5's own; MetricValue, the currency those tracks were ranked on: level with the winner on rows, behind on points, and the table says so. The IPC-5 propositional, time and metric-time tracks are quality-scored by plan length and makespan against the archive field; the 2018 and 2023 classical boards against the best-known bounds. - Coverage-only means no aligned reference exists: no official per-instance archive is vendored for IPC-6/7 and the 2014 corpus, and the constraints board records no quality currency.
- vs field (in
STANDINGS.md) is a coverage-RATE placement against the competition's actual entrants at ~1/30th of the official budget — a floor, not a result;docs/ipc-rankings.mdholds the provenance.
Failure classes are counted per unsolved instance: timeout (budget
exhausted), mem-cap (the runner's memory cap hit — tracked separately
from engine verdicts), early-exit (the engine returned without a plan
before its budget), engine-reject/error (a named rejection —
hold-during / hold-after, or a parse error), spawn-fail and
VAL-RED (the harness's own failures, never silently folded in).
The optimal tracks (⚖️) are entered since 0.19 under Mode::Optimal — A*
with an admissible LM-cut heuristic — and their coverage IS proof rate.
The sequential multi-core tracks are entered under their competition rule
(wall-clock with all cores; per-thread-count determinism still holds) on
the sweep box's 4P+6E cores, with the t8 row marked as oversubscribed.
IPC standings — the one honest table per competition
Generated by python3 benchmarks/standings.py (do not hand-edit;
regenerate after any sweep). Raw inputs are the per-instance JSONLs
and the vendored official IPC-5 archive — see the module docstring
for scoring semantics and the failure-class definitions.
IPC-5 (2006)
| track | entered | coverage | quality | failure classes |
|---|---|---|---|---|
| propositional | yes | 386/450 | len vs best-of-field: 54W/44T/182L, mean quality 0.90 (280 scored) | 64 timeout |
| time | yes | 126/130 | makespan vs best-of-field: 31W/3T/92L, mean quality 0.65 (126 scored) | 4 timeout |
| metric-time | yes | 119/200 | makespan vs best-of-field: 75W/1T/32L, mean quality 0.93 (108 scored) | 11 mem-cap, 70 timeout |
| constraints | yes | 28/120 | coverage-only (no quality currency recorded) | 15 early-exit, 8 mem-cap, 69 timeout, 7 solved VAL-unavailable (engine-oracle only; see benchmarks/val-availability.py) |
| simple-preferences (full corpus) | yes | 130/130 | IPC score vs SGPlan5: 97.3 / 115.1 over 130 cells (32W/21T/76L; 1 solved unpriced) | none |
| qualitative-preferences (full corpus) | yes | 100/100 | IPC score vs SGPlan5: 64.6 / 91.4 over 100 cells (28W/4T/68L; 0 solved unpriced) | 12 solved VAL-unavailable (engine-oracle only; see benchmarks/val-availability.py) |
| complex-preferences (full corpus) | yes | 81/108 | IPC score vs SGPlan5: 54.5 / 100.0 over 105 cells (12W/5T/61L; 1 solved unpriced) | 14 mem-cap, 13 timeout, 4 solved VAL-unavailable (engine-oracle only; see benchmarks/val-availability.py) |
| simple-preferences | yes | see board | reference-scored — ipc5-scoreboard.md | — |
| qualitative-preferences | yes | see board | reference-scored — ipc5-qualitative-scoreboard.md (24W/4T/10L vs SGPlan5 — ahead of the winner; rovers/storage/tpp won outright) | — |
| complex-preferences | superseded — the full-corpus board above (entered 0.25) | — | — | — |
IPC-6 (2008)
| track | entered | coverage | quality | failure classes |
|---|---|---|---|---|
| seq-sat | yes | 298/300 | coverage + VAL (no official per-instance archive vendored) | 2 timeout |
| tempo-sat | yes | 338/390 | coverage + VAL (no official per-instance archive vendored) | 5 mem-cap, 47 timeout |
| net-benefit | yes | 270/270 | coverage + VAL (no official per-instance archive vendored) | none |
| seq-opt | yes (first entry, 0.19 — Mode::Optimal) | 157/270 | coverage = PROOF RATE (A* + admissible LM-cut, h^max sprint first; every plan certified + VAL) | 113 timeout |
| tempo-opt | out of scope by design (satisficing temporal path) | — | — | — |
IPC-7 (2011)
| track | entered | coverage | quality | failure classes |
|---|---|---|---|---|
| seq-sat | yes | 245/280 | coverage + VAL | 35 timeout |
| tempo-sat | yes | 174/240 | coverage + VAL | 8 mem-cap, 58 timeout |
| seq-mco t2 | yes (first entry, 0.16) | 249/280 | wall-clock per competition rule (--threads N, one instance at a time; 4P+6E box — t8 oversubscribed by construction) | 31 timeout |
| seq-mco t4 | yes (first entry, 0.16) | 253/280 | wall-clock per competition rule (--threads N, one instance at a time; 4P+6E box — t8 oversubscribed by construction) | 1 mem-cap, 26 timeout |
| seq-mco t8 | yes (first entry, 0.16) | 255/280 | wall-clock per competition rule (--threads N, one instance at a time; 4P+6E box — t8 oversubscribed by construction) | 1 early-exit, 24 timeout |
| seq-opt | yes (first entry, 0.19 — Mode::Optimal) | 140/280 | coverage = PROOF RATE (A* + admissible LM-cut, h^max sprint first; every plan certified + VAL) | 140 timeout |
The modern corpora (IPC 2014 / 2018 / 2023 — first entered 0.17)
| track | entered | coverage | quality | failure classes |
|---|---|---|---|---|
| 2014 seq-sat | yes (first entry, 0.17) | 180/280 | coverage + VAL | 4 mem-cap, 96 timeout |
| 2014 seq-agile | yes (first entry, 0.17) | 174/280 | coverage + VAL | 1 early-exit, 1 mem-cap, 104 timeout |
| 2014 tempo-sat | yes (first entry, 0.17) | 124/200 | coverage + VAL | 2 mem-cap, 74 timeout |
| 2014 seq-mco t2 | yes (FIRST ENTRY, 0.25 — the table grows) | 181/280 | wall-clock per competition rule (one instance at a time; 4P+6E box) | 6 mem-cap, 93 timeout |
| 2014 seq-mco t4 | yes (first entry, 0.17) | 184/280 | wall-clock per competition rule (--threads 4, one instance at a time; 4P+6E box) | 7 mem-cap, 89 timeout |
| 2014 seq-mco t8 | yes (FIRST ENTRY, 0.25 — the table grows) | 186/280 | wall-clock per competition rule (one instance at a time; 4P+6E box, t8 oversubscribed by construction) | 11 mem-cap, 83 timeout |
| 2014 seq-opt | yes (first entry, 0.19) | 82/256 | coverage = PROOF RATE (Mode::Optimal, A* + admissible LM-cut, h^max sprint first; every plan certified + VAL) | 174 timeout |
| 2018 seq-sat | yes (first entry, 0.17) | 99/240 | vs best-known bounds: 0W/2T/30L, mean quality 0.79 (32 scored) | 24 mem-cap, 117 timeout, 10 solved VAL-unavailable (engine-oracle only; see benchmarks/val-availability.py) |
| 2018 seq-opt | yes (FIRST ENTRY, 0.25 — the table grows) | 94/240 | coverage = PROOF RATE (Mode::Optimal, A* + admissible LM-cut, h^max sprint first; every plan certified + VAL) | 21 mem-cap, 125 timeout, 10 solved VAL-unavailable (engine-oracle only; see benchmarks/val-availability.py) |
| 2023 classical | yes (first entry, 0.17) | 52/140 | vs best-known bounds: 1W/21T/30L, mean quality 0.84 (52 scored) | 17 mem-cap, 71 timeout |
| 2023 seq-sat | yes (FIRST ENTRY, 0.25 — the table grows) | 52/140 | vs best-known bounds: 0W/22T/30L, mean quality 0.84 (52 scored) | 20 mem-cap, 68 timeout |
| 2023 seq-opt | yes (FIRST ENTRY, 0.25 — the table grows) | 33/140 | coverage = PROOF RATE (Mode::Optimal, A* + admissible LM-cut, h^max sprint first; every plan certified + VAL) | 8 mem-cap, 99 timeout |
| 2023 agile ENTRY (300s) | yes (OFFICIAL-BUDGET entry, 0.19) | 67/140 | OFFICIAL 300 s budget — a competition-methodology ENTRY, not a baseline | 17 mem-cap, 56 timeout |
| 2023 numeric | yes (first entry, 0.17) | 289/400 | field CSVs vendored (ipc-2023n/results) — per-domain comparison in the audit record | 1 engine-reject/error, 52 mem-cap, 58 timeout, 47 solved VAL-unavailable (engine-oracle only; see benchmarks/val-availability.py) |
| 2023 numeric-opt | yes (FIRST ENTRY, 0.25 — the table grows) | 82/400 | coverage = PROOF RATE over the numeric corpus; the track's official field CSV (ipc-2023n/results/opt.csv) is the vs-field referee | 137 early-exit, 1 engine-reject/error, 86 mem-cap, 94 timeout, 3 solved VAL-unavailable (engine-oracle only; see benchmarks/val-availability.py) |
| 2026 numeric (first board) | yes (FIRST ENTRY, 0.20 — new corpus) | 224/320 | coverage + VAL; the corpus ships -sat/-opt domain PAIRS, all swept satisficing-style on this first board | 2 early-exit, 94 timeout, 11 solved VAL-unavailable (engine-oracle only; see benchmarks/val-availability.py) |
| 2026 numeric-opt | yes (FIRST ENTRY, 0.21 — the -opt pairs, ⚖️) | 23/60 | coverage = PROOF RATE (Mode::Optimal over the three -opt pairs; LENGTH optima — the vendored corpus carries no active :metric; every certificate VAL-checked) | 21 early-exit, 16 timeout |
| 2026 numeric-opt FULL | yes (FIRST ENTRY, 0.25 — the table grows) | 80/260 | coverage = PROOF RATE over the official 13-domain/260 Overall Optimal constituency (the 3-pair board above is the like-for-like slice) | 112 early-exit, 24 mem-cap, 44 timeout |
The 2023 classical corpus is swept on its agile instances at the standard 60 s satisficing budget (the competition's agile budget is 300 s — these rows are BASELINES, marked as such, not competition entries).
Reading the boards
benchmarks/ipc5-scoreboard.md— IPC-5 simple preferences, ferroplan vs the official field.benchmarks/ipc5-qualitative-scoreboard.md— IPC-5 qualitative preferences, reference-grafted with the full W/T/L accounting.benchmarks/ipc67-results.md/benchmarks/ipc67-temporal.md/benchmarks/ipc67-netben.md— the standing IPC-6/7 scoreboards (seq-sat, tempo-sat, net-benefit), per-variant.benchmarks/ipc5-prop.mdand siblings (ipc5-time,ipc5-metric-time,ipc5-constraints) — the 2006 deterministic-track sweeps, first entered in 0.16.benchmarks/ipc7-mco-t{2,4,8}.md— the multi-core rows.
Per-cycle history (what moved and why, cut by cut) lives in the
docs/roadmap-0.*.md records; this chapter is always the CURRENT
standing.