Template field reference
plan.template.json, binding.template.json and outcomes.template.json in this directory are minimal, internally-consistent, one-trial examples of the three files scripts/challenge.py's evaluate() scores (see ../../README.md#the-three-records for why there are three separate files instead of one). They are templates, not submittable packets -- every field a real submitter must supply is marked TEMPLATE -- replace: with an example of the kind of value expected. A real submission also needs at least 8 held-out providers, each with at least 10 trials / 5 sessions / 3 UTC dates -- see "Minimum you need before you start" in ../../SUBMITTING.md; these templates show one trial for one provider purely to keep the file readable.
Each template carries a _template_note root field that is not part of the real schema -- evaluate() ignores unrecognized fields, but delete it before submitting anyway.
The plan_sha256 inside binding.template.json, and the plan_sha256/ binding_sha256 inside outcomes.template.json, are the real, freshly computed SHA-256 hashes of the other two template files' exact bytes (see "Exact hashing rules" in SUBMITTING.md) -- that's why the three templates validate as a hash-consistent triple against each other, even though the rest of their content is placeholder. Once you edit plan.template.json, its bytes change and its hash changes with them; you must recompute plan_sha256 in your binding and outcomes files (or just run scripts/csv_to_packet.py, which always computes fresh hashes for you).
plan.template.json (schema secondrun.rating-plan.v4)
| Field | Meaning |
|---|---|
schema | Always the literal string secondrun.rating-plan.v4. |
synthetic | true/false. false for a real submission -- true is reserved for the repo's own demo/test fixtures. |
study_id | A short slug you choose, identifying this study. Free text, must be nonempty. |
rating_name | The rating this plan is about, e.g. "ClusterMAX". Must match binding.json's rating_name exactly. |
rating_version | The version you expect the binding to name, e.g. "3.0". Must match binding.json's rating_version exactly (this is how the scorer catches a binding to the wrong rating vintage). |
frozen_at | RFC3339 UTC timestamp: when you locked this design, before any trial's predicted_at/started_at. Everything below this point in the plan must exist by this moment. |
minimum_providers | Integer, must be >= 8. The hard floor on distinct held-out providers. |
minimum_lift | Number in [0, 1]: the smallest medal-permutation-test ranking gain T (see "The medal permutation test" in ../../README.md) you'd call a real (not just nonzero) positive/negative signal, decided before you see outcomes. |
outcome_definition | Free text: the plain-English endpoint your per-trial gates encode (e.g. "complete with >=950/1000 accepted, p95<=500ms, cost<=$25"). |
disclosure | Object with five required, nonempty, non-placeholder string fields -- see below. evaluate() holds the packet if any is empty or still says CONFIRM/TBD/TODO/???/"describe"/etc. |
disclosure.relationship | Your relationship, if any, to any provider or rating named in this plan (employee, investor, customer, none, ...). |
disclosure.compensation | Any compensation you received connected to this plan or its providers/rating. |
disclosure.credits | Any GPU-cloud credits received (name provider + amount, or state none). |
disclosure.special_support | Any non-ordinary account access or support you received (or state none -- customer_role/support on every trial must independently say ordinary_tenant/standard regardless of what you write here). |
disclosure.editorial_influence | Whether any named party reviewed, approved or influenced this plan before you froze it. |
binding_rules | Object committing, in advance, to which official sources the later binding must cite and which tier set it must use. |
binding_rules.medal_source | Free text naming the official source you expect the binding's source to point at (e.g. "the official ClusterMAX medal table page"). |
binding_rules.rubric_source | Same, for the rubric the binding's rubric will point at. |
binding_rules.tiers | List of tier names -- must be exactly the five keys of transform.anchors (order doesn't matter, set equality does). |
transform | The complete medal-to-probability transform object, inline -- copy the exact contents of ../../design/transform.json; its canonical hash must match an entry in ../../design/registry.json's transforms array or the packet holds. Schema secondrun.rating-transform.v1, with anchors (one probability per medal tier, strictly decreasing Platinum > Gold > Silver > Bronze > Underperforming), weight, reference_anchor, alternative_weights, description. |
transform_sha256 | SHA-256 of the canonical JSON encoding of the transform object above -- json.dumps(transform, sort_keys=True, separators=(',',':'), ensure_ascii=False), then SHA-256 of the UTF-8 bytes. This is the one hash in this repo that is not a plain file-bytes hash -- see "The medal-to-probability transform" in README.md. scripts/csv_to_packet.py and make_demo.py both compute it the same way; do it by hand only if you're not using the converter. |
baseline / with_rating | The two predictor definitions for the ablation -- same shape, with_rating.inputs must be baseline.inputs plus exactly one extra input named "rating", and both must share the same training_providers set. |
*.description | Free text describing the predictor. |
*.model_sha256 | SHA-256 of the exact file/artifact that defines this predictor (a model file, a spec document, a scoring formula -- whatever you'd need someone else to inspect to check your predictor). |
*.inputs | List of distinct input-feature names. |
*.training_providers | Nonempty list of provider ids used to fit/tune this predictor -- an empty list holds the packet (the held-out claim is vacuous without training providers), and every id must also be disjoint from every trial's provider_id below (a held-out provider leaking into training holds the packet). |
*.frozen_at | RFC3339 timestamp, must be <= plan.frozen_at. |
trials | List of per-job plan rows (the "design" side; outcomes.json supplies the matching terminal-outcome side per trial_id). |
trials[].trial_id | Your own unique id for this planned job; must also appear, unchanged, in the matching outcomes.json row. |
trials[].provider_id | Which held-out provider this trial tests. If your binding names a registered rating version (currently ClusterMAX 3.0), this must exactly match that provider's key in the registered transcription -- see ../../launch/release-3.0/clustermax-3.0.json (or design/registry.json's ratings[].transcription.tiers) for the exact spelling. |
trials[].service_id | Which specific service/product/SKU on that provider (a managed-cluster claim needs a managed-cluster service_id, not a bare-metal or token-endpoint one). |
trials[].region | Region/location of the job. |
trials[].workload_sha256 | SHA-256 of the exact workload spec file this job ran -- see "What workload_sha256 and validator_sha256 mean" in SUBMITTING.md. |
trials[].validator_sha256 | SHA-256 of the exact acceptance-validator file/script used to decide accepted/attempted for this job. |
trials[].customer_role | Must be the literal string "ordinary_tenant" -- anything else holds the packet. |
trials[].support | Must be the literal string "standard" -- anything else holds the packet. |
trials[].session_id | Which test session this trial belongs to (used for the >=5-distinct-sessions-per-provider floor). |
trials[].predicted_at | RFC3339 timestamp of this specific prediction; must be <= plan.frozen_at and strictly before this trial's started_at in outcomes.json. |
trials[].p_baseline | Number in [0, 1]: the baseline model's predicted probability of success for this job, computed before the job ran. |
trials[].gates | Object: accepted_min (positive integer), p95_max_ms (number), cost_max_usd (number) -- the pass/fail thresholds for this specific job. These must be byte-for-byte identical across every trial sharing a workload_sha256 -- the scorer holds the packet otherwise -- decided before you know the provider's medal; see "Minimum you need before you start" in SUBMITTING.md. |
binding.template.json (schema secondrun.rating-binding.v2)
| Field | Meaning |
|---|---|
schema | Always secondrun.rating-binding.v2. |
plan_sha256 | SHA-256 of the exact bytes of the plan.json file this binding is for. Not canonicalized -- literally sha256(open('plan.json','rb').read()). |
rating_name / rating_version | Must equal the plan's rating_name/rating_version exactly. |
source | Where you got the medal table. source.url, source.sha256 (of the exact page/image bytes you retrieved), source.retrieved_utc (RFC3339, must be <= bound_at). |
rubric | Same shape, for the rubric/methodology document the medals were graded against. |
bound_at | RFC3339 timestamp: when you actually filled in this binding. Must be >= plan.frozen_at and strictly before every trial's started_at in outcomes.json -- you cannot bind a medal after a job already ran. |
medals | Object mapping every provider_id that appears in plan.trials (no more, no fewer) to one of the five frozen tier names (Platinum/Gold/Silver/Bronze/Underperforming). An unrated/Unavailable provider cannot be silently mapped to a tier -- drop it from the plan's provider set instead. If rating_name/rating_version names a rating registered in design/registry.json (currently ClusterMAX 3.0), every key here must be spelled exactly as in that rating's registered transcription and its value must equal that provider's registered tier, or the packet holds. |
outcomes.template.json (schema secondrun.rating-outcomes.v4)
| Field | Meaning |
|---|---|
schema | Always secondrun.rating-outcomes.v4. |
plan_sha256 | Same value as binding.json's plan_sha256 (both must point at the same plan bytes). |
binding_sha256 | SHA-256 of the exact bytes of the binding.json file. |
trials | List of terminal outcome rows, one per plan.trials[].trial_id, no more, no fewer. |
trials[].trial_id ... session_id | Must exactly match the corresponding fields on the plan.json row with the same trial_id (the scorer checks every CONTEXT field for an exact match -- a changed value here is treated as a plan/outcome mismatch, not an update). |
trials[].started_at | RFC3339 timestamp the job actually started; must be strictly after this trial's predicted_at and after plan.frozen_at. |
trials[].finished_at | RFC3339 timestamp the job ended; must be >= started_at. |
trials[].status | One of complete, timeout, error, provision_failed, aborted. Every attempted trial gets a terminal status -- don't drop failed ones. |
trials[].accepted / trials[].attempted | Integers, 0 <= accepted <= attempted. |
trials[].p95_ms | Number, or JSON null if genuinely unavailable (e.g. the job never completed) -- but a status: "complete" row must have a real number here. |
trials[].total_cost_usd | The normal list-price charge for this job, before any credits. Always what gates and cost_per_1000_accepted are computed from. |
trials[].credits_redeemed_usd | Optional, defaults to 0. Any GPU-cloud credits redeemed toward this specific job; cannot exceed total_cost_usd. |
trials[].receipt_ref | Free text pointing at your actual receipt/invoice (an id, filename, or URL) -- don't publish the receipt itself if it has billing details you want private. |
trials[].receipt_sha256 | SHA-256 of that receipt/invoice file. |
CSV_to_packet.py header JSON
scripts/csv_to_packet.py takes a jobs CSV (one row per trial: trial_id, provider_id, service_id, region, session_id, predicted_at, p_baseline, started_at, finished_at, status, accepted, attempted, p95_ms, total_cost_usd, credits_redeemed_usd, receipt_ref, receipt_sha256) plus a second, small JSON file -- the "header" -- carrying everything that's constant across the cohort rather than per-row. The header's top-level fields map straight onto the plan.template.json fields documented above, plus a binding object and the two shared per-job hashes/gates that the CSV itself does not carry a column for:
| Header field | Goes into | Meaning |
|---|---|---|
study_id, rating_name, rating_version, frozen_at, minimum_providers, minimum_lift, outcome_definition, disclosure, binding_rules, transform, baseline, with_rating | plan.json (same field name, copied straight through) | See the plan.template.json table above -- identical meaning. |
synthetic | plan.json | Optional, defaults to false. |
workload_sha256 | every plan.json/outcomes.json trial row | The one workload hash applied to every CSV row. If you ran more than one distinct workload, split your CSV (and run the converter once per workload) rather than trying to vary this per row -- the converter does not support a per-row override. |
validator_sha256 | every plan.json/outcomes.json trial row | Same, for the acceptance validator. |
customer_role | every plan.json/outcomes.json trial row | Optional, defaults to "ordinary_tenant". |
support | every plan.json/outcomes.json trial row | Optional, defaults to "standard". |
gates | every plan.json trial row | The shared {accepted_min, p95_max_ms, cost_max_usd} object applied to every row -- see "Uniform gates per workload" in SUBMITTING.md. |
binding.source, binding.rubric, binding.bound_at, binding.medals | binding.json | Copied straight into the generated binding.json -- see the binding.template.json table above. |
Worked example header JSON (paired with a CSV that has one row per trial for providers ACME_CLOUD and ZETA_GPU, and so on for at least 6 more to clear the 8-provider floor):
{
"study_id": "acme-cloud-inference-2026-11",
"rating_name": "ClusterMAX",
"rating_version": "3.0",
"frozen_at": "2026-11-01T00:00:00Z",
"minimum_providers": 8,
"minimum_lift": 0.01,
"outcome_definition": "Complete with >=950/1000 accepted, p95<=500ms, cost<=$25.",
"disclosure": {
"relationship": "No relationship with any provider or rating named in this plan.",
"compensation": "None received.",
"credits": "None redeemed toward these jobs.",
"special_support": "None; ordinary self-signup account and standard support only.",
"editorial_influence": "None."
},
"binding_rules": {
"medal_source": "Official ClusterMAX medal table page, as published by SemiAnalysis.",
"rubric_source": "Official ClusterMAX rubric/methodology document for the bound rating_version.",
"tiers": ["Platinum", "Gold", "Silver", "Bronze", "Underperforming"]
},
"transform": { "...": "paste the exact contents of design/transform.json here" },
"baseline": {
"description": "Specs-and-price baseline: GPU type, allocation size, list price/hour.",
"model_sha256": "<sha256 of your baseline model/spec file>",
"inputs": ["gpu_type", "allocation_size", "list_price_per_hour", "workload_class", "region"],
"training_providers": ["TRAIN_PROVIDER_A", "TRAIN_PROVIDER_B"],
"frozen_at": "2026-10-25T00:00:00Z"
},
"with_rating": {
"description": "Same baseline plus the ClusterMAX medal.",
"model_sha256": "<sha256 of your with-rating model/spec file>",
"inputs": ["gpu_type", "allocation_size", "list_price_per_hour", "workload_class", "region", "rating"],
"training_providers": ["TRAIN_PROVIDER_A", "TRAIN_PROVIDER_B"],
"frozen_at": "2026-10-25T00:00:00Z"
},
"workload_sha256": "<sha256 of bench.yaml, the exact workload spec you ran>",
"validator_sha256": "<sha256 of validate.py, the exact acceptance validator you used>",
"customer_role": "ordinary_tenant",
"support": "standard",
"gates": {"accepted_min": 950, "p95_max_ms": 500, "cost_max_usd": 25.0},
"binding": {
"source": {"url": "https://...medal-table...", "sha256": "<sha256 of that page/image>", "retrieved_utc": "2026-11-02T00:00:00Z"},
"rubric": {"url": "https://...rubric...", "sha256": "<sha256 of that document>", "retrieved_utc": "2026-11-02T00:00:00Z"},
"bound_at": "2026-11-02T01:00:00Z",
"medals": {"ACME_CLOUD": "Gold", "ZETA_GPU": "Silver"}
}
}Run it, then self-check with submit_check.py the same as a hand-authored packet:
python scripts/csv_to_packet.py --csv jobs.csv --header plan-header.json \
--plan-out submissions/acme-cloud-2026-11/plan.json \
--binding-out submissions/acme-cloud-2026-11/binding.json \
--outcomes-out submissions/acme-cloud-2026-11/outcomes.json
python scripts/submit_check.py submissions/acme-cloud-2026-11