# Retrospective study R1: do ClusterMAX medals track providers' own public incident records?

Status: **frozen before any incident data was collected** (git commit on branch
`agent/clustermax-standing-challenge`, 2026-09-23). The plan hash in `plan.json`
covers this file. Anything changed after collection is reported as a deviation.

This is a look back at public records, not the prospective customer-outcome test.
It asks a narrow question about one of ClusterMAX's ten criteria (reliability),
using the only outcome data that is public and free: each provider's own status page.

## Question

Among providers rated in a ClusterMAX release, do higher medals go with fewer
self-reported service incidents in the 180 days after that release?

## Inputs

* **Ratings.** ClusterMAX 2.0 (published 2025-11-06) is the primary release.
  ClusterMAX 1.0 (published 2025-03, date pinned in `ratings/`) is secondary.
  Medals are taken from the official published tables/images, with source URL and
  file SHA-256 recorded. Ordinal: Platinum 5, Gold 4, Silver 3, Bronze 2,
  Underperforming 1. "Unavailable"/unrated providers are excluded.
* **Outcomes.** Each provider's public status page incident history, captured with
  retrieval time, URL and raw-file SHA-256.
* **Window.** Primary: [2.0 publication date, +180 days). Secondary: [1.0 publication
  date, +180 days).

## Eligibility (decided before looking at counts)

A rated provider is included if its public status history is retrievable and
covers the entire window (the earliest retrievable record, or an explicit
"no incidents" archive month, predates the window start). Otherwise it is listed
as excluded with the reason. **Hot Aisle is excluded from the primary analysis**
because the author has a working relationship with it; its numbers are reported
separately and labelled.

## Outcome measures

* **Primary:** count of incidents whose provider-assigned impact is major or
  critical (or the provider's equivalent top two severities), started inside the window.
* **Secondary:** count of all incidents of any impact; sum of incident durations
  in hours (start to resolved).

Scheduled maintenance is excluded. Incidents are not filtered by component
(component naming differs too much across providers to filter without judgement);
this is a limitation, not a choice made after seeing results.

## Analysis

* Spearman rank correlation (average ranks for ties) between medal ordinal and
  each outcome measure, across included providers.
* Uncertainty: 5000 provider-level bootstrap resamples, seed 20260923, 95%
  percentile interval. Exact two-sided permutation p-value (or 20000 random
  permutations if the provider count makes exact enumeration impractical).
* Reading rule for the primary measure:
  * interval entirely below 0: **consistent** with medals tracking reported incidents;
  * interval entirely above 0: **inconsistent** (higher medals, more incidents);
  * otherwise: **inconclusive**.
* Minimum: fewer than 8 included providers means the result is reported as
  **insufficient** and no reading is given.

## Known limits, stated in advance

1. Status pages are self-reported. A provider that reports honestly and in detail
   looks worse than one that reports nothing. This transparency effect can reverse
   the sign of any correlation and cannot be removed with public data.
2. Status pages cover whole companies; ClusterMAX rates GPU cloud offerings.
3. Reliability is one of ClusterMAX's ten criteria. A null or inverse result here
   does not show the medals are wrong overall.
4. Retrospective and observational. Nothing here is causal or prospective.

Every excluded provider, every raw file and every deviation from this plan is published
alongside the result.