Methodology

Index version 0.1.0 · machine-readable at /api/v1/methodology · full spec in the public docs (docs/methodology)

IRT-based scoring (2PL)

Every item carries an estimated difficulty and discrimination. Items that fail to separate strong from weak models are down-weighted and auto-flagged for retirement. Ability (θ) is estimated with a 2-parameter logistic model — never raw accuracy.

Dynamic item generation

Items are instantiated from versioned templates with value substitution and paraphrase perturbation, fresh for every scored batch. A fixed anchor subset (≤20%) is kept for longitudinal comparability; the fixed-vs-perturbed accuracy gap is published as contamination_delta.

Pairwise Bradley-Terry layer

Open-ended domains (writing, safety refusal quality) are scored by LLM-judged pairwise duels feeding a Bradley-Terry model, with ≥2 judges from different providers, position swapping, and judge-agreement reporting. A model never judges its own duels.

Consistency

The same item is asked k times / in k paraphrases; answer stability is a first-class metric. A model that flips answers is less trustworthy at equal accuracy.

Calibration

Models must report confidence with every answer; Brier score / expected calibration error per domain becomes the calibration sub-metric.

Efficiency frontier

Score-per-dollar and score-per-second are published as a Pareto frontier from OpenRouter pricing and measured latency — never blended into a single number.

Weights (v0.1.0)

Domain weights (Global Index)

  • code0.125
  • math0.125
  • reasoning0.125
  • writing0.125
  • knowledge0.125
  • multilingual0.125
  • instruction following0.125
  • safety refusal quality0.125

Sub-metric weights (domain composite)

  • accuracy_irt0.55
  • consistency0.15
  • calibration0.15
  • contamination_resistance0.15

Latency and cost are intentionally excluded — they live on the efficiency frontier.

IRT hyperparameters

{
  "model": "2PL",
  "priors": {
    "theta": {
      "dist": "normal",
      "mean": 0,
      "sd": 1
    },
    "difficulty_b": {
      "dist": "normal",
      "mean": 0,
      "sd": 1.5
    },
    "log_discrimination_a": {
      "dist": "normal",
      "mean": 0,
      "sd": 0.5
    }
  },
  "maxIterations": 500,
  "tolerance": 0.000001,
  "minDiscrimination": 0.3,
  "maxAbsDifficultyLogits": 3,
  "maxAnchorFraction": 0.2
}

Every methodology change bumps the index version (semver) with a public changelog entry. Item templates are public; instantiated items and answer keys stay server-side.