Coherence training for forecasts · Kelvin, MATS

Now: 32B baseline → 0.6B

Qwen3-32B first: same family and training cutoff as the 0.6B student.

Target

Test set: 1,066 pairs of real Manifold questions (May–Dec 2025), 12 logical forms each. A model must beat the 12-number base-rate table.

1Qwen3-32B zero-shot: done, 0.616 (below). Does not beat the table.
2Qwen3-0.6B, 8 arms with fixed prompts (consistency λ 0.3/1/3 and two controls): running.
3Check the gain is coherence, not shrinkage to the prior.

Baseline result

Done

Qwen3-32B, zero-shot, 23 September 2026. 1,066 test pairs, 12 forms each, scored once. Prompt chosen on validation (6 variants). No unreadable answers.

Predictor	Log loss, observed form	Log loss, all 12 forms	Accuracy	ECE
Constant 0.5	0.693	0.693
Form-only prior	0.607	0.606
Base-rate table	0.578	0.570
Qwen3-32B	0.616	0.611	0.703	0.062
Qwen3-32B, calibrated on validation	0.597	0.595	0.703	0.037

32B does not beat the base-rate table. Weakest on negations: P and not-P differ from summing to 1 by more than 0.2 in 35% of pairs.

Below the diagonal at low probabilities: the 32B model is too confident that things will not happen. ECE 0.062.

Frontier traces remember, not forecast

On pre-2025 questions a frontier model beats the market price: it recalls outcomes. So its traces are not used for training, and all evaluation is post-cutoff.

Now: 32B baseline → 0.6B

Target

Next

Baseline result

Frontier traces remember, not forecast