model selection

Part 2 of my Jev test, pre-registered before the first call: up to five similar past records added to every prompt. Opus went from 23% to 73% on one routing question, and neither model showed a clear effect on another. Per class results for Jev, Claude Opus 5.5 and memory alone.

A pre-registered test of TypeSafe's Jev against Claude Opus 5.5 and a local Qwen3-14B on 340 decisions with known answers: accuracy, calibration, cost and speed, per class.

A newer, higher-benchmarking model was available at the same price. I rolled my chief-of-staff agent fleet back to the older version anyway. Here is the evidence on both sides, including the part that disagrees with me, and why for agentic systems harness fit beats model tier.