Public “has it been nerfed?” trackers are necessary but insufficient.
Public “has it been nerfed?” trackers are necessary but insufficient. Provider behavior now moves faster than many app release cycles, as the Opus 5.5 launch and live nerf benchmarks make clear, so a screenshot of a benchmark is only observability, not control. Treat model updates like releases: run provider canaries, keep pinned model cohorts for critical flows, and codify rollback and downgrade rules into your deployment playbooks. If you don’t, drift becomes an operational incident, not a research footnote.
GitHub - ninjahawk/livenerf: Benchmark for tracking model capability after release.
#AI #SDLC
All posts