95% at launch, 65% a month later

The number everyone quoted from Anthropic’s June post on internal analytics was 21 to 95. Without hand-written skills, Claude’s accuracy on analytics questions never got above 21%. With them, accuracy climbed above 95%, reaching near 99% in some domains. It’s a compelling figure, and a fair argument for writing the knowledge down.

The sentence I keep going back to is the next one:

we watched our offline accuracy drift from ~95% at launch to ~65% over a month before we treated this as an engineering problem

Nobody deleted a skill or edited one. The files that produced 95% were still sitting there, still reading as correct, yet a third of the accuracy vanished in four weeks.

I own the content quality of a curated knowledge layer: a set of hand-written entries an AI retrieves from when answering questions about running a membership business. Every quality gate I have runs before an entry goes live: deduplication against the live corpus, tone and voice checks, taxonomy validation, mechanism-based claim grounding, and human review. All of it measures an entry on the day it lands. My coverage report describes the corpus on the morning I measured it, but says nothing about the corpus a month later. I don’t know yet whether our curve mirrors theirs, which is most of the problem: there is no second measurement to compare against the first.

The fix they landed does not port

Anthropic’s answer was to put knowledge next to the code it describes. Skill documentation lives alongside transformation code, and roughly 90% of their data-model PRs now include a skill change in the same diff. Change the data model without updating the docs, and someone catches it in code review rather than six weeks later.

That works because the subject of the knowledge lives inside the repository: their columns, their pipeline. Every event that could invalidate a skill shows up as a pull request a human must approve.

Curated knowledge about the outside world has no such repository. An entry on recovering failed renewals goes stale when a payment processor changes its retry schedule, when a plugin ships a setting that renders half the advice obsolete, or when site owners quietly shift their practices. None of that arrives as a pull request I review. Colocation handles changes that announce themselves, which is not the case I have.

Two ways knowledge rots

The decay I can actually point at did not happen in the content; it happened one layer up, in the rules governing it.

In August, an instruction file for my authoring work told me to attribute advice to a cohort of a stated size. That requirement had been deliberately stripped out of every live entry weeks earlier. The repo’s authoring reference reflected the unquantified version, as did every live entry. The instruction file was the only place the old rule lingered, and following it would have reintroduced removed phrasing back into live advice. In that same pass, the same file told me a set of platform tags was forbidden, citing an open ticket that had shipped weeks prior. Neither rule was wrong when written.

The second example comes from my own system, making it harder to externalize blame. One of my knowledge bases carried a rule prohibiting external research. Upon inspection, eight of its fifteen sources were external articles filed in June and July, before the rule was even written in August. A single instruction file had drifted, and that drift was being retroactively applied to a corpus that never followed it. Enforcing the rule would have discarded valid material. I replaced the rule instead, adding a field on each admitted source that names the internal question it answers. This turns “this earns its place” into an auditable claim rather than a one-time judgment call.

Both failure modes behave identically: nothing breaks when an agent reads a stale rule. There is no thrown exception and no failing test, just output that looks indistinguishable from every other output. I wrote about a flag that did not exist a few weeks ago and took it for one bad line in one file. The class is wider than that. Retrieval quality sits in it too.

What replaces colocation

Without a diff to react to, whatever prompts a review must trigger independently. Three viable candidates emerge.

Measure on a recurring schedule rather than at static milestones. A launch coverage report confirms the corpus was solid on a Tuesday in August. If accuracy can drop thirty points in a month with zero edits, measurement must run as frequently as the underlying domain shifts. Monthly is a starting guess, and a far better one than measuring once. Implement a golden evaluation set that can fail a build. Define expected queries and outputs, running tests on every merge and automated timer. Because retrieval degradation carries no runtime exceptions, it will ship silently unless explicitly asserted against. Assign an owner and a next-review date to every entry. Anthropic tied reviews to diffs because diffs existed. Without them, an assigned person and an explicit expiration date are the only honest substitutes. If no one will put their name on an entry, delete it.

None of this infrastructure is expensive. It is simply work without a launch headline attached, which means it continuously loses priority to writing the next fifty entries, until the month after launch produces a number like 65%.

Most readers took Anthropic’s post as a case for writing knowledge down. It is also the only published metric showing what the month after launch costs if you stop there, and a thirty-point drop is far steeper than I would have guessed before reading it.

Source: How Anthropic enables self-service data analytics with Claude, June 2026.