A skill told me about a flag that didn't exist

For most of July, one of my own instruction files told me to run a command that could not work.

The file was a skill: a folder of instructions that loads into an AI session and tells the assistant how to do one particular job. Mine covered a recurring authoring task. Near the top it said to validate new entries with a dry run, passing --include-drafts so unpublished files got picked up too.

There is no --include-drafts. The script it pointed at has draft inclusion hardcoded inside the function that loads the files. The flag was never implemented, never deprecated, never real. Run the documented command and the script exits on an unrecognized argument.

That line sat in the file for weeks. It cost time across three pull requests before anyone traced the problem back to the instructions.

Why it survived weeks

A wrong flag in an instruction file does not announce itself as a documentation problem. It announces itself as a broken machine.

The session runs the documented command, the command fails, and both of us reach for the same explanation: something is wrong with the local setup. Wrong virtual environment. A service that isn’t running. A checkout that drifted. This is a reasonable first guess, and in my case it was actively encouraged by the fact that two of those things were also true. The script needs a local Postgres up even for a dry run, and it silently misbehaves under the wrong Python virtual environment. Neither fact was written down anywhere either.

So the search goes to the environment, finds two real problems there, fixes them, and the documented command still fails. At which point you are three layers deep in your own tooling and the file that started it has never once been treated as a suspect.

Code fails once. Instructions fail every time.

Put a wrong constant in a function and it breaks at a specific line, with a stack trace pointing at the line, in front of whoever ran it. It gets fixed because the failure and the cause arrive together.

An instruction file doesn’t execute. It gets read, believed, and acted on by a fresh session that has no memory of the last time this went wrong. Each session pays the cost again. None of them can see the others paying it. There is no shared surface where the failures accumulate into something that looks like a bug worth chasing.

Worse, the assistant routes around it. It tries a variant, drops the flag, gets a plausible-looking result and moves on. The work still ships. Nobody files anything, because from inside any single session nothing went badly enough to report. The instruction survives precisely because it was survivable.

This is the property that makes instruction files the highest-leverage place in an AI setup for a wrong fact to live. Not the highest-risk, since a wrong instruction rarely destroys anything. The highest-leverage, because one line propagates into every future session for free.

The copy you edit is not the copy that loads

When I did find it, the obvious fix quietly did nothing.

Skills can install through a plugin, and the copy sitting on disk under the plugin cache path is a cache. Editing it is read-only in effect. The file you are looking at changes, the file that actually loads does not, and nothing errors to tell you which one you just edited. The next session reads the original and repeats the same wrong instruction back to you, which is a genuinely disorienting few minutes.

The real edit has to go through the tool that writes the skill, with overwrite set, so the authoritative copy changes.

The general form of this, for anyone running a similar setup: when your instructions are distributed through a package manager or a plugin system, find out which copy is authoritative before you edit anything. There will be more than one copy, and the writable-looking one is not always the one being read.

What I changed

A skill is an instruction file, so it now follows the rule my other instruction files already followed. Any change means a dated edit inside the file, plus one line in a decisions log recording what changed and why.

The log line is the part that earns its keep. Without a date on the edit, you cannot tell whether a rule was correct when it was written and went stale, or was wrong from the first day. Those need different fixes. Stale rules mean your review cadence is too slow. Wrong-from-the-start rules mean whoever wrote it was describing an intention rather than a behavior, which is a much more interesting problem and shows up again elsewhere.

The same pass wrote down the two environment facts that had been camouflaging the original bug, along with several other things I had learned by doing the work and never recorded.

The gate I’d actually want

Instruction files describe an environment, and environments move. Prose can’t be tested. A command can.

If a file tells an agent to run something, that command should be runnable in CI against the current state of the repository, or the file should not state it. The cheap version is a script that walks every fenced command in your instruction files, runs each one with a help flag or in a no-op mode, and fails the build on an unrecognized argument. It catches nothing about bad judgment or stale reasoning. It would have caught this on the day the flag was written.

The rest of that file was good. That’s the part worth sitting with. One wrong line in an otherwise reliable instruction file is more expensive than a file full of nonsense, because a bad file gets rewritten and a good file gets trusted.