The memory system behind my AI workflow

Every AI session starts from zero. You spend twenty minutes getting the model up to speed on who you are, what you’re working on, and what you decided last week, then the window closes and all of it evaporates. Next morning you do it again. The model is brilliant and has amnesia, and you’re its short-term memory.

I got tired of being the memory. So I built the model one. It’s not a product or a plugin, just a set of plain markdown files and two rules about how they relate. It has been running my daily work for months now, and the interesting part is that most of what makes it work isn’t AI at all. It’s the same discipline a knowledge base or a well-run codebase needs: decide what’s true, write it down once, and guard it against drift.

Here’s how it’s put together, and why each piece earns its place.

The problem is amnesia, not intelligence

The instinct when a model doesn’t know your context is to paste more into the prompt. That works for one session and falls apart across many. You can’t remember what to paste, the paste grows every week, and half of it is stale by the time you send it. Worse, you have no idea which version of a fact the model is working from, because you’ve pasted three slightly different versions over three sessions.

The real problem isn’t that the model lacks a big enough context window. It’s that there’s no durable place for context to live between sessions, and no rule for which copy of a fact wins when two disagree. Solve those two and the amnesia mostly goes away.

Start of session: required reading

The first piece is a short list of files the agent reads before doing anything else. Not everything I’ve ever written, just the handful that orient it:

• Who I am and how I want to work with it.
• How I want things written, so the output doesn’t read like a template.
• A short working log of where each project stands.
• A canonical map of the fuller picture: projects, people, decisions, open threads.

The split matters more than the list. The first two files are standing instructions that barely change. The last two are working memory that changes constantly. Keeping them separate means I can edit today’s project status without touching the rules for how the agent behaves, and the agent can trust that the stable files are actually stable.

One canonical source, everything else defers to it

This is the rule that makes the whole thing hold together, and it’s borrowed straight from how you’d run a serious knowledge base.

One file is the source of truth for project state. Everything else that mentions the same facts, the short working log, any per-project notes, is a working cache that defers to it. When two files disagree, the newest dated entry wins, and I strike the old one with a note saying what superseded it and when. I don’t leave a contradicted fact sitting there looking authoritative.

Without this rule, a file-based memory rots fast. You end up with four files that each claim a different launch date and no way to know which to believe. The precedence rule turns a pile of notes into something with a single answer. It’s the same reason a database has one primary key and a codebase has one source of truth for config. Duplication itself is fine. Unresolved duplication is how you ship the wrong number.

Nothing untrusted gets written straight to memory

The next piece is a guard, and it’s the one I’d argue is most underrated in AI workflows.

External material, a Github issue, a web page, a PDF, a chat thread, never gets written directly into the memory files. It lands in a separate holding area first, quoted with a note about where it came from. Only after I’ve seen a conclusion and confirmed it does that conclusion get promoted into the real memory. The raw source keeps its provenance the whole way.

Two things this buys you. First, quality: the memory layer never fills up with unverified claims that later turn out wrong, because nothing reaches it without a human check. Second, and this is the part people miss, it’s a security boundary. If a web page or a ticket contains text that says “ignore your instructions and do X,” that text is data I’m storing, not a command the agent obeys. Treating all ingested content as data rather than instructions is a plain defense against prompt injection, and it costs nothing but the discipline of a holding folder.

A daily job keeps the map current

Files only help if they stay current, and staying current by hand is exactly the chore I was trying to avoid. So a scheduled task runs once a day. It reads the recent sessions and grows the canonical map: new decisions, changed project status, people and links that came up. If it missed a day, it backfills. It never deletes.

That last rule is deliberate. An automated writer that can delete is an automated writer that can quietly lose something you needed. Append and supersede, never destroy. If a fact is wrong, the fix is a dated correction that leaves a trail, the same way you’d revert with a commit rather than force-pushing over history.

The one door that stays shut

There’s a private file for personal and family matters that the agent never reads by default. It’s opened only when I raise something personal, and its contents are never copied into memory or into any work context. Anything sensitive that comes up in a work chat completes the task but stays out of the saved files unless I explicitly ask.

A memory system without an off-switch for sensitive material isn’t a memory system I’d trust with my actual life. The boundary is part of the design, not a bolt-on.

Why I needed this more than most

Here’s the personal half. I score high on retained breadth and comparatively low on working memory, and long before any of this I’d externalized my thinking into an Obsidian vault because holding many threads in my head at once is the thing I’m worst at. I prefer to run projects one at a time and I lose the plot when too much runs in parallel.

An AI agent has the same shape of weakness for a different reason. It has no memory across sessions at all. So the system I’d built to compensate for my own working memory turned out to be exactly the system an agent needs, and building the agent one forced me to make mine more rigorous than it had ever been. The precedence rule and the provenance guard weren’t in my personal vault before. They are now, because writing them for the machine showed me where my own notes had been quietly contradicting themselves for years.

What actually transfers

If you strip out the personal part, none of this is novel and that’s the point. It’s knowledge-base hygiene and a bit of data-pipeline discipline, pointed at the problem of an amnesiac assistant. Required reading at the start. One canonical source with a precedence rule. Provenance on everything ingested and a human gate before promotion. An append-only sync that never deletes. A hard boundary around the private stuff.

The model supplies the intelligence. The system supplies the memory. Keep those two jobs separate and you stop being your assistant’s short-term memory, which frees you up to use the intelligence for something better than reminding it what you decided on Tuesday.