Karpathy's 4 principles for AI memory, implemented end-to-end
Karpathy's April 2026 post named the four constraints any serious personal AI memory system needs: Explicit, Yours, File over app, BYOAI. Here's each one translated into concrete implementation, with the pieces you actually install on day one.
- Time
- 2 hours initial setup + 15 min/day
- Cost
- $0
- Stack
- Obsidian (or any markdown editor)gitClaude Code / Codex / Gemini CLIiCloud / Dropbox / Syncthing
You’re stuck with
You've tried the 'memory' feature on three different AI apps. Each one locked, opaque, single-vendor. You restart a conversation and the AI doesn't know who you are. Your context is trapped inside one tool.
You end up with
A local markdown wiki that every AI agent you install. Claude, Codex, Gemini, whatever ships next, reads as its source of truth. Inspectable, portable, and compounding daily. A living artifact, not a feature flag.
The recipe
1. Lock the four principles as non-negotiable
Karpathy's four principles are the constraints. When you catch yourself about to take a shortcut, "I'll just let the agent remember in its own memory", check the shortcut against these four. Every shortcut loses at least one.
- Explicit. The memory is navigable. You can see exactly what the AI knows. The LLM writes, you read and verify. Nothing hidden. No opaque summary buffer.
- Yours. Your data sits on your local computer, in files you control. No vendor owns the memory layer. If the vendor disappears tomorrow, your memory does not.
- File over app. Markdown, images, flat files. Universal formats that every tool, CLI, and future agent can read. The same vault works in Obsidian, VS Code, grep, a new model released in 2027.
- BYOAI. Plug in whichever model you want. Swap Claude for Codex for Gemini without losing a single note. The wiki is the constant; the agent is the variable.
These four constraints sound abstract on day one. They earn their keep the first time you switch models, change editors, or lose trust in a vendor. Build the wiki so you're ready before any of that happens.
2. Design the folder skeleton before you write anything
The skeleton should have a fixed, small vocabulary. When you can't decide where a note goes, the skeleton is too loose.
your-vault/
soul.md # who you are, what you value
AGENTS.md # how agents should behave here
NOW.md # what matters this week
00-inbox/ # raw capture, no structure
01-daily/ # one file per day
YYYY-MM/
reviews/
templates/
02-people/ # one file per human who matters
03-opportunities/ # active leads, pipelines
04-projects/ # one folder per active project
05-backlog/ # someday / maybe
06-reference/ # curated, promoted, stable
learnings/
strategy/
tech/
07-archive/ # completed or inactive
The numbers matter. They force a rough priority order in file browsers, and they mean "refs always appears below projects" without any tooling. The skeleton is finished the day you stop creating new top-level folders.
3. Write the three operating documents at the root
These three files are the always-loaded context for every agent. Keep them short.
soul.md, identity, values, decision principles. Written in first person. Agents read this to understand who they're working with. Rewrite rarely.AGENTS.md, rules about how agents should behave in this vault. Which files they can write, which skills are available, what sessions look like.NOW.md, what matters this week. Updated weekly. Contains the North Star, today's top 3, and any "waiting on" items.
Opening any agent in the vault should load these three files first. If an agent doesn't know what NOW.md is, train the reflex: paste it at the start of every session until a hook or skill automates it.
Your operating documents are the handshake. Get them tight and everything downstream gets easier.
4. Decide the capture vs promotion rhythm
Every note lives in one of three states:
- Captured. Raw, fast, unstructured, no-schema. Daily notes and
00-inbox/hold this. You paste, scribble, and keep moving. - Promoted. A captured moment turned out to matter. You rewrote it as a learning, a decision log, or a reference doc. Lives in
06-reference/. - Archived. It mattered once, it doesn't now. Moved to
07-archive/rather than deleted, so it stays grep-able.
Capture is cheap, promotion is the discipline. Most people fail at promotion, they capture forever and the vault becomes a junk drawer. Add a weekly review slot that is specifically for promoting, not for creating new notes.
5. Give your first agent the full vault
When you launch Claude Code (or Codex, or Gemini CLI, or whatever) from the vault root, it sees every file. That's the point. You do not hand the agent "relevant slices", you hand it the vault and let it navigate.
One rule: the agent reads soul.md, AGENTS.md, NOW.md, and the current daily note on session start. Everything else is pulled on demand via grep and glob. Don't pre-stuff the context window.
If your agent can't see the vault from its working directory, move the agent, not the vault. The vault is the source of truth.
6. Add the second agent without breaking the first
This is where it gets interesting. BYOAI means different agents with different strengths reading and writing the same files.
The risk is collision: two agents editing the same file, or two agents writing to the same index without a deconflict rule. The fix is three explicit rules in AGENTS.md:
- Ownership. Name the one agent that owns each shared file. Example: "Claude Code owns all vault writes except daily-note Codex sections and
00-inbox/ingests." - Session files. Every agent writes a
memory/active-session-YYYY-MM-DD-N.mdfile on session start and updates it as it works. Other agents can read it to see what's in flight. - No shadow state. Nothing important lives in an agent-specific memory store. Everything important gets written to the vault, in markdown, where every other agent can read it.
Do this before you bring in the second agent. Retrofitting is painful.
7. Build the session protocol so agents don't collide
The session protocol is the smallest set of rules that lets many agents coordinate through the vault. At minimum, it has:
- Atomic session numbering. The next session number is
max(counter in NOW-style doc, highest N in memory/active-session files, highest logged session)+ 1. Compute it atomically. Prevents two tabs both claiming session 5. - Compaction-safe bridge. The active-session file persists when a context window compacts. On wake-up, the agent re-reads its active-session file instead of starting over.
- Doc drift check. Before logging the session as done, verify that operating documents reflect any state changes. Do the docs first, then the session log.
When you read the session protocol for the third time, it'll feel like overkill. Run it for two weeks and it becomes the thing you trust most.
Why markdown beats everything else here
Every AI tool speaks markdown. It's the pidgin of the agent era. A plain markdown vault can be read by the CLI, the grep toolchain, a new Claude release, a new model from a vendor that doesn't exist yet, your terminal, Obsidian, VS Code, a static-site generator, an AI crawler.
A vendor's memory feature is a rental. When the vendor shuts down the feature, changes its pricing, or re-scopes what "memory" means, your continuity goes with it. A file over app, you own.
The wiki is your compounding moat. Every session, every promoted note, every decision logged, all of it accrues to the same artifact that every future agent will read. This is the opposite of starting a new AI conversation and re-establishing who you are for the seventeenth time.
Steal this starter
Day-one folder skeleton (bash):
#!/usr/bin/env bash
# create-vault.sh <vault-name>
set -euo pipefail
root="${1:?usage: create-vault.sh <vault-name>}"
mkdir -p "$root"/{00-inbox,01-daily/templates,01-daily/reviews,02-people,03-opportunities,04-projects,05-backlog,06-reference/{learnings,strategy,tech},07-archive,memory}
cd "$root"
cat > soul.md <<'EOF'
# Who I Am
<write in first person: role, values, decision principles>
EOF
cat > AGENTS.md <<'EOF'
# Agent Rules
## Ownership
- Primary agent owns: …
- Secondary agent owns: …
## Session Protocol
- On start: read soul.md, AGENTS.md, NOW.md, today's daily note
- Bridge file: memory/active-session-YYYY-MM-DD-N.md
EOF
cat > NOW.md <<'EOF'
# NOW
## North Star
## This Week's Top 3
- [ ]
- [ ]
- [ ]
EOF
git init -q && git add -A && git commit -q -m "init vault skeleton"
echo "Vault ready at $root"
Daily note template (copy into 01-daily/templates/daily.md):
# {{date}}, {{dddd}}
#type/daily
**Claude Code Sessions: 0**
**Other Agent Sessions: 0**
## Intentions
- [ ]
## Top 3
- [ ]
- [ ]
- [ ]
## Captures
<!-- raw notes, links, quotes, half-thoughts -->
## Promoted (to review weekly)
<!-- anything worth moving to 06-reference/ -->
## Session Logs
<!-- each agent appends its own summary here -->
First-day checklist:
- Create the folder skeleton
- Write
soul.mdin 20 minutes, first draft only - Write
NOW.mdwith exactly one North Star and three Top 3 items - Write
AGENTS.mdwith one ownership rule and the session-start ritual - Run
git initand commit the skeleton - Launch your first agent in the vault and ask it to summarize
soul.md - Schedule a weekly review slot in your calendar, 30 min, Sunday
- Let it run for seven days before tuning anything
After seven days you'll know which folders you're not using, which captures you never promote, and which operating doc is too long. Cut first, add second. The wiki stays small by default and earns every new folder.
Get new workflows and breakdowns in your inbox.