LLM Wiki — Compiling Knowledge Instead of Re-reading It
Contents
Notes on LLM Wiki, the pattern Karpathy posted in April: let an LLM build and maintain a wiki for you.
Purpose, structure, real usage, and efficiency, in that order. The efficiency part is the interesting one — the short version is that the original contains no numbers at all.
Purpose
The original exists as an X post and a gist. The gist has the full text, so quotes come from there.
The problem statement is one sentence:
This works, but the LLM is rediscovering knowledge from scratch on every question. There’s no accumulation. … NotebookLM, ChatGPT file uploads, and most RAG systems work this way.
Load 20 papers and ask a question, and the model works through them as if seeing them for the first time. Yesterday’s reasoning is gone. Same sources, same thinking, over and over.
Hence the proposal:
Instead of just retrieving from raw documents at query time, the LLM incrementally builds and maintains a persistent wiki — a structured, interlinked collection of markdown files that sits between you and the raw sources.
The line the original itself flags as the point:
This is the key difference: the wiki is a persistent, compounding artifact.
Don’t reconstruct knowledge at query time. Compile it at ingest time and keep it.
Worth noting how the gist describes itself:
This document is intentionally abstract. It describes the idea, not a specific implementation.
It is an idea file meant to be pasted into your own agent, not a tool.
The core: three layers, three operations
| Layer | From the original | Who writes it |
|---|---|---|
| raw sources | “immutable — the LLM reads from them but never modifies them. This is your source of truth.” | nobody |
| the wiki | “The LLM owns this layer entirely. … You read it; the LLM writes it.” | the LLM |
| the schema | “a document (e.g. CLAUDE.md … or AGENTS.md)” | you |
The framing is nice:
Obsidian is the IDE; the LLM is the programmer; the wiki is the codebase.
Three operations:
Ingest — read a new source and fold it in. The original notes that “A single source might touch 10-15 wiki pages.” One document does not become one page; it scatters across many.
Query — answer from the wiki. Good answers get filed back as new pages.
Lint — check for contradictions, stale claims, orphan pages, missing cross-references. This is the part that keeps the wiki alive.
Plus index.md as a catalogue and log.md as an append-only timeline.
Why human wikis die
Humans abandon wikis because the maintenance burden grows faster than the value. LLMs don’t get bored… The wiki stays maintained because the cost of maintenance is near zero.
“Near zero” is a claim, not a measurement. More on that below.
Reported effects
This is where the sourcing matters, so it is split by type.
The one controlled comparison
arXiv 2605.18490 (Theodore O. Cochran, 2026-05-18). Preregistered, blind LLM judging.
- Scale: 24 papers, 13 questions, same answer-generating model on both sides
- Wiki won: cross-paper synthesis, and citations that actually supported the claims made
- RAG won: single-fact lookup
- Cost, from the abstract: “the wiki used far more query tokens than RAG, so it could not recover any upfront build cost through cheaper queries”
The conclusion was no architecture was best on all three — evidence organisation, citation
support, and cost.
The common claim that the wiki saves query tokens is contradicted here. Though 24 papers and 13 questions is a small study.
Self-reported
Not measurements, but first-hand accounts.
One HN user fed in three books (~155K words across 68 source files) and got 210 concept pages
with 4,597 cross-references. Output exceeded input, which they summarised as
It's not compression — it's synthesis. They also reported the wiki surfacing an unstated
contradiction between two of the books.
The more useful part of that account: source granularity decided the outcome. One file per
book produced junk. Splitting by chapter changed the output categorically. Same model, same prompts — the only variable was source granularity.
Going the other way, someone who used it for a month and accumulated ~760 pages judged the time spent maintaining it to be about equal to the time saved — a wash. No automatic updates, and manual fact-checking still required.
Numbers with nothing behind them
- A major implementation’s README claims
recall 58.2% → 71.4%with no methodology or dataset — and it is not LLM Wiki vs RAG, it is that product’s own vector search on/off - One service advertises
10x better performance; asked on HN for the basis, no answer came - One cost estimate exists, and its author labels it
back-of-envelope. Another line from the same piece sums the situation up:nobody is publishing the real numbers
How people use it
The cases found differ a lot in character.
Personal archive into a knowledge graph — treat your own blog posts as raw sources and let the
LLM maintain an Obsidian vault of source notes, concepts, projects and maps. That author’s
conclusion is worth repeating: wiki quality depends as much on the quality of your AGENTS.md
as on the model.
Regulatory and legal review — the most original of the lot. The layers get extended to
raw/ → sources/ → drafts/ → review-drafts/ → context/, and only human-approved context/
may be cited as grounds by the AI. Past review cases are treated as hints rather than grounds,
because the service structure, contracts and statute versions at the time may all differ.
They also refuse to copy the existing internal repository into the wiki — that would create two sources of truth. And lint gets redefined as a safety gate that decides whether a document may be used as grounds at all. Their summary: the point of LLM Wiki is not automating ingest, it is designing who gets to decide what counts as grounds afterwards.
Codebases and research notes — the most common shape: drop in papers or repos, get concept pages.
When not to use it
- Hundreds of thousands of documents → conventional search or a vector DB
- Real-time freshness → API plus RAG
- Exact clause matching → full-text search
- Immutable originals required (legal, audit) → a DMS
The original concedes the scale limit too: the index.md approach works well up to roughly 100
sources and a few hundred pages, above which it tells you to add a search engine. Karpathy
himself says to reintroduce retrieval past a certain size. “Replaces RAG” is not a framing the
primary source supports.
Efficiency
| Claim | Evidence |
|---|---|
| Maintenance cost near zero | Asserted in the original. Not measured |
| Saves query tokens | Contradicted by the only controlled experiment |
| recall 58.2% → 71.4% | Own feature on/off. No methodology |
| 10x faster | No basis given |
| Strong at cross-source synthesis | Confirmed in arXiv 2605.18490 |
| Citations support claims well | Confirmed in the same study |
Adopting this for efficiency is likely to disappoint. The verified gains are synthesis and citation reliability, not cost. Cost may well go up.
The intuition — pay once at ingest, query cheaply forever — is appealing. The only experiment that tested it says otherwise.
Criticism
The HN thread has sharp objections.
“This is just RAG” — This is just RAG. Yes, it's not using a vector database - but it's building an index file of semantic connections… this is RAG. The counterargument is the write
loop: in vanilla RAG the corpus is static; here the LLM authors and maintains it.
Outsourcing your thinking — the most common objection.
Most of the value of writing docs or a wiki is not in the final artifacts, it's that the process of writing docs updates your own mental models.
The most honest account in the thread:
I miss thinking harder… But the wiki workflow is just too addictive to stop.
Second-order drift — reading the wiki instead of the sources accumulates subtle errors, and a hallucination baked into the wiki can produce wrong answers even where the model would have answered correctly if asked directly.
Review burden — I'd rather have it source the original document everytime, then an LLM-generated wiki which I most likely wouldn't have the time to fact-check and review myself.
Lint scaling — finding contradictions across N files is an N×N comparison.
A collapse point — past some size the agent can no longer keep the wiki current and the human can no longer grasp it.
Implementations
| Repo | Stars | Language | License |
|---|---|---|---|
| nashsu/llm_wiki | 15,404 | TypeScript | GPL-3.0 |
| AgriciDaniel/claude-obsidian | 9,993 | Python | — |
| Astro-Han/karpathy-llm-wiki | 1,647 | Python | — |
| lucasastorian/llmwiki | 1,425 | Python | — |
Star counts as of 2026-07-28.
The largest, nashsu/llm_wiki, is a Tauri v2 + React 19 desktop app with prebuilt binaries for
macOS, Windows and Linux. It supports OpenAI, Anthropic, Google and Ollama, runs locally, and
ships an MCP server. It keeps the three layers, the ingest/query/lint loop, index.md,
[[wikilink]] and Obsidian vault compatibility, and adds a knowledge graph, community detection
and a web clipper.
GitHub shows the license as NOASSERTION, but the LICENSE file is the standard GPLv3 text — a
copyright line at the top just defeats the automatic detector.
Notably, most of the real usage accounts do not use any of these apps. They read the gist and make the folders themselves.
Wrapping up
The idea is solid: compile instead of re-reading, and hand the maintenance humans never manage to an LLM.
The original points at Memex, from 1945:
The part he couldn't solve was who does the maintenance. The LLM handles that.
But efficiency is a bad reason to adopt it. The verified benefits are synthesis and citation reliability; the one experiment that measured cost found it worse. The month-long user who called it a wash fits the same picture.
If you do adopt it: keep the scale small (under ~100 sources), split sources finely, and don’t treat the wiki as if it were the source. The regulatory case is the better model — keep human control over what may be cited as grounds.
An idea with no numbers, surrounded by numbers. Worth revisiting in a few months.
References
- Karpathy’s gist (full text) — https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f
- Original X post — https://x.com/karpathy/status/2039805659525644595
- HN discussion — https://news.ycombinator.com/item?id=47640875
- Preregistered comparison — https://arxiv.org/abs/2605.18490
- nashsu/llm_wiki — https://github.com/nashsu/llm_wiki