Retrieval

Reading files by hand stops scaling somewhere past a few hundred notes. The retrieval layer gives every compiled vault a local hybrid search index so an agent can ask a question and get the right chunks back — while the answer to "what can this search see?" stays exactly the same as everywhere else in the system: only what the person may read.

The boundary comes for free

The index for a person is built only from their compiled vault — never from master. The compiler already removed everything they may not read, so the index physically cannot contain it. There is no query-time permission filter to get wrong: retrieval inherits the structural boundary.

Rendering diagram...

This is why there is no "master index filtered per query" option: that would move enforcement from the compiler (structural) to a filter (a bug away from a leak). A randomized leak property test proves no query for a person ever returns a chunk outside their readable spaces.

What the index leaves out

Two things in a compiled vault are never indexed. The generated files — the protocol, Map.md, Shares.md — because they are regenerated state describing the vault rather than knowledge in it. And Sessions/, the processed archive.

That second exclusion is a judgement, so it is worth stating plainly. Sessions/ holds one summary per episode: what was decided, and what passed the admission tests. That is a narrative restatement of what the same routing pass already filed elsewhere — the facts went to Memory.md, the actions to the tracker, the decisions to the shared space. Indexed, one episode's content is in the index twice, and the two copies double-vote in rank fusion. That is the same failure doctor's dup-near check exists to name; here it is structural rather than accidental, so it is excluded by construction instead of reported.

Excluded, those files stay on disk, stay readable, and stay linkable by path or by Obsidian wikilink. What they stop being is search results, backlink sources, and nodes in the typed-relation graph. So a wikilink into Sessions/ — a fact line's [source:: [[the note]]], for instance — records as unresolved, exactly as any other target the index does not hold. The citation still tells a reader which episode established the claim; it just will not resolve to a hit.

How a query is answered

Search fuses three independent legs by reciprocal-rank fusion (RRF), and needs no score calibration between them:

Keyword — SQLite FTS5 (BM25) over chunk text, headings, and the name of the space each chunk sits in, so Acme finds the notes under Clients/Acme/ even when none of them says "Acme". The shared space's name is left out: it is on every shared note, and is usually an everyday word like "company".
Vector — semantic nearest-neighbours via sqlite-vec.
Graph — personalized PageRank over the note graph: wikilinks, fact co-mentions, and typed relations, seeded by entity names and aliases matched in the query, the top text hits, and optionally a center note (--center, or the center argument of the brain_search tool — typically the note you're working from).

Reciprocal-rank fusion means a chunk multiple legs surface outranks one only a single leg found. Unlike the two text legs, the graph leg can introduce results of its own that never textually match; such hits are attributed to the graph source. Notes are chunked heading-aware (each chunk carries its Heading > Subheading path), code fences are never split, and results are deduped to at most two chunks per file. If no embedding provider is configured — or sqlite-vec can't load — search degrades to keyword + graph with a warning, never an empty result.

An unknown center note is dropped with a warning; the graph leg still runs from the query's other seeds — entity names, aliases, and top text hits.

sqlite-vec loads as a SQLite extension, so it needs a Python whose sqlite3 was compiled with loadable-extension support. Homebrew, pyenv, uv-managed, and most Linux distro Pythons qualify; the python.org macOS installer ships it disabled. Verify with python -c "import sqlite3; print(hasattr(sqlite3.Connection, 'enable_load_extension'))" — False means the vector leg is unavailable and search runs keyword-only.

Typed relations

Notes can declare how they relate, right in their frontmatter — five fixed keys, each holding wikilinks:

123456---
title: PPR Retrieval
up: [[Retrieval]]
same: [[HippoRAG notes]]
next: [[Eval harness]]
---

up and down express hierarchy, same marks peers, prev and next chain sequences. Declare one direction; the mirror is derived automatically (a note you point up at knows you as down). Three more edge kinds are mined from structure you already have: a note whose folder holds an index note (like Clients/Acme/Acme.md) gets up to it, dated notes in the same folder chain prev/next, and pages sharing an entity: type become same siblings.

In search, a declared relation outweighs a casual mention (an explicit typed pair counts 3× a lone wikilink; mined edges count half a wikilink). Agents can also walk the structure directly — the brain_graph MCP tool and brain graph CLI command return a note's neighborhood with every edge labeled by relation and by why it exists (explicit, inverse, folder, date, entity).

The boundary comes for free here too: relation targets are ordinary wikilinks, so the compiler stubs any target a person may not read — in their vault that edge simply never exists. brain doctor flags notes with no connections at all (unlinked-notes): they're invisible to the graph leg until someone links them.

Duplicates distort retrieval from the other direction: two notes carrying the same content vote twice in the rank fusion, and their backlinks and facts split across two pages, so neither ranks as well as the one merged page would. brain doctor flags identical and near-duplicate notes a common reader sees (dup-exact, dup-near — near-duplicates are grouped within a space and paired across spaces; the near check combines text-shingle overlap with cosine over already-indexed embeddings, read from the cache brain cycle writes, <master>/_meta/cache/embeddings.db, only), plus title-stem collisions (stem-collision), since a bare [[wikilink]] resolves by stem and quietly picks whichever file sorts first. The fix is the normal editorial path: fold one page into the other with a mode: patch promotion. See the CLI reference for severities and exemptions.

The `.brain/` contract

The index lives at <vault>/.brain/index.db. It is machine-local state, on the same footing as .git:

Never committed. The compiler emits a .gitignore (.brain/, .obsidian/), so it never enters the vault's git history.
Rebuilt where it's used. Employees run brain index after they pull; hosted agents get theirs from brain cycle --index server-side.
Preserved across recompiles. A recompile swaps the vault contents but carries .brain/ across, exactly like .git.

Upgrading across a schema change

The index carries a schema version in SQLite's PRAGMA user_version. Upgrading brainkit needs no migration step from you — but the two directions behave differently, and the asymmetry is the point:

Index older than the code → rebuilt automatically. brain index notices the older version and forces a full rebuild rather than an incremental one. It has to: a migration adds empty tables, and the incremental path skips unchanged files, so those tables would stay empty for every file that hadn't moved — a silently half-populated index that answers queries with less than it should.
Index newer than the code → refuses to open, naming the fix:
```
12index at <path> was built by a newer brainkit
(schema 6 > 5); rebuild with: brain index --full
```
This is the fail-closed direction. A newer schema may hold columns this code does not know to read, and answering a query from a file it half-understands is worse than not answering. It matters in the real deployment shape, where the brain box and someone's laptop can sit on different versions for a while.

Because the index is derived state that is never backed up, both paths cost only the rebuild. Nothing is lost, so the safe move after any upgrade is simply brain index — or nothing at all, since brain cycle --index gets there on its own schedule.

Incremental by design

The compiler does a full rebuild every run, so the indexer diffs the manifest's per-file sha256 against what the index already holds: unchanged files are skipped, and a changed file only re-embeds chunks whose content hash misses the embedding cache. The cache is keyed by (chunk_sha, model) and can be shared across people, so a chunk two vaults have in common embeds exactly once.

Scale tiers

The vector store sits behind a three-method seam (add/delete/knn), so the backend swaps without touching chunking, FTS, or fusion. The unit of scale is one person's vault, not the whole company.

Corpus (per vault)	Backend	Trade-off
≤ 50K pages	sqlite-vec brute-force (default)	Exact KNN, zero infra, single file. Storage grows linearly.
50K–500K pages	+ int8 quantization	4× smaller, near-exact, same file — still linear scan.
millions	LanceDB IVF-PQ or sqlite-vec DiskANN	Sublinear ANN; a heavier dependency or a pre-release.

Ship tier 1; reach for the others only when a real vault crosses the line.

Facts and time

Search finds relevant text; some questions instead need a queryable state — "is this client still active", "who's the current contact" — answered as of a date rather than by relevance. Those are written as fact lines: a bullet with a [from::] date and a [source::] back to the note that established it, e.g.

123- Acme is a client [from:: 2025-03] [source:: [[2025-03-12 Kickoff call]]]
- Acme's plan is Enterprise [from:: 2025-03] [until:: 2026-01] [source:: [[2025-03-12 Kickoff call]]]
- Acme's plan is Growth [from:: 2026-01] [source:: [[2026-01-08 Downgrade]]]

A fact is never edited or deleted once superseded — the old line gets an [until::] and a new line is added, so the full history stays on the record.

Forgetting that [until::] is the failure mode this grammar makes detectable: the superseded line and its successor are both "true now," and a query returns a coin flip. brain doctor flags open facts that assign two values to the same attribute of one entity (fact-conflict) and identical open fact lines from a double-landed ingest (fact-dup) — close the superseded line with [until::], or delete the duplicate via write-back.

Three questions read that history differently:

Current (default) — facts true today.
As-of a date (--as-of) — facts true on that date, using what the vault knows now. Answers "what was true then," with the benefit of hindsight.
Believed-on a date (--believed-on) — what the vault would have told you if asked on that date, replayed from the git history at that point in time. Answers "what did we think then," including facts since corrected.

That last distinction is why transaction time is git: valid time (when a fact was true) lives in the [from::]/[until::] dates in the text, while transaction time (when the vault learned it) is just which commit you read the file at. No separate audit log is needed — git already keeps one.

This is a different job from the [source](URL), as of YYYY-MM citations used elsewhere: citations back narrative prose ("Acme relaunched their site"), while fact lines are for state a query can ask about later. Use a citation for a one-off observation; use a fact line for anything future questions will need to look up.

Query facts with brain facts, or the brain_facts MCP tool. The dashboard shows them too: a Facts tab in the live dashboard, and a Facts section in the static snapshot — each with a date control for what was true on a given day.

Using it from an agent

Register the MCP server once and any MCP client (Claude Code, Cursor, Codex, Claude Desktop) can search the vault as a tool:

1claude mcp add brain -- brain mcp --vault ~/brain

It exposes six read-only tools: brain_search (hybrid search with keyword, vector, and graph legs; optional center note as a strong seed for graph-proximity ranking), brain_read (a path-scoped file read that refuses symlinks, vault escapes, and anything outside a readable space), brain_links (a note's backlinks and outgoing wikilinks), brain_graph (a note's typed relations — up, down, same, prev, next — each edge labeled with why it exists), brain_recent (recently changed notes, from the vault's git history), and brain_facts (time-stamped fact lines — current, as-of, or believed-on a date). See the CLI reference for brain index, brain search, brain graph, brain facts, and brain mcp.

Ask AI about this page