The Chronicle Learns to Search, or How We Measured Before We Guessed

October 4, 2026 AI agentsSearchSRESQLite

A shared memory for coding agents, two weeks later: a ruler made of our own questions, a bug it found, ranking without a database, and an audit of the Chronicle against the code

In the previous tale the familiars got a memory: the Chronicle, a repository of markdown in git where every finding goes through a merge request. I finished that tale with a confession about search. It finds exact words only, there is no ranking, and “the day it stops being enough we will know what to take from the third traveller”.

I thought that day would come in a few months. It came in two weeks, and the question did not come from the familiars. It came from us.

🗺 First, the map

After the last tale a few people asked me the same thing: how is all of this connected? The Book with plugins, the Chronicle, the skills, the hook, the project repos. Fair question, so here is the map before the story.

The interactive version lets you follow six flows step by step (set up a laptop, look something up, capture a finding, onboard a repo, the monthly ritual, readers without a laptop) and click any part to see what it does: https://devops01ua.github.io/agent-knowledge-sample/architecture.html

In short:

📖 the Book (marketplace repo) gives every agent the same plugins. One of them is the wiki plugin: a stop hook and a setup skill
💻 the setup skill clones the Chronicle to the laptop, installs the CLI and links five skills into the agent
🧠 the Chronicle holds the schema, the wiki, the memory, the raw sources and a log, and nothing gets into main without a merge request
🧭 every project repo has its own AGENTS.md, and the Chronicle builds the repo catalogue from them
🤖 readers without a laptop get the same data another way: a clone in CI, an export for the hosted agent

Now the story.

🧪 The temptation

When a search misses, the first idea is always to bring a bigger tool. Embeddings. A vector store. A managed memory service. A knowledge graph. Or the new open source “context database” that somebody dropped in the team chat on Monday morning (there is always one).

So before building anything, I asked for a deep research: is markdown in git still the right shape for a few hundred pages, or is it time to move?

The answer came in three parts:

🧱 the shape is fine. For a corpus of about three hundred pages, none of the verified sources shows a vector store, a managed memory or a graph doing better
🔍 the weak spot is the search itself: exact substrings, every word required, no order. Ranked lexical search first, embeddings maybe later
🕳 and none of the benchmarks measured a corpus as small as ours. They talk about hundreds of thousands of documents

The last part was the useful one. If nobody measured a corpus of our size, we had to measure it ourselves.

📏 A ruler made of our own questions

Every session of a familiar leaves a transcript. Every time a familiar searched the Chronicle, the command is in there, next to the question the builder asked.

I went through them:

121 real searches from 36 past sessions
62 of them had a clear intent behind them
60 of those had an answer somewhere in the Chronicle, 2 did not

Then somebody had to say which page is the right answer for each question. I gave that job to four familiars, a quarter of the questions each, with one rule: you cannot use the search we are measuring. Only grep, the index and reading pages. Labels made with the tool under test would measure nothing.

A hit means the right page is in the first ten lines of the output. The first run:

misses: 12 of 60 (20%)
first page right: 32
in the first three: 44

One search in five did not find a page that was there. Not great, not terrible.

🐛 The ruler found a bug

The misses fell into two piles, and the first one surprised me.

with the repo filter: 8 of 20 missed (40%)
without filters:     4 of 40 missed (10%)

Every record in memory/ has a repos field. No page in wiki/ has one. So the repo filter (--in-repo platform in the sample) quietly threw away every wiki page. And the look-up skill told the familiar to search with the repo filter first. Yes, I wrote that skill.

A deep research would never find this, because it is three lines of our own code. The fix: with search words, the repo filter narrows the records and keeps the shared pages. Without words it still lists the records of that repo, as before.

misses: 12 → 8

📚 Ranking without a database

The second pile was about words. The page says “release flow”, the familiar asks for “release strategy”, and since every word was required, the page was gone.

I did not want a database for three hundred files. Then I remembered that SQLite has a full text engine with BM25 inside, and Python ships with SQLite. So the CLI now builds an in-memory FTS5 table on every call. For a few hundred files it takes about 0.2 seconds, and there is no index file to keep fresh.

What changed in the output:

🥇 pages with every word come first, best first. A word in the path, the id, the summary or the tags counts more than a word somewhere in the body
🧩 pages with only some of the words come after a marker line, so one unlucky word does not empty the whole result
✂️ ten lines by default, --limit if you want more

acme-kb search gitops token --limit 3

memory/findings/F-2026-09-16-001.md  [fixed]  The GitOps server cannot issue API tokens for SSO users ...
wiki/systems/gitops-server.md  [wiki]  The GitOps server that deploys every workload ...
-- not every term matched --
memory/incidents/I-2026-09-25-001.md  [closed]  The payments rollout in prod paused at 1 of 4 replicas ...

misses:              8 → 2
first page right:   33 → 51
in the first three: 46 → 56

If the local SQLite has no FTS5, the same pages come back, sorted by how many words they have. Worse order, same pages.

🏷 The names people actually use

The last two misses were “gitflow” and “release strategy”. Nobody uses these words on the release flow page, but people use them all the time when they ask.

So now a page or a record can have aliases, the other names people search by:

aliases: [gitflow, "release strategy", "branching strategy"]

Search treats them like the summary.

misses: 2 → 0

Zero looks nice, but I wrote those two aliases for exactly those two questions. So it proves that aliases work, not that search is now perfect.

📜 A contract, and a question about MCP

The text output is one result per line with the path and the summary. A familiar reads it fine, a script does not. So there is now --json: the same ranked results as records, with path, kind, match (full or partial), id, status, confidence, summary, systems, repos and dates. The schema sits in the repo as a JSON Schema file with a version, and a test checks the real output against it, so they cannot drift apart.

This came from a question in the team: “should we build an MCP server, we have contracts?”. It depends on who reads:

💻 familiars on laptops: no MCP needed, they have a shell and a clone
🤖 agents in CI: no MCP either, a clone and the same CLI with --json is enough
☁️ a hosted ops agent that cannot run a CLI: only here, and only if a static export is not enough

So we got the contract from the CLI. MCP stays an option for one reader.

🔎 The Chronicle checks itself

After two weeks of captures I wanted to know how much of the Chronicle is still true. So we ran an audit: every open record against the code it describes, and every recent commit against the pages.

Six familiars, one group of repositories each, reading only the remote branches, no live systems:

open records: 114
still accurate:        78
fixed in code:         12
text no longer true:   12
cannot tell from git:  12
pages behind the code: 51 places

Two things we agreed on before changing anything:

✅ merged does not mean fixed. Ten fixes were in the code, but nobody had looked at the cluster yet. They stay open, with the commit written next to them, until somebody checks
🧐 an audit made by familiars is a list of leads, not a list of facts. The familiars that applied the corrections had to check every claim again before writing it. They threw out several: a “fourth user” of a secrets store that was only a mention in a text file, a fix credited to the wrong commit, a data source with the wrong name

And one finding was not about the Chronicle at all. Somebody renamed a node pool, but two other lists still had the old name, and both lists are validated against the pool names. The current release was fine. The next one would fail at plan time. We filed a ticket and told the author before anyone cut a new tag.

🔁 Two more pieces, in review

📒 a search log. Every search writes one line to a local file that git ignores: the words, how many pages matched, the first results. acme-kb queries shows how often searches miss or get rephrased. Once a month the misses (only the misses) go to the Chronicle as a merge request, and each one gets an alias or a page. So the ruler keeps measuring after this tale is published
🗺 a map instead of a phone book. The index had one line per page and grew to more than 80 KB. Now index.md has one line per system: its page, how many records mention it and how many of those are open. About 11 KB. The full list is still there, in the next file

⚠️ What the ruler does not tell you

the labels were made by familiars, and no human has reviewed them yet
all 62 questions came from my own sessions
“no matches” is rare now: even a question without an answer gets a few partial matches under the marker, and the familiar has to notice the marker
the ruler measures search, not answers. A familiar can still find the right page and read it wrong

⚡ Epilog

The third traveller said that vectors and rerankers can wait as long as plain search copes. I did not have to guess when that moment comes. We built a ruler, and the ruler said: not yet, but your filter is broken.

For the builders the habits stay the same. Ask the familiar “how do we do this here”. Say “capture this” when you dig something up. And when a search misses, tell it what you would call the thing. That is an alias now.