fleet ops · · 4 min

The answer that used to be true

An agent taking over acts on a fact someone else wrote. Overwriting memory hands back a stale answer with nothing on it that says so.

On this page · 4 sections
  1. One entry, one file; one change, one commit
  2. Ask in Russian, answer from English
  3. Two thirds of the base is facts that moved
  4. Two minutes with no memory at all

An incident opened on Thursday, was worked on Friday, and was finished on Monday by a different agent. It asked the knowledge base where the service writes, got an address, and went to fix it there. The address had moved on Friday evening — moved by the agent that no longer existed on Monday.

Nothing failed. The answer was plausible, complete and wrong, and there was nothing on it to say it was stale. That is the failure we stopped overwriting the fleet’s shared memory to avoid.

One entry, one file; one change, one commit

KYB is one HTTP service and one CLI over a git repository. An entry is a markdown file, a change is a commit, and the commit sha is the version id. The Tantivy index is a disposable cache: delete the directory and it rebuilds from git on the next start.

Writing a fact and changing it are the same call — add on the same key:

$ kyb add --key checkout-write-path --title "Checkout write path" \
    --tags infra,checkout <<< "Checkout writes go through svc-checkout on node-3, port 8080."
{"action":"created","changed":true,"key":"checkout-write-path","sha":"b8f2187..."}

$ kyb add --key checkout-write-path --title "Checkout write path" \
    --tags infra,checkout <<< "Checkout writes go through svc-checkout on node-3, port 9090."
{"action":"updated","changed":true,"key":"checkout-write-path","sha":"7b7d4eb..."}

created against updated is not cosmetic: the second call did not replace the first, it added a version. The chain, and any link in it:

$ kyb history checkout-write-path
{"key":"checkout-write-path","versions":[
  {"change":"modified","sha":"7b7d4eb...","committed_at":"2026-09-19T02:07:51+00:00"},
  {"change":"added",   "sha":"b8f2187...","committed_at":"2026-09-19T02:07:50+00:00"}]}

$ kyb get checkout-write-path --at b8f2187
Checkout writes go through svc-checkout on node-3, port 8080.

--at takes any revision git can resolve. That is the answer to “what did we believe then” — the one the Monday agent did not have.

The schema is not neutral either. An incident filed without an executable check gets told what is missing from it, in the response:

$ kyb incident --key inc-2026-05-14-checkout-timeouts --service svc-checkout \
    --severity high --title "Checkout timeouts after the port move" <<< "Latency past 5 s for 40 minutes."
{"action":"created","hints":[
  "no detection: add an executable 'is it still happening?' check with the expected healthy result (--detection)",
  "no affected windows: if data or a period got poisoned, record {scope,from,to} in --affected ...",
  "no 'Root cause' section: state it and mark the confidence — verified | suspected | unknown"]}

A report that cannot tell you whether the thing is still happening is a story, not a report.

Ask in Russian, answer from English

Search is hybrid: BM25 fused by reciprocal rank with vectors from multilingual-e5-small int8 on CPU. The vectors do not rerank the lexical result, they retrieve, and one flag shows the difference.

The entries above are in English. The question is in Russian:

$ kyb query "куда пишет оформление заказа" --limit 2
{"semantic":true,"count":2,"hits":[{"key":"checkout-write-path","score":0.774}, ...]}

The same question with the semantic side switched off, ?semantic=false:

{"semantic":false,"count":0}

Zero. There is nothing to rerank: the question shares no token with the entry, so BM25 returns no candidates at all. This is not vectors improving an order — it is the only way anything is found.

Two thirds of the base is facts that moved

After two months of fleet use we measured the base instead of estimating it:

measured 2026-09-18

866   keys carry history (live entries plus archived incidents and tasks)
3217  versions in total
3.71  mean depth per key
74 %  of keys have two or more versions
2351  transitions between versions

The last line is the interesting one. Two thirds of the base is not write-once notes, it is facts that changed: a port moved, a host was replaced, a decision was reversed. Exactly what an overwriting store erases silently.

What is superseded does not disappear, it moves behind --history. The string 8080 now lives only in the first version, and a lexical search says so literally:

$ kyb query "8080" --limit 5                # current state
{"count":0}

$ kyb query "8080" --history --limit 5      # every version
{"count":1,"hits":[{"key":"checkout-write-path","sha":"b8f2187...","is_head":false}]}

is_head:false on every hit is a machine-readable mark that the answer came from the past rather than the present. A client can check it, not only a reader.

One caveat, which we nearly wrote wrongly ourselves: hybrid search finds that entry without --history too, by meaning. The zero above is lexical. Compare against the thing you mean to compare against.

Two minutes with no memory at all

The index and the vectors are rebuilt from git on every start, and nobody had measured what that costs either. We did:

2.1 s     fixed startup (model load)
244 ms    per embedded passage
565       live entries at the time -> about 140 s before the first answer

Two minutes in which the fleet has no shared memory. The symptom is indistinguishable from “the service is still coming up” — which it is: nothing logged the duration, /healthz only answers once it finishes, and an agent asking inside that window got a connection error and carried on without the base. It reads as a network blip, not as a systematic hole.

The fix followed from the same foundation. A version’s text never changes, so a vector for it cannot go stale, so it can go on disk keyed by the hash of that text and never be invalidated:

200 entries, no cache   75.3 s
200 entries, cached      8.2 s

A cache that cannot go stale by construction is rare, and here it came free. But “cannot go stale” is true of the content and false of the model: vectors from different models are not comparable, and a cache that never invalidates would never throw the old ones out. So the cache carries a fingerprint of the model and of the recipe — prefixes and token budget — not only the weights: the model is not handed the text you hashed, it is handed that text with a prefix, truncated to a budget.

We would have measured sooner. All three numbers were available from day one and were never taken until somebody asked directly. The two-minute start had been happening in plain sight every week.

The rule this post argues for: if a decision depends on a magnitude, say what magnitude you expect before you read the measured one. A wrong label on a right number survives any amount of re-checked arithmetic — it contradicts nothing until you have stated an expectation for it to contradict.

KYB is open source, MIT: github.com/alex09x/kyb. Everything above is version 0.2.1, and every line of output was taken from a running instance. The temporal query layer — “what did we believe on this date” and “what moved this week” — is built but not released; that is its own post, once it ships.

Cite this article
Citation
Alexander Panasenko (2026-09-19). The answer that used to be true. https://prod.codes/blog/the-answer-that-used-to-be-true/