fleet ops · · 6 min · updated 2026-10-27

The answer that used to be true

An agent taking over acts on a fact someone else wrote. Overwriting memory hands back a stale answer with nothing on it that says so.

On this page · 4 sections
  1. One entry, one file; one change, one commit
  2. Ask in Russian, answer from English
  3. Most of the base is versions that changed an entry
  4. Two minutes with no memory at all

First published on 2026-09-19. The measurements reported in this article are historical and have not been re-run for this revision. The article reports version 0.2.1 for its search measurements; the vector-cache measurement cannot be 0.2.1 as released and its build is not established (see the correction in the text).

An incident opened on Thursday, was worked on Friday, and was finished on Monday by a different agent. It asked the knowledge base where the service writes, got an address, and went to fix it there. The address had moved on Friday evening - moved by the agent that no longer existed on Monday.

Nothing failed. The answer was plausible, complete and wrong, and there was nothing on it to say it was stale. That is the failure we stopped overwriting the fleet’s shared memory to avoid.

One entry, one file; one change, one commit

KYB is one HTTP service and one CLI over a git repository. An entry is a markdown file, a change is a commit, and the commit sha is the version id. The Tantivy index is a disposable cache: delete the directory and it rebuilds from git on the next start.

Writing a fact and changing it are the same call - add on the same key:

$ kyb add --key checkout-write-path --title "Checkout write path" \
    --tags infra,checkout <<< "Checkout writes go through svc-checkout on node-3, port 8080."
{"action":"created","changed":true,"key":"checkout-write-path","sha":"b8f2187..."}

$ kyb add --key checkout-write-path --title "Checkout write path" \
    --tags infra,checkout <<< "Checkout writes go through svc-checkout on node-3, port 9090."
{"action":"updated","changed":true,"key":"checkout-write-path","sha":"7b7d4eb..."}

created against updated is not cosmetic: the second call did not replace the first, it added a version. The chain, and any link in it:

$ kyb history checkout-write-path
{"key":"checkout-write-path","versions":[
  {"change":"modified","sha":"7b7d4eb...","committed_at":"2026-09-19T02:07:51+00:00"},
  {"change":"added",   "sha":"b8f2187...","committed_at":"2026-09-19T02:07:50+00:00"}]}

$ kyb get checkout-write-path --at b8f2187
Checkout writes go through svc-checkout on node-3, port 8080.

--at takes a revision that git can resolve to a commit. That is the answer to “what did we believe then” - the one the Monday agent did not have.

The schema is not neutral either. An incident filed without an executable check gets told what is missing from it, in the response:

$ kyb incident --key inc-2026-05-14-checkout-timeouts --service svc-checkout \
    --severity high --title "Checkout timeouts after the port move" <<< "Latency past 5 s for 40 minutes."
{"action":"created","hints":[
  "no detection: add an executable 'is it still happening?' check with the expected healthy result (--detection)",
  "no affected windows: if data or a period got poisoned, record {scope,from,to} in --affected ...",
  "no 'Root cause' section: state it and mark the confidence - verified | suspected | unknown"]}

A report that cannot tell you whether the thing is still happening is a story, not a report.

Ask in Russian, answer from English

Search is hybrid: BM25 fused by reciprocal rank with vectors from multilingual-e5-small int8 on CPU. The vectors do not rerank the lexical result, they retrieve, and one flag shows the difference.

The entries above are in English. The question is in Russian:

$ kyb query "куда пишет оформление заказа" --limit 2
{"semantic":true,"count":2,"hits":[{"key":"checkout-write-path","score":0.774}, ...]}

The same question with the semantic side switched off, ?semantic=false:

{"semantic":false,"count":0}

Zero. There is nothing to rerank: the question shares no token with the entry, so BM25 returns no candidates at all. This is not vectors improving an order - it is the only way anything is found.

Most of the base is versions that changed an entry

After two months of fleet use we measured the base instead of estimating it:

measured 2026-09-18

866   keys carry history (live entries plus archived incidents and tasks)
3217  versions in total
3.71  mean depth per key
74 %  of keys have two or more versions
2351  transitions between versions

The last line is the interesting one. The 2351 transitions are about 73 % of the 3217 versions, and 74 % of the 866 keys have two or more versions: most of the base is not write-once notes, it is entries that changed. The count is of edits, not of their kind - a port moved, a host was replaced, a decision was reversed are examples of what such an edit can be, not a breakdown of the 2351. Exactly what an overwriting store erases silently.

Correction, 2026-10-27. As published on 2026-09-19, this section said that two thirds of the base is facts that moved. The figures shown establish something different on both counts: the 2351 transitions are about 73 % of the 3217 versions, and the per-key share shown is 74 %, so neither denominator gives two thirds; and a transition is a new version of an entry, which the measurement does not classify as a fact that moved rather than any other edit. The “two thirds” figure is withdrawn, and the claim is narrowed to versions that changed an entry.

What is superseded does not disappear, it moves behind --history. The string 8080 now lives only in the first version, and a search of the current state, as the article ran it, came back empty:

$ kyb query "8080" --limit 5                # current state
{"count":0}

$ kyb query "8080" --history --limit 5      # every version
{"count":1,"hits":[{"key":"checkout-write-path","sha":"b8f2187...","is_head":false}]}

Correction, 2026-10-27. As published on 2026-09-19, this section presented the first command above, kyb query "8080" --limit 5, as the lexical search of the current state. The CLI of version 0.2.1 sends no semantic=false with that command, so it does not force a lexical query: with a semantic model loaded and working it runs as a hybrid query, and without a model, or when the hybrid path falls back, as a lexical one (main.rs at 7fa48b6e). Which of the two this historical run was is not on record; the lexical side alone is what the ?semantic=false request earlier in this post selects. The {"count":0} output is kept as published. The claim that the command as shown is the lexical comparison is withdrawn as unestablished; if the run was hybrid, the caveat below - that hybrid search can find the entry by meaning - applies to it. is_head:false on every hit is a machine-readable mark that the answer came from the past rather than the present. A client can check it, not only a reader.

One caveat, which we nearly wrote wrongly ourselves: with a working semantic model, hybrid search can find such an entry without --history, by meaning, so a zero from a hybrid query is not the lexical comparison this paragraph is about. Whether the zero above came from a lexical or a hybrid run is not on record (see the correction). Compare against the thing you mean to compare against.

Two minutes with no memory at all

The index and the vectors are rebuilt from git on every start, and nobody had measured what that costs either. We did:

2.1 s     fixed startup (model load)
244 ms    per embedded passage
565       live entries at the time -> about 140 s before the first answer

The 140 s is an extrapolation from the two measured figures (2.1 s plus 565 passages at 244 ms each), not a timed start; the article does not record the corpus, model or hardware behind those figures. By that estimate, about two minutes in which the fleet has no shared memory. The symptom is indistinguishable from “the service is still coming up” - which it is: nothing logged the duration, /healthz only answers once it finishes, and an agent asking inside that window got a connection error and carried on without the base. It reads as a network blip, not as a systematic hole.

The fix followed from the same foundation. A version’s text never changes, so a vector for it cannot go stale, so it can go on disk keyed by the hash of that text and never be invalidated:

200 entries, no cache   75.3 s
200 entries, cached      8.2 s

Correction, 2026-10-27. As published on 2026-09-19, this post placed everything, including the cache, under version 0.2.1. What the source establishes is narrower: released 0.2.1 keeps vectors in memory and has no on-disk cache (embed.rs at 7fa48b6e), so the cache described here cannot be 0.2.1 as released. Which build produced the cache run, and when, is not on record, and no measurement log survives for either number above; both timings are author-reported and unverified, and no version is assigned to them.

A cache that cannot go stale by construction is rare, and here it came free. But “cannot go stale” is true of the content and false of the model: vectors from different models are not comparable, and a cache that never invalidates would never throw the old ones out. So the cache carries a fingerprint of the model and of the recipe - prefixes and token budget - not only the weights: the model is not handed the text you hashed, it is handed that text with a prefix, truncated to a budget.

We would have measured sooner. All three numbers were available from day one and were never taken until somebody asked directly. By that estimate, a start of about two minutes had been happening in plain sight on every restart; how often the service restarted was not recorded, so no outage frequency is claimed.

The rule this post argues for: if a decision depends on a magnitude, say what magnitude you expect before you read the measured one. A wrong label on a right number survives any amount of re-checked arithmetic - it contradicts nothing until you have stated an expectation for it to contradict.

KYB is open source, MIT: github.com/alex09x/kyb. Every line of output above was taken from a running instance, as reported; the search output is from version 0.2.1, while the cache experiment cannot be 0.2.1 as released and its build is not on record (see the correction above). The temporal query layer - “what did we believe on this date” and “what moved this week” - is built but not released; that is its own post, once it ships.

Cite this article
Citation
Alexander Panasenko (2026-09-19). The answer that used to be true. https://prod.codes/blog/the-answer-that-used-to-be-true/