notes · · 5 min · updated 2026-10-27

What our ripgrep report can and cannot show

We reassessed our September ripgrep report. Its published excerpts do not substantiate the claimed tool coverage or speedups; issue #735 documents a narrower failure in prod-code.

On this page · 4 sections
  1. What the report can show
  2. Where the text and its excerpts part ways
  3. A traceable prod-code bug report
  4. What changes

First published on 2026-09-29. The measurements reported in this article are historical and have not been re-run for this revision.

This is a correction. On 2026-09-29 we published a report on pointing prod-code at BurntSushi/ripgrep; it claimed a full 67-tool evaluation, a table of speed-ups over a local machine, and bugs found along the way. On 2026-10-11 we reassessed it against its own evidence; on 2026-10-11 a notice replaced the original at this address, and on 2026-10-11 this corrected account replaced the notice. The original text is kept in the site’s source history and is not republished.

What the report can show

We do not have a complete execution record for the reported evaluation. The article preserves quoted commands, outputs and patches, but the raw logs are unavailable to this review and the exact ripgrep checkout revision is not recorded in the available material; crate version strings such as grep-printer 0.2.2 do not identify it. Issue #735 records client version 0.3.19 for one reported slice call; it does not establish the tool version used for every example. The hardware description and the high-level cache setup the original article described, a cold sync after the remote workspace storage was cleared and then a warm incremental sync, are author-reported; their actual state during execution and the number of repetitions are not independently established. An excerpt is part of a claim, not evidence that the command ran as shown: this review can check whether the prose agrees with the excerpt next to it and whether the excerpts agree with each other; it cannot independently establish that an excerpt came from the stated execution.

Correction, 2026-10-27. As revised on 2026-10-11, this paragraph said cache conditions “are not documented”. The original article did describe them at a high level: it cleared the remote workspace storage before a cold sync and then ran a warm incremental sync. That description is restored here as author-reported; what remains unverified is the actual cache state during each run and how many times each was repeated.

With that reading, several rows of the report’s 67-tool matrix have no command and no output anywhere in the text: code_symbols, code_outline, code_prune_orphans, code_validate_edits, code_diagnose_failure, code_move_module, and the eight cross-language refactorings from inheritance-to-delegation to parameter objects. The available report does not substantiate the claim that all 67 tools were evaluated. When execution records are missing, the status is unverified; we use “not run” only when we know no execution was attempted, and keep failed, empty-result and unsupported outcomes separate.

The report contains result messages, patch excerpts and, for some examples such as the rename, quoted check and test output. Other transformations assert follow-up checks in prose without showing corresponding output. These fragments record what the article attributes to the tools; they do not independently establish that the commands ran, that the patches were applied or that the results were correct. The evaluation matrix does not replace execution records for each listed tool.

Where the text and its excerpts part ways

Three examples, for three different reasons, show why the summary cannot be taken at face value.

A direct contradiction: the text says “91 AST matches across 13 files”. The excerpt says 91 match(es) in 2 file(s) (13 scanned in 18.88ms).

An unsupported comparison: the summary presents a 0.4 s lint run as an “11x faster” result. The excerpt says rust lint: FAILED in 0.4s on linux x86_64; 9 error(s), 0 warning(s). The quoted output reports a failed lint run with nine errors, which the report does not discuss; it does not establish a clean pass. Read as an excerpt, the tool passed a failure through, which is what it should do; that is not a successful lint, and “11x” compares it with a local baseline for which no supporting measurement is available. No supporting measurements are provided for the whole local-Mac comparison column of that table, and every speed-up derived from it goes with it. The description promised “sub-millisecond sync”; the two sync excerpts say 969 ms and 171 ms.

An unwarranted conclusion: the table presents the reported 6.5 s command duration as “Non-throttled benchmarking”. The excerpt reports 6.5 seconds elapsed and zero benchmarks. It therefore provides no benchmark measurement supporting the claimed advantage.

A traceable prod-code bug report

The report’s title asked whether 67 tools could break ripgrep. The report does not establish a defect in ripgrep. The excerpt reports a module cycle in grep-searcher, lib.rs to sink.rs and back; the report called it architectural. A cycle in a tool-generated module graph is not, on its own, evidence of a software defect, and Rust allows import cycles inside one crate. The clippy findings are listed, not examined.

What does have a trace is a bug in prod-code. Issue #735 reports that running prod-code slice on crates/globset/src/lib.rs failed with a range ends before it starts: the language server returned a symbol whose range ended before it began, and the slicer rejected the whole answer. The issue was filed 2026-09-30 02:20 UTC with client 0.3.19. Commit 84d01dee, 04:18 UTC the same day, adds parse_symbol_span, which swaps an inverted start and end, and a regression test, inverted_document_symbol_ranges_are_normalized_gracefully. That is source-level evidence of the change, not of the test passing or of an independent reproduction; we have not re-run that test for this post. For the report’s third item, a “workspace purge” request type described as designed, we did not locate a supporting implementation or design record in the repository history inspected for this audit.

What changes

A report like this one now needs, before publication: the commit of the evaluated repository; the versions of prod-code, the analyzer and the compiler; the node and its state; and for every command it quotes, the command, its context, its exit status and its output, if any, kept outside the article. A command that fails, runs nothing or returns an empty graph is reported as exactly that. A tool without an execution record is “execution unverified”, not “verified”. Performance comparisons need both sides measured, with repetitions. Bug reports are credited to the tool that broke.

We will apply the same evidence checks to the remaining reports in the series, one at a time.

Cite this article
Citation
Alexander Panasenko (2026-09-29). What our ripgrep report can and cannot show. https://prod.codes/blog/can-67-ast-tools-break-ripgrep/