What our ripgrep report can and cannot show
We reassessed our September ripgrep report. Its published excerpts do not substantiate the claimed tool coverage or speedups; issue #735 documents a narrower failure in prod-code.
On this page · 4 sections
First published on 2026-09-29. The measurements reported in this article are historical and have not been re-run for this revision.
This is a correction. On 2026-09-29 we published a report on pointing prod-code at BurntSushi/ripgrep; it claimed a full 67-tool evaluation, a table of speed-ups over a local machine, and bugs found along the way. On 2026-10-11 we reassessed it against its own evidence; on 2026-10-11 a notice replaced the original at this address, and on 2026-10-11 this corrected account replaced the notice. The original text is kept in the site’s source history and is not republished.
What the report can show
We do not have a complete execution record for the reported evaluation. The article
preserves quoted commands, outputs and patches, but the raw logs are unavailable to this
review and the exact ripgrep checkout revision is not recorded in the available material;
crate version strings such as grep-printer 0.2.2 do not identify it. Issue #735 records
client version 0.3.19 for one reported slice call; it does not establish the tool version
used for every example. The hardware description and the high-level cache setup the original
article described, a cold sync after the remote workspace storage was cleared and then a warm
incremental sync, are author-reported; their actual state during execution and the number of
repetitions are not independently established. An excerpt is part of a claim, not evidence that the
command ran as shown: this review can check whether the prose agrees with the excerpt next
to it and whether the excerpts agree with each other; it cannot independently establish
that an excerpt came from the stated execution.
Correction, 2026-10-27. As revised on 2026-10-11, this paragraph said cache conditions “are not documented”. The original article did describe them at a high level: it cleared the remote workspace storage before a cold sync and then ran a warm incremental sync. That description is restored here as author-reported; what remains unverified is the actual cache state during each run and how many times each was repeated.
With that reading, several rows of the report’s 67-tool matrix have no command and no
output anywhere in the text: code_symbols, code_outline, code_prune_orphans,
code_validate_edits, code_diagnose_failure, code_move_module, and the eight
cross-language refactorings from inheritance-to-delegation to parameter objects. The
available report does not substantiate the claim that all 67 tools were evaluated. When
execution records are missing, the status is unverified; we use “not run” only when we
know no execution was attempted, and keep failed, empty-result and unsupported outcomes
separate.
The report contains result messages, patch excerpts and, for some examples such as the rename, quoted check and test output. Other transformations assert follow-up checks in prose without showing corresponding output. These fragments record what the article attributes to the tools; they do not independently establish that the commands ran, that the patches were applied or that the results were correct. The evaluation matrix does not replace execution records for each listed tool.
Where the text and its excerpts part ways
Three examples, for three different reasons, show why the summary cannot be taken at face value.
A direct contradiction: the text says “91 AST matches across 13 files”. The excerpt says
91 match(es) in 2 file(s) (13 scanned in 18.88ms).
An unsupported comparison: the summary presents a 0.4 s lint run as an “11x faster”
result. The excerpt says rust lint: FAILED in 0.4s on linux x86_64; 9 error(s), 0 warning(s).
The quoted output reports a failed lint run with nine errors, which the report does not
discuss; it does not establish a clean pass. Read as an excerpt, the tool passed a failure
through, which is what it should do; that is not a successful lint, and “11x” compares it
with a local baseline for which no supporting measurement is available. No supporting
measurements are provided for the whole local-Mac comparison column of that table, and
every speed-up derived from it goes with it. The description promised “sub-millisecond
sync”; the two sync excerpts say 969 ms and 171 ms.
An unwarranted conclusion: the table presents the reported 6.5 s command duration as “Non-throttled benchmarking”. The excerpt reports 6.5 seconds elapsed and zero benchmarks. It therefore provides no benchmark measurement supporting the claimed advantage.
A traceable prod-code bug report
The report’s title asked whether 67 tools could break ripgrep. The report does not
establish a defect in ripgrep. The excerpt reports a module cycle in grep-searcher,
lib.rs to sink.rs and back; the report called it architectural. A cycle in a
tool-generated module graph is not, on its own, evidence of a software defect, and Rust
allows import cycles inside one crate. The clippy findings are listed, not examined.
What does have a trace is a bug in prod-code. Issue #735
reports that running prod-code slice on crates/globset/src/lib.rs failed with a range ends before it starts: the language server returned a symbol whose range ended before it
began, and the slicer rejected the whole answer. The issue was filed 2026-09-30 02:20 UTC
with client 0.3.19. Commit
84d01dee,
04:18 UTC the same day, adds parse_symbol_span, which swaps an inverted start and end, and
a regression test, inverted_document_symbol_ranges_are_normalized_gracefully. That is
source-level evidence of the change, not of the test passing or of an independent
reproduction; we have not re-run that test for this post. For the report’s third item, a
“workspace purge” request type described as designed, we did not locate a supporting
implementation or design record in the repository history inspected for this audit.
What changes
A report like this one now needs, before publication: the commit of the evaluated repository; the versions of prod-code, the analyzer and the compiler; the node and its state; and for every command it quotes, the command, its context, its exit status and its output, if any, kept outside the article. A command that fails, runs nothing or returns an empty graph is reported as exactly that. A tool without an execution record is “execution unverified”, not “verified”. Performance comparisons need both sides measured, with repetitions. Bug reports are credited to the tool that broke.
We will apply the same evidence checks to the remaining reports in the series, one at a time.
Cite this article
Alexander Panasenko (2026-09-29). What our ripgrep report can and cannot show. https://prod.codes/blog/can-67-ast-tools-break-ripgrep/