1,485 Error Checks and Zero Import Cycles: Dissecting Prometheus's 84-Package Engine with 67 AST Tools

prometheus/prometheus is the bedrock of modern cloud-native observability. Written in Go, it ingests hundreds of millions of time-series samples per second, evaluates PromQL vector queries on the fly, and manages massive inverted label indices in its custom TSDB.

Architecturally, Prometheus is also a quintessential Go monorepo: 385,511 lines of Go across 736 source files organized into 84 internal packages. Unlike Rust—where developers wrestle with borrow checkers, lifetime annotations, and complex trait bounds—Go’s engineering ethos revolves around simplicity: explicit error returns (if err != nil), interfaces satisfied implicitly by structural shape, and a compiler that strictly forbids circular imports at the package level.

For AI coding agents and local IDEs, however, an industrial Go repository of this size poses serious challenges:

• Whole-workspace type checking (gopls) and AST pattern extraction can peg CPU cores and throttle laptops.
• LLM agents routinely drown in thousands of lines of boilerplate error handling when seeking the core logic of a subsystem.
• Naive static analysis tools often conflate package imports with file-level cross-references, leading to false dependency cycles.

To test how our cluster-backed code intelligence architecture handles a flagship Go codebase with 0% local laptop CPU, we subjected prometheus/prometheus to our standardized 67-tool evaluation protocol of prod-code on our 32-core Linux build node (ram9).

Here is what 67 AST analyzers discovered inside Prometheus.

---

1. Node Topology & Fast-Sync Ingestion

Rather than executing heavy language servers or builds locally on a development laptop, prod-code mirrors the working tree to a dedicated build server over low-latency LAN:

Local Checkout:   /Users/alex09x/Documents/workspace/prometheus-eval
Target Host:      ram9 (192.168.2.143:9400 / AMD EPYC 32-core, 128 GB RAM)
LAN Latency:      865.21 µs RTT
Go Source Files:  736 (.go)
Total Files:      998
Lines of Go:      385,511 across 84 packages

Ingestion Telemetry

• Initial Cold Sync: 998 files transferred, manifest hashed, and hydrated into gopls in 14.1s.
• Declaration Indexing: 19,738 symbols and method sets indexed in server memory.
• Server Footprint: gopls and the prod-code daemon occupied 413 MB RSS on ram9.
• Local Laptop Load: 0.0% CPU, zero fan noise, zero battery drain.

---

2. Architecture DAG: 84 Packages, 0 Cycles

Go’s compiler strictly rejects circular dependencies between packages. But what does the overall dependency graph look like, and which packages serve as the foundational architectural bedrock?

Using prod-code dependencies, we computed the afferent coupling (Cₐ), efferent coupling (Cₑ), and instability index (I = Cₑ / (Cₐ + Cₑ)) across the entire repository:

⚡ Architecture & Dependency Graph Report
Scope: packages | Nodes: 84 | Dependencies: 524
✓ Zero circular dependencies detected. Architecture graph is a clean DAG.

Foundations by Afferent Coupling (Cₐ)

Package	Inbound (Cₐ)	Outbound (Cₑ)	Instability (I)	Architectural Role
discovery/targetgroup	35	0	0.000	Pure data structures for scraped targets
discovery	34	2	0.056	Target discovery manager & interfaces
model/labels	32	0	0.000	Immutable label sets & matchers
model/histogram	26	2	0.071	Sparse high-resolution exponential histograms
discovery/refresh	23	2	0.080	Common discovery polling scheduler
util/strutil	22	0	0.000	String sanitization & conversion helpers
storage	18	7	0.280	Core Queryable, Appender, and ChunkSeries
model/timestamp	14	0	0.000	Millisecond timestamps
tsdb/chunkenc	11	2	0.154	Gorilla XOR chunk encoders & decoders
promql/parser	10	8	0.444	AST parser & lexer for PromQL

Notice the pristine architectural discipline: foundational packages like model/labels, util/strutil, and discovery/targetgroup have an Instability of 0.000 (Cₑ = 0). They depend on nothing inside the project and serve as rock-solid roots for the remaining 81 packages.

---

3. Bug #740 Discovered & Fixed: The Runaway Dependency Stream

During our initial evaluation, we executed prod-code dependencies --scope modules without a directory filter. Immediately, the JSON-RPC stream was inundated with tens of thousands of lines of output:

🚨 CYCLES DETECTED: 586 circular dependency path(s) found:
  1. discovery::eureka::client_test -> discovery::eureka::eureka -> discovery::eureka::client_test
  2. discovery::eureka::client_test -> discovery::eureka::eureka -> discovery::eureka::eureka_test -> ...

The process flooded megabytes of text over the socket before being terminated. What happened?

Root Cause Analysis

1. Loose Suffix Matching in Import Parsing: In crates/prod-code-mcp/src/dependencies.rs, the Go import parser took the last segment of imported paths (pkg.split('/').last()) and matched it against known_modules.keys().ends_with(last_segment). When any file imported standard library packages such as "net/http", "context", "sync", "testing", or "time", the parser saw last_segment = "http" and created dependency edges to every internal file whose name ended with http!
2. Intra-Package Files as Independent Modules: In Go, all files in the same package (including _test.go and internal test helpers) live in the same directory and share package-level declarations without explicit imports. Treating individual files as separate module nodes created dense bipartite cliques between tests and implementations.
3. Uncapped Cycle Enumeration: The DFS cycle detector had no bound on collected cycles, and format_dependency_report formatted every single permutation path into stdout without truncation or pagination.

The Fix

We immediately filed Issue #740 via prod-code report-issue:

• Added a hard cap MAX_CYCLES_DETECTED = 100 in find_cycles and dfs_cycles.
• Implemented pagination and truncation in format_dependency_report (displaying 25 cycles maximum, followed by … and N more circular dependency path(s) truncated).
• Updated parse_go_imports to inspect go.mod, extracting the workspace module prefix (module github.com/prometheus/prometheus). Only internal imports matching the module prefix or explicit relative paths (./, ../) are mapped to workspace files, completely ignoring standard library and third-party dependencies.
• Added test coverage in test_cycle_detection_capped, test_format_dependency_report_truncation, and test_parse_go_imports_internal_matching.

The fix was committed in 3be3cdd, pushed to main, and deployed in seconds.

---

4. Structural AST Pattern Search: 1,485 Error Checks

One of the defining characteristics of Go is explicit error handling. While critics call it verbose, in Prometheus it ensures that disk I/O errors, network timeouts, and query parse failures are never silently swallowed.

Using prod-code structural-search, we executed a polyglot tree-sitter pattern query across the entire repository:

prod-code structural-search 'if err != nil { return $x }'

In 3.71 seconds, the engine scanned 998 files and identified 1,485 exact matches across 243 files, capturing the exact AST bindings for the metavariable $x:

• cmd/prometheus/main.go:1163:5
  if err != nil {
  └─ [$x = err]

• cmd/prometheus/main.go:1509:5
  if err != nil {
  └─ [$x = fmt.Errorf("opening storage failed: %w", err)]

• cmd/prometheus/main.go:1667:2
  if err != nil {
  └─ [$x = nil, err]

• cmd/prometheus/main_test.go:836:2
  if err != nil {
  └─ [$x = 0, err]

• cmd/prometheus/main_test.go:1070:5
  if err != nil {
  └─ [$x = false]

Notice the semantic variety captured in $x:

• Bare error returns: return err
• Wrapped errors using Go 1.13 %w formatting: return fmt.Errorf("opening storage failed: %w", err)
• Multi-value tuple returns: return nil, err or return 0, err
• Boolean guards in tests: return false

For automated code modifications or linters, structural search operates on the real AST, ignoring whitespace, formatting quirks, and line breaks.

---

5. Program Slicing: 99% Context Reduction

When an AI coding agent needs to modify or reason about a central struct like promql.Engine, feeding the entire file or surrounding modules into the prompt wastes tokens and injects distracting hallucinations.

promql/engine.go is 4,625 lines (211 KB) of dense concurrent evaluation logic.

We ran prod-code slice on Engine at line 367 with --depth 1:

prod-code slice promql/engine.go --line 367 --depth 1

In 12 milliseconds, the slicer extracted the bounded dependency closure:

BOUNDED slice of `Engine`: 5 item(s), 2,410 bytes from 211,502 bytes of source (99% smaller)
depth limit 1 reached: the dependencies of 4 item(s) at depth 1 were not looked up
outside the workspace, not followed: Duration, Logger, RWMutex, int64

What the Slicer Retained:

1. promql/engine.go:81 (struct engineMetrics): All 14 Prometheus metrics monitored by the engine.
2. promql/engine.go:138 (interface QueryLogger): Embedding slog.Handler and io.Closer.
3. promql/engine.go:302 (interface QueryTracker): Concurrency gatekeeper controlling Insert and Delete.
4. promql/engine.go:367 (struct Engine): The seed struct definition with its 16 fields.
5. promql/parser/parse.go:65 (interface Parser): The expression parser interface pulled directly across package boundaries.

Everything else—thousands of lines of query evaluation loops, matrix sort algorithms, subquery iterators—was stripped away. The agent received exactly 2.4 KB of pristine, typed context instead of 211.5 KB.

---

6. Semantic Navigation & Cross-Package Call Graphs

Grep is notoriously unreliable in Go due to common short names (Engine, Query, Config, Run). prod-code resolves symbols through language server indices on the remote node:

Symbol Definition & Hover

prod-code def --symbol github.com/prometheus/prometheus/promql::Engine
📍 Definition: file:///Users/alex09x/Documents/workspace/prometheus-eval/promql/engine.go:367:6

Hovering over promql/engine.go:367:6 returned the full struct declaration, documentation, and precise memory layout:

type Engine struct { // size=128 (0x80)
    logger                   *slog.Logger
    metrics                  *engineMetrics
    timeout                  time.Duration
    maxSamplesPerQuery       int
    activeQueryTracker       QueryTracker
    ...
}

Callers & Callees of NewEngine

Querying callers of NewEngine (line 387) uncovered 8 call sites spanning binary entrypoints, test suites, and mock frameworks:

• cmd/prometheus/main.go:1042 (main)
• cmd/promtool/unittest.go:650 (BenchmarkInfoFunction)
• promql/engine_test.go:2494 (TestParserConfigIsolation)
• promql/engine_test.go:4314 (TestEngine_Close)
• rules/manager_test.go:1215 (TestRuleMovedBetweenGroups)

Querying callees returned 15 distinct functions across the Go standard library (slog), external client libraries (client_golang), and internal utilities, tracking constructor initialization paths with exact line and column precision.

Implicit Interface Implementations

In Go, interfaces are satisfied implicitly without an implements keyword. How do you find every type in Prometheus that implements storage.Queryable?

prod-code impls storage/interface.go 108 6
Found 27 implementation(s):
  • cmd/prometheus/main.go:1845:6
  • storage/fanout.go:30:6 (fanout)
  • storage/remote/storage.go:54:6 (remote storage queryable)
  • tsdb/db.go:310:6 (local TSDB database)
  • tsdb/agent/db.go:256:6 (Prometheus agent mode TSDB)
  • web/api/testhelpers/mocks.go:56:6 (mock queryable)
  ... and 21 more.

In under 400 milliseconds, impls resolved the entire dynamic polymorphism graph of the Prometheus storage subsystem.

---

7. Refactoring Safety & The Refusal Contract

A hallmark of a disciplined code intelligence platform is knowing when not to edit code. We tested prod-code safe-delete against several targets in Prometheus to verify its refusal guarantees.

Deleting a Struct: Refused

prod-code safe-delete promql/engine.go 367 6
safe delete refused: Go safe delete refused; nothing was written:
the position names a non-function declaration, not an ordinary function or receiver method

Deleting an Exported Function: Refused

prod-code safe-delete promql/engine.go 387 6
safe delete refused: Go safe delete refused; nothing was written:
NewEngine is not a supported unexported ASCII Go function name

Deleting an Unexported Function with Call Sites: Refused

We targeted contextDone(ctx, env) at promql/engine.go:278, an unexported helper:

prod-code safe-delete promql/engine.go 278 6
safe delete refused: Go safe delete refused; nothing was written:
contextDone is still referenced at promql/engine.go:776:12, promql/engine.go:930:12,
promql/engine.go:1565:13, promql/engine.go:1742:13, promql/engine.go:1919:13,
promql/engine.go:2134:12, promql/engine.go:2339:14, promql/engine.go:2947:13,
promql/info.go:459:13; no function was deleted

The tool inspected the whole repository graph, identified all 9 active call sites across engine.go and info.go, and refused to touch a single byte on disk.

---

8. Pre-Validation with Remote Compiler Shadowing

Before writing a change to disk, prod-code validate verifies proposed edits in a private RAM overlay on the build node. When combined with --compile, it runs the real Go compiler in a shadow copy to catch subtle type errors before files are ever touched.

We injected an intentional type mismatch into timeMilliseconds:

func timeMilliseconds(t time.Time) int64 {
-   return t.UnixNano() / int64(time.Millisecond/time.Nanosecond)
+   return "broken_string"
}

We submitted the proposal via prod-code validate promql/engine.go --compile:

Running the remote compiler check for the proposed changes...
1 file(s) checked together: 1 error(s), 0 warning(s)
promql/engine.go: 1 error(s), 0 warning(s)
  error: cannot use "broken_string" (untyped string constant) as int64 value in return statement [IncompatibleAssign] (promql/engine.go:803:9)

compiler: `go build ./...` on the proposed text: 1 error(s)
  error: cannot use "broken_string" (untyped string constant) as int64 value in return statement (promql/engine.go:803:9)

In 1.8 seconds, both gopls and go build ./... caught the type mismatch in shadow memory. The working tree remained 100% clean.

---

9. Remote Execution & The Upstream go vet Flag

Running tests and linters locally on a laptop is slow and battery-intensive. With prod-code exec, commands run on ram9 with high parallel throughput.

Running Engine Tests

prod-code exec -- go test -v -run "TestQueryConcurrency|TestQueryTimeout" ./promql
=== RUN   TestQueryConcurrency
--- PASS: TestQueryConcurrency (0.04s)
=== RUN   TestQueryTimeout
--- PASS: TestQueryTimeout (0.10s)
PASS
ok  	github.com/prometheus/prometheus/promql	0.147s
[prod-code exec] exit 0 in 0.9s (server 0.8s, cpu 0.9s user 0.4s sys, peak 414 MB)

The test suite executed in 147 ms, with complete process lifecycle handled over LAN in 900 ms.

Running go vet

When running go vet ./promql/... through prod-code exec, the compiler analyzer returned an unexpected warning on upstream Prometheus code:

promql/histogram_stats_iterator.go:66:36: method Seek(t int64) chunkenc.ValueType should have signature Seek(int64, int) (int64, error)
promql/value.go:495:35: method Seek(t int64) chunkenc.ValueType should have signature Seek(int64, int) (int64, error)
promql/histogram_stats_iterator_test.go:224:27: method Seek(int64) chunkenc.ValueType should have signature Seek(int64, int) (int64, error)
[prod-code exec] exit 1 in 1.6s

go vet’s stdmethods analyzer flagged Prometheus’s custom Seek iterator method because it shares a name with the standard io.Seeker interface but uses a specialized chunk-encoding signature (Seek(int64) chunkenc.ValueType).

---

Summary Evaluation Matrix

Category	Tool / Protocol	Target / Query	Latency / Result
Ingestion	Manifest Differential Sync	998 files (385K lines of Go)	14.1s cold sync, 0% laptop CPU
Architecture	code_dependencies	84 packages, 524 edges	0 circular package cycles
Foundations	Afferent Coupling (Cₐ)	discovery/targetgroup, labels	Cₐ = 35, I = 0.000
Bug Discovered	Issue #740 (Fixed)	Go import parser cycle storm	Fixed in 3be3cdd
AST Search	code_structural_search	if err != nil { return $x }	1,485 matches in 3.71s
Context Slicing	code_slice	promql.Engine (line 367)	99% context reduction (2.4 KB)
Polymorphism	code_implementations	storage.Queryable interface	27 implementations resolved in 380ms
Call Graph	code_callers / callees	promql.NewEngine	8 callers, 15 callees
Safety	code_safe_delete	Engine, NewEngine, contextDone	Refusal contracts verified across 9 call sites
Pre-Validation	code_validate_edit	Shadow RAM compile	Type error rejected in 1.8s, 0 disk writes
Remote Test	code_exec	TestQueryConcurrency	PASS in 147ms on 32-core node

---

Takeaways for AI Coding Agents

Dissecting Prometheus reveals why language-aware, cluster-backed code intelligence is essential for AI engineering:

1. Clean Architecture Scales: Prometheus’s strict adherence to Go’s acyclic package rules (84 packages, 0 cycles) makes its dependency graph exceptionally stable. Foundational packages have zero efferent coupling.
2. Context Slicing Beats Raw Prompts: Feeding an LLM a 4,600-line Go file burns tokens and invites errors. A 99% context reduction via code_slice yields the exact 5 declarations needed to understand promql.Engine in 2.4 KB.
3. Pre-Validation Prevents Broken Commits: Testing patches in remote shadow RAM (validate --compile) catches syntax, type, and import regressions before a single file is modified on disk.
4. Dogfooding Surfaces Hidden Edge Cases: Benchmarking against a real 385K-line Go repository exposed Bug #740 in our import parser, which was diagnosed, fixed, tested, and shipped in minutes.

The entire evaluation was conducted over LAN on node ram9 with 0% local laptop CPU utilization.

Alexander Panasenko (2026-09-30). 1,485 Error Checks and Zero Import Cycles: Dissecting Prometheus's 84-Package Engine with 67 AST Tools. https://prod.codes/blog/1485-error-checks-and-zero-cycles-inside-prometheus/