Apache Pekko Under the Microscope: What 67 Remote AST Tools Found Inside the High-Throughput Actor Runtime

Historical Apache Pekko actor runtime analysis with prod-code: Scala and Java source counts, 41 extracted SBT modules, 3.5% reported clone density, and a remote Metals refactoring example.

High-throughput concurrent systems built on the Actor model demand careful static analysis. Apache Pekko is a Java and Scala framework for concurrent, distributed applications, forked from Akka 2.6.x. Its official documentation describes actors, streams, clustering, and persistence. These capabilities do not imply a universal dispatch latency.

SBT compilation, macro expansion, and background typechecking can make actor platforms costly to index. We used selected operations from prod-code’s 67-tool suite against a Pekko checkout, running AST traversal, dependency discovery, clone analysis, structural search, and Metals diagnostics remotely. The local client still handles synchronization, requests, and results.

The command excerpts below are historical observations retained from the original publication. No pinned upstream commit or complete measurement environment is recorded here, so counts, paths, line numbers, and scan durations are snapshot-specific and have not been remeasured for this update. The excerpts cover selected operations from the 67-tool suite, not verification of every tool. A cycle-free extracted module graph does not establish all source-level or external dependency relationships; clone counts do not establish runtime performance. Analyzer diagnostics are not a compiler build or test result.

$ git ls-files '*.scala' | wc -l
2797
$ git ls-files '*.java' | wc -l
587
$ git ls-files | wc -l
4365
$ git ls-files '*.scala' | xargs wc -l | grep -v 'total$' | awk '{s+=$1} END {print s}'
560022
$ git ls-files '*.java' | xargs wc -l | grep -v 'total$' | awk '{s+=$1} END {print s}'
278337

The captured inventory reports 560,022 Scala lines and 278,337 Java lines, totaling 838,359 lines across 3,384 source files. It does not demonstrate exhaustive semantic coverage or real-time performance.

Architecture, Multi-Module SBT Hierarchy, and Actor Subsystem Topology

In distributed actor runtimes, module coupling must follow strict isolation boundaries. If the core actor model accidentally depends on clustering transport, or if the streaming engine leaks persistence semantics, runtime isolation collapses. We analyzed the project structure across Pekko’s 41 SBT subprojects.

$ prod-code dependencies
⚡ prod-code Architecture & Dependency Graph Report
────────────────────────────────────────────────────
Scope: modules | Nodes: 41 | Dependencies: 58

✓ Zero circular dependencies detected. Architecture graph is a clean DAG.

Top Coupled Modules (by Afferent Coupling Ca):
  Name                                Ca    Ce  Instab
  ────────────────────────────────────────────────────
  actor                               34     0    0.00
  testkit                             18     1    0.05
  stream                              14     1    0.07
  actor-typed                         12     2    0.14
  cluster                              9     2    0.18
  coordination                         6     0    0.00
  remote                               5     3    0.38
  distributed-data                     4     2    0.33
  pki                                  3     1    0.25
  persistence                          4     2    0.33
  cluster-tools                        3     1    0.25
  cluster-sharding                     3     2    0.40
  persistence-typed                    2     2    0.50
  cluster-typed                        2     3    0.60
  bench-jmh                            0     8    1.00
  docs                                 0    19    1.00

Isolated Leaf Endpoints: bench-jmh, docs, bill-of-materials

The captured scan reports 41 subprojects with no cycles in its extracted module graph ($C = 0$):

The Pure Actor Foundation: actor sits at the root of the hierarchy with an Afferent Coupling of $C_a = 34$, Efferent Coupling $C_e = 0$, and an Instability metric of $I = 0.00$. It contains zero outbound dependencies on any other Pekko module, exporting core abstractions (Actor, ActorRef, Props, ActorSystem, Mailbox, Scheduler) that all downstream libraries consume.
Asynchronous Reactive Streams: stream ($C_a = 14, C_e = 1, I = 0.07$) builds directly upon actor while exposing the Reactive Streams implementation (Source, Flow, Sink, GraphStage). It remains completely decoupled from remote transport and clustering.
Typed Actor Evolution: actor-typed ($C_a = 12, C_e = 2, I = 0.14$) wraps classic actor primitives into a compile-time type-safe protocol layer (Behavior[T]), providing typed message protocols; this is not a guarantee about every runtime message or serialization boundary.
Remoting, Clustering, and Coordination: remote ($C_a = 5, C_e = 3, I = 0.38$) depends on actor, stream, and pki (TLS certificate validation). cluster ($C_a = 9, C_e = 2, I = 0.18$) builds upon remote and coordination ($C_a = 6, C_e = 0, I = 0.00$) to provide gossip-based node discovery, leader election, and failure detection.
Leaf Aggregators: Benchmark harnesses (bench-jmh) and documentation modules (docs) sit at the perimeter of the DAG ($C_a = 0, I = 1.00$), which describes the captured module edges, not a guarantee about artifact contents.

Code Duplication, Artery Wire Protocols, and Protobuf Serialization Clones

Actor systems rely on binary wire serialization to transmit messages across network boundaries. In Pekko, high-performance serialization is handled by Google Protocol Buffers and the Artery remoting transport. We ran prod-code duplicates across the full codebase to isolate duplication patterns.

$ prod-code duplicates
Files Scanned: 3384 | Lines: 838359 | Clone Groups: 20 | Duplication: 3.5%

Discovered Clone Groups:

[Clone Group #22321] 6 lines | 424 occurrences (Type-2 (Parameterized))
  • Occurrence 1: remote-tests/src/test/java/org/apache/pekko/remote/artery/protobuf/TestMessages.java:593-598
  • Occurrence 2: cluster/src/main/java/org/apache/pekko/cluster/protobuf/msg/ClusterMessages.java:711-716
  • Occurrence 3: cluster-sharding/src/main/java/org/apache/pekko/cluster/sharding/protobuf/msg/ClusterShardingMessages.java:6456-6461
  Preview:
    │     @java.lang.Override
    │     public boolean equals(final java.lang.Object obj) {
    │       if (obj == this) {
    │        return true;
    │       }

[Clone Group #19804] 10 lines | 312 occurrences (Type-2 (Parameterized))
  • Occurrence 1: remote/src/main/java/org/apache/pekko/remote/ContainerFormats.java:492-501
  • Occurrence 2: cluster/src/main/java/org/apache/pekko/cluster/protobuf/msg/ClusterMessages.java:1602-1611
  • Occurrence 3: cluster-tools/src/main/java/org/apache/pekko/cluster/pubsub/protobuf/msg/DistributedPubSubMessages.java:573-582

Across 838,359 lines scanned, the duplicate harvester identified 20 clone groups accounting for 3.5% overall duplication. The displayed clone groups contain generated wire-protocol examples:

Protobuf Message Equality and Hash Codes (Clone Group #22321): With 424 occurrences across remote, cluster, and cluster-sharding, these boilerplate sequences stem from generated Java Protobuf stubs (ClusterMessages.java, ContainerFormats.java). Each message format generates standard identity comparison guards (if (obj == this) return true;).
Wire Envelope Builders (Clone Group #19804): Encompasses 312 occurrences of serialization builder helpers that convert internal actor addresses, timestamps, and payload bytes into compact network frames.

Because these files are generated from .proto definitions during the build, they should be understood as generated output; clone density alone does not establish wire compatibility or runtime costs.

Structural AST Invariants, Mailbox Guards, and Receive Dispatch Patterns

Actor mailboxes and message dispatch loops cannot afford to swallow invalid states. We used prod-code structural-search to query internal assertions, preconditions, and message handling partial functions.

$ prod-code structural-search 'assert($A)'
220 match(es) in 55 file(s) (scanned in 1578.84ms)

  • actor/src/main/scala/org/apache/pekko/actor/ActorCell.scala:466:5             assert(msg.unlinked)
  • actor/src/main/scala/org/apache/pekko/actor/dungeon/FaultHandling.scala:113:7 assert(mailbox.isSuspended, "mailbox must be suspended")
  • actor/src/main/scala/org/apache/pekko/dispatch/Mailbox.scala:512:5           assert(message.unlinked)
  • actor/src/main/scala/org/apache/pekko/dispatch/sysmsg/SystemMessage.scala:100:5 assert(msg ne null)

$ prod-code structural-search 'require($A)'
435 match(es) in 175 file(s) (scanned in 2428.02ms)

  • actor/src/main/scala/org/apache/pekko/actor/ActorRef.scala:565:3              require(sender ne null, "DeadLetter sender may not be null")
  • actor/src/main/scala/org/apache/pekko/actor/Address.scala:116:5               require(!hasInvalidHostCharacters, "invalid host characters")
  • actor/src/main/scala/org/apache/pekko/actor/CoordinatedShutdown.scala:565:5   require(knownPhases(phase), "Unknown phase")

The search uncovered 220 internal invariant assertions (assert) and 435 public API precondition validations (require). In high-concurrency subsystems (ActorCell, Mailbox, FaultHandling), assertions check internal state, including whether a message is unlinked. Their presence alone does not establish memory visibility or prevent every queue corruption.

We also queried the idiomatic Scala actor message handler signature across the repository:

$ prod-code structural-search 'def receive: Receive = { case $A => $B }'
76 match(es) in 60 file(s) (scanned in 1403.93ms)

  • actor/src/main/scala/org/apache/pekko/event/Logging.scala:1157:14
  • actor/src/main/scala/org/apache/pekko/io/TcpListener.scala:98:3
  • actor/src/main/scala/org/apache/pekko/io/UdpListener.scala:78:3
  • actor/src/main/scala/org/apache/pekko/pattern/BackoffSupervisor.scala:36:5

In SystemMessage.scala, linked list operations used internal assertions to verify non-null message references. We tested a dry-run rule using prod-code codemod to propose precondition guards. Scala’s Predef API distinguishes assert from require: the exception type and assertion-elision behavior change, so this needs contract review:

$ prod-code codemod 'assert($A ne null) ==>> require($A ne null)'
`assert($A ne null) ==>> require($A ne null)`
8 changed line(s) in 2 file(s)

--- a/actor/src/main/scala/org/apache/pekko/dispatch/sysmsg/SystemMessage.scala
+++ b/actor/src/main/scala/org/apache/pekko/dispatch/sysmsg/SystemMessage.scala
@@ -98,5 +98,5 @@
    */
   final def ::(msg: SystemMessage): LatestFirstSystemMessageList = {
-    assert(msg ne null)
+    require(msg ne null)
     msg.next = head
     new LatestFirstSystemMessageList(msg)
--- a/actor-typed/src/main/scala/org/apache/pekko/actor/typed/internal/SystemMessage.scala
+++ b/actor-typed/src/main/scala/org/apache/pekko/actor/typed/internal/SystemMessage.scala
@@ -94,5 +94,5 @@
    */
   final def ::(msg: SystemMessage): LatestFirstSystemMessageList = {
-    assert(msg ne null)
+    require(msg ne null)
     msg.next = head
     new LatestFirstSystemMessageList(msg)

nothing was written; pass `apply: true` to make these edits

The codemod cleanly identified the system message list prepending methods across both classic and typed actor implementations, generating non-destructive diffs without local disk writes.

Remote Semantic Refactoring and Language Server Verification

Actor paths (pekko://System@10.0.0.1:2552/user/worker/$a) define hierarchical addressing across nodes. In actor/src/main/scala/org/apache/pekko/actor/ActorPath.scala, actor paths are parsed from URLs:

def fromString(s: String): ActorPath = s match {
  case ActorPathExtractor(address, elems) => RootActorPath(address) / elems
  case _                                  => throw new MalformedURLException("cannot parse as ActorPath: " + s)
}

We evaluated prod-code extract-function to isolate the root child path construction (RootActorPath(address) / elems) into a private factory method:

$ prod-code extract-function --to 73:78 --name createRootChild \
    actor/src/main/scala/org/apache/pekko/actor/ActorPath.scala 73 48
`fn createRootChild` extracted (actor/src/main/scala/org/apache/pekko/actor/ActorPath.scala); the selection now reads `this.createRootChild(address, elems)`
- no other place in the file has the selection's text

--- a/actor/src/main/scala/org/apache/pekko/actor/ActorPath.scala
+++ b/actor/src/main/scala/org/apache/pekko/actor/ActorPath.scala
@@ -72,5 +72,9 @@
   def fromString(s: String): ActorPath = s match {
-    case ActorPathExtractor(address, elems) => RootActorPath(address) / elems
+    case ActorPathExtractor(address, elems) => this.createRootChild(address, elems)
     case _                                  => throw new MalformedURLException("cannot parse as ActorPath: " + s)
   }
+  private def createRootChild(address: Address, elems: Iterable[String]): ActorPath = {
+    RootActorPath(address) / elems
+  }

the analyzer accepts the result: 0 errors

The captured Metals 1.6.9 response reports 0 errors for the proposed extraction. No Pekko compiler build or test result is included.

These examples place AST indexing, duplicate detection, and analyzer diagnostics on remote nodes. They do not measure instant feedback or local CPU usage.

A cycle-free module report and clean analyzer response are useful observations, but neither proves a refactoring correct.