rapidfire: the channel between a market feed and a decision

An exchange publishes an order-book update over a WebSocket. A reader task decrypts the frame, parses it, stamps arrival, and hands it to the strategy thread. The strategy evaluates its model, and an order goes to the gateway. In high-frequency trading (HFT)—reacting to market updates on microsecond timescales—that sequence is the product: when participants react to the same event, execution latency can affect queue priority and fill likelihood at the matching engine.

Physical transit, matching engines, and TLS parsing dominate live elapsed time. Benchmark reader tasks model WebSocket ingestion without live network jitter. The thread hand-off between reader tasks and strategy loop is different: it is entirely in-process, runs on every message, and a slow channel burns a visible slice of the budget.

We wrote our own channel for this boundary. It is open source as rapidfire:

[dependencies]
rapidfire = "0.1"

Our rules focus strictly on the hot path: no locks, no steady-state allocation, and no syscalls while messages flow. The consumer parks only when the queue is empty, paying for a waker only then. Producer tail and consumer head are separately padded to 128 bytes to eliminate false sharing.

What the queue looks like

The channel is an intrusive linked list of blocks. Each holds 63 value slots and a 64th sentinel slot marking the next block. Slots carry a state word tagged with a lap counter so recycled slots are never mistaken for written ones.

Producer tail and consumer head are separately padded to 128 bytes; each block also keeps reader-completion marks in a separately padded header. An uncontended unbounded producer reserves a slot with fetch_add; under contention this path uses CAS retries with backoff, which reduced adjacent-slot contention in our measurements. Bounded senders reserve with a capacity check and CAS. After reservation, the producer writes the value and publishes the slot state with a release store. A consumer checks that state before CAS-advancing the head, so it never claims an unwritten slot. Blocks stay allocated while the channel is live and return to a spare slot or pool only after their readers finish. Recycling retains high-water memory until the channel is dropped.

The async layer is a mutex-protected waker list touched only when parking. To avoid SeqCst fences on the hot path, parking consumers and active producers synchronize through acquire-release sequences: the parking side performs fetch_add(0, AcqRel) on the tail index, forcing its waker registration into a total order with producer progress; the sender then performs a relaxed load of the separate waiter count after its claim. Every index mutation follows this protocol. Loom caught a bug early when a plain store on the head broke synchronization.

What broke on the way

Pre-release drafts passed on workstations but segfaulted on a 128-core Neoverse-N1: a lagging block walker followed a prev link into a recycled block whose links were cleared. (No version existed before 0.1.0; these were pre-release rewrite bugs.) The fix: a block’s start index is committed last with a release store, links are never nulled on recycle, and walkers validate each block by its start.

The second failure on the N1 was an MPMC deadlock in early drafts. Consumers previously claimed slots before verifying they were written. Under load, the head bypassed an unwritten slot, preceding blocks were recycled, and a producer following prev links backwards from the tail was stranded. We removed claim-first consumers; receivers now verify slot readiness before claiming, yielding if busy.

Two issues came from a delegated review task in an isolated !prod worktree. The reviewer wrote targeted probes and found a lost wake-up: a recv() future cancelled immediately after selection by notify_one took the notification with it. Dropping a pending receiver now forwards that already-selected notification to the next waiter. The review also flagged the send-versus-close race shared with tokio and async-channel; we documented it rather than paying for a fence. Finally, throttling bounded head re-reads was reverted because try_send returned Full on a non-full queue: exactness won.

What the sampler said about assembly

We expected to write inline assembly. On Zen 4, unbounded try_send compiles to one lock xadd (or lock cmpxchg under contention) and about twenty instructions. On AArch64, it emits one LSE atomic (ldaddal), ldapr loads, and a single stlr. Three publish variants were tested (only one uses inline asm: dmb ishst + store, alongside fence(Release) + store and an atomic RMW publish): all were slower than the compiler’s stlr. They remain in the tree behind --cfg switches as evidence; the default hot path contains no inline assembly.

perf stat on Ryzen 9 7950X confirmed where cycles go. In SPSC, rapidfire and crossbeam SegQueue miss L1 data cache equally often (2.3 million misses per 2 million messages) because slot lines cross cores either way. rapidfire is faster because it executes 43 percent fewer instructions (213 million versus 372 million) and half the locked operations. On the N1, 86 percent of producer samples sit on the instruction after ldaddal, stalled on cross-core invalidation. In 4x4 MPMC, about three quarters of samples sit in pause and isb back-off loops from index contention. Hardware profiles found no compiler code-generation reason for assembly; physical coherence is the limit.

The numbers, and where we lose

Benchmark values denote amortized elapsed time per operation (total wall-clock time divided by message count) representing amortized throughput cost, not individual hand-off latency, network latency, or order latency. Harnesses were compiled with rustc 1.97.1 (-C target-cpu=native, lto = "fat"); the tables report medians of 5 runs using: crossbeam-queue 0.3.14, flume 0.11.1, async-channel 2.5.0, tokio 1.53.1. Complete benchmark tables across twelve test machines and perf profiles are in benches/RESULTS.md.

These are the original 0.1.0 measurements, retained as a historical snapshot. The raw harness added a Tokio-only receive mutex and a shared per-message completion counter in MPMC tests, and the harnesses started timing after releasing workers. Pin lists assigned only the listed workers; additional workers were unpinned. Later corrections to the harness mean the original ratios should not be interpreted as measurements from the corrected setup. See the pinned raw_bench.rs and real_case.rs.

In raw unbounded SPSC on a Ryzen 9 7950X, rapidfire records 5.4 ns per message (crossbeam SegQueue 14.5 ns, std::sync::mpsc 14.8 ns, async-channel 40.7 ns, flume 104.7 ns, tokio::sync::mpsc 106.0 ns).

For multi-producer fan-in, 40 asynchronous WebSocket readers fed one consumer on a Ryzen 9 7900 running Tokio with 4 workers.

Across 64, 256, and 1024-byte payloads, rapidfire scales from 43 ns to 127 ns per message, maintaining lower amortized cost than competitors as payload sizes increase.

The losses are in the tables too. On unpinned Apple M3 Pro, SegQueue leads in unbounded SPSC (7.6 ns versus rapidfire’s 8.6 ns) and ArrayQueue wins bounded SPSC (6.1 ns versus 11.6 ns). Bounded channels under contention poll the consumer head line, causing transfers where ArrayQueue polls its slot. In 8-producer runs, a short pin list does not establish placement for every worker. When consumers park frequently on large payloads, async-channel leads because rapidfire guards waiters with a mutex.

The rule we would give anyone doing this: benchmark the topology you run, on the hardware you run it on, and let the sampler decide whether assembly is the next step. For us it said no, twice.