A Benchmark Smelled Funny

Mauro Pinto — Sun, 19 Jul 2026 00:00:00 +0000
I've been building go-spmc-ring</a>, a single-writer, multiple-reader ring buffer in Go. The whole point of the project is mechanical sympathy: every design decision is supposed to have a benchmark behind it. I came across a benchmark that I didn't quite understand, and went digging deeper.</p>
The accusation #</span></a> </h2>
The ring buffer has two ways to publish an event. You can hand it a value:</p>
rb.</span>Publish</span>(event)</span></span></code></pre>
Or you can hand it a callback that fills the destination slot in place:</p>
rb.</span>PublishFunc</span>(</span>func</span>(</span>slot</span> *</span>object</span>) {</span></span>
    slot.x[</span>0</span>]</span> =</span> '</span>0</span>'</span></span>
})</span></span></code></pre>
My payload is object{ x [64]byte }</code>, which is exactly one cache line. And my benchmark table said this:</p>
Benchmark</th> ns/op</th> Notes</th></tr></thead>

Publish</code></td> 5.19</td> callback API, keep-up reader</td></tr>
Publish_Direct</code></td> 9.21</td> constructs and publishes a value/event</td></tr>
</tbody></table>
There it is. Publishing by value is 1.77x slower</strong>. The obvious lesson writes itself: don't pass structs by value in
a hot loop, use the callback.</em></p>
Something nagged, though. 1.77x is a lot</em> for one struct copy. 64 bytes is four 16-byte SSE moves, or one AVX-512 store
on a good day. It should be nearly free. I decided to take it apart.</p>
Four benchmarks, one variable at a time#</span></a>
</h2>
Instead of trusting one row that changes several things at once, I wrote four benchmarks that each change exactly one
thing. (They live in a tiny standalone repo</a> so you can run them
yourself. No ring buffer required.) Every one writes a different</em> value each iteration, so none of them can be
waved away as a constant the compiler folded flat:</p>
// build a fresh value every iteration, then publish by value</span></span>
func</span> BenchmarkConstructed</span>(</span>b</span> *</span>testing</span>.</span>B</span>) {</span></span>
    var</span> seq</span> byte</span></span>
    for</span> b.</span>Loop</span>() {</span></span>
        o</span> :=</span> </span>object</span>{}</span></span>
        o.x[</span>0</span>]</span> =</span> seq</span></span>
        publishByValue</span>(o)</span></span>
        seq</span>++</span></span>
    }</span></span>
}</span></span>
</span>
// publish by value from buffers built AHEAD of the loop</span></span>
func</span> BenchmarkPrepared</span>(</span>b</span> *</span>testing</span>.</span>B</span>) {</span></span>
    var</span> bufs [</span>4</span>]</span>object</span></span>
    for</span> i</span> := range</span> bufs {</span></span>
        bufs[i].x[</span>0</span>]</span> =</span> byte</span>(i)</span></span>
    }</span></span>
    var</span> i</span> int</span></span>
    for</span> b.</span>Loop</span>() {</span></span>
        publishByValue</span>(bufs[i</span>&</span>3</span>])</span> // value changes, but it was built long ago</span></span>
        i</span>++</span></span>
    }</span></span>
}</span></span>
</span>
// fill all 64 bytes of the slot in place, via callback</span></span>
func</span> BenchmarkFullWrite</span>(</span>b</span> *</span>testing</span>.</span>B</span>) {</span> /* ... */</span> }</span></span>
</span>
// write a single changing byte into the slot, via callback</span></span>
func</span> BenchmarkOneByteWrite</span>(</span>b</span> *</span>testing</span>.</span>B</span>) {</span> /* ... */</span> }</span></span></code></pre>
Stripped of the ring buffer's own overhead, the gap is stark. On my Ryzen 5 9600X:</p>
BenchmarkConstructed-12     5.51 ns/op</span></span>
BenchmarkPrepared-12        1.69 ns/op</span></span>
BenchmarkFullWrite-12       1.61 ns/op</span></span>
BenchmarkOneByteWrite-12    1.53 ns/op</span></span></code></pre>
Read that table and the story flips completely.</p>
Payload size is free.</strong> FullWrite</code> fills all 64 bytes; OneByteWrite</code> writes one. They're basically the same
(1.61 vs 1.53 ns). Writing the whole cache line costs almost nothing over writing a single byte, because what
you're really paying for is the cache line changing hands, not the bytes going into it. So the gap was never
"we copy 64 bytes."</p>
The by-value API is cheap too.</strong> Prepared</code> passes a 64-byte struct by value, exactly like Constructed</code> does
(and it runs at 1.69 ns), right down near the floor. Passing a struct by value, when the value was built
ahead of time, is not the villain either.</p>
So what is Constructed</code> doing that the others aren't? Only one thing: it builds the value in place, right
before the call</em>. That construction (o := object{}; o.x[0] = seq</code>), is the entire difference between 1.69 ns
and 5.51 ns.</p>
The original "1.77x API penalty" was a lie of composition. The honest by-value cost is a rounding error. Everything
else was the caller's construction pattern, and the benchmark had quietly folded the two together.</p>
So why does building it in place</em> cost 3x?#</span></a>
</h2>
This is where you have to stop reading Go and start reading assembly. go test -gcflags=-S</code> on Constructed</code> shows
exactly what the CPU is being asked to do:</p>
; o := object{}; o.x[0] = seq</span></span>
MOVUPS</span> X15, (</span>CX</span>)        </span>; 16-byte zeroing store</span></span>
MOVUPS</span> X15, </span>16</span>(</span>CX</span>)      </span>; 16-byte zeroing store</span></span>
MOVUPS</span> X15, </span>32</span>(</span>CX</span>)      </span>; 16-byte zeroing store</span></span>
MOVUPS</span> X15, </span>48</span>(</span>CX</span>)      </span>; 16-byte zeroing store</span></span>
MOVB   </span>DL</span>, o+</span>80</span>(</span>SP</span>)     </span>; 1-byte store        <- note the size</span></span>
</span>
; publishByValue(o): copy the argument out</span></span>
MOVUPS</span> (</span>CX</span>), X14        </span>; 16-byte LOAD over the bytes just written</span></span>
MOVUPS</span> X14, (</span>DX</span>)</span></span></code></pre>
Zeroing the struct is four 16-byte stores. Setting the first byte is one 1-byte store. Then, to pass o</code> by value,
the argument copy immediately reads those bytes back</em>, starting with a 16-byte load that overlaps the first
16-byte zeroing store and</strong> the 1-byte store.</p>
And this is the crux.</p>

Stores don't go to the cache. Not at first</strong>: When the CPU executes a store, it does not write to L1. It appends
an entry to a small hardware queue called the store buffer (~64 entries on Zen): "address X, N bytes, this data."
The entry sits there until the instruction retires, and only then drains to L1, quietly, in the background. Why?
Speculation. The CPU executes ahead of branches it has guessed. If the guess was wrong, it must throw work away,
and you can't "unwrite" the cache. The store buffer is the holding pen for writes that aren't yet official.</li>
That creates a problem for every load</strong>: If pending writes live in the store buffer and not in the cache,
then the cache is stale by definition. So every load must check: "is there a store buffer entry for my
address, newer than me?" If yes, reading L1 would return old data.</li>
The fast path: forwarding</strong>: If a single pending store entry fully contains the load's bytes, the hardware
just copies the data straight out of that entry into the load. No cache involved, done in a few cycles. This
is store-to-load forwarding, and it's why x = 5; y = x costs nothing. AMD's own
optimization guide</a> states the covering rule plainly: the
load-store unit forwards "when there is an older store that contains all of the load's bytes</strong>"
(Software Optimization Guide for the AMD Zen5 Microarchitecture, #58455, ch. 2, Load-Store Unit).</li>
The slow path: our stall</strong>: Now the crux. Each store buffer entry is one packet: (addr, size, data). The
forwarding hardware is a matcher, not a merger: it can grab data from one entry, but it has no ALU to
stitch bytes from entry #1 together with bytes from entry #5, in age order, in the load's shadow.



So when our 16-byte load arrives and finds byte 0's newest value in the MOVB entry and bytes 1–15's newest value
in the older MOVUPS entry, the matcher shrugs: no single entry covers me. The only correct fallback is to make
the cache authoritative again: wait for those entries to retire and drain to L1, then read L1 normally. That
wait, pipeline twiddling its thumbs while stores age out, is the 15–20 cycles
(Agner Fog</a> measures ~12 for the partial-overlap case
on Zen 5, §25.17 of the microarchitecture manual; the end-to-end gap here works out to ~20 cycles per iteration).



That's the whole thing. Not a cache miss, not memory bandwidth. A pattern-matching limitation in a queue-lookup
circuit, and the penalty is "wait until the queue drains." (For a perf-counter walkthrough of the same
narrow-write-then-wide-read shape on Intel, see
Store forwarding by example</a> on Easyperf.)</li>
</ol>
Here's the collision, drawn out. Building the value leaves several stores sitting in the store buffer, all still
in flight (newest at the bottom, in program order):</p>
  store buffer (pending writes to o on the stack)</span></span>
  ┌────────────────────────────────────────────────┐</span></span>
  │  MOVUPS X15  ->  bytes [ 0 .. 15]  (16 bytes)  │  ◄ zeroes byte 0</span></span>
  │  MOVUPS X15  ->  bytes [16 .. 31]  (16 bytes)  │</span></span>
  │  MOVUPS X15  ->  bytes [32 .. 47]  (16 bytes)  │</span></span>
  │  MOVUPS X15  ->  bytes [48 .. 63]  (16 bytes)  │</span></span>
  │  MOVB   seq  ->  byte [0]          (1 byte)    │  ◄ overwrites byte 0 (newest)</span></span>
  └────────────────────────────────────────────────┘</span></span>
</span>
Then the argument copy immediately issues:  LOAD bytes [0 .. 15].</span></span>
Draw every pending store and the load as spans over the same byte axis:</span></span>
</span>
           byte offset within o</span></span>
           0               15 16          31 32   ...</span></span>
           |----------------|----------------|--------</span></span>
</span>
  pending  ################                     MOVUPS zero [ 0..15]</span></span>
  stores                    ################     MOVUPS zero [16..31]</span></span>
                                          ...    (two more, [32..63])</span></span>
           #                                     MOVB seq -> [0] <- NEWEST</span></span>
</span>
  the load ================                     LOAD [ 0..15]</span></span>
           ^</span></span>
           byte 0's newest data lives in the MOVB entry,</span></span>
           bytes 1..15's newest data in the first MOVUPS:</span></span>
           no single store spans the load window</span></span>
           ->  cannot forward  ->  stall</span></span></code></pre>
The 1-byte store runs last</em>, so at load time byte 0's freshest value lives in the MOVB</code> while bytes 1..15 live in
the earlier 16-byte MOVUPS</code>. The load wants all 16 as a unit, but its bytes have two different most-recent
writers. No single pending store covers the read, so the core can't shortcut it. It stalls until those writes
land in L1, then reads them back the slow way.</p>
Now compare Prepared</code>. Its loop assembly is almost identical</em>, it still does the double copy, the value into the
argument area and then into the slot. The one difference is where the construction lives:</p>
; buffers built ONCE, above the loop:</span></span>
MOVUPS</span> X15, (</span>CX</span>)</span></span>
...                 </span>; the stores that fill bufs</span></span>
</span>
loop</span>:</span></span>
MOVUPS</span> (</span>CX</span>), X14    </span>; same 16-byte readback...</span></span>
MOVUPS</span> X14, (</span>DX</span>)    </span>; ...but the stores that wrote CX retired long ago</span></span>
...</span></span></code></pre>
Same instructions, same straddling load. But the stores that built those buffers ran before</em> the loop. They retired
and drained to L1 long before the first readback. Draw the same moment and the collision is simply gone:</p>
  store buffer (writes to bufs already retired)</span></span>
  ┌──────────────────────────────┐</span></span>
  │           (empty)            │   <- nothing pending</span></span>
  └──────────────────────────────┘</span></span>
</span>
  L1 cache (bufs live here, settled)</span></span>
  ┌──────────────────────────────┐</span></span>
  │ bytes [0 .. 63] of bufs[i&3] │</span></span>
  └──────────────────────────────┘</span></span>
           ▲</span></span>
           │  LOAD bytes [0 .. 15]</span></span>
           └──  no pending store to reconcile</span></span>
               served straight from L1  ->  no stall</span></span></code></pre>
By the time the copy runs a million times, there is nothing fresh in the store buffer for it to trip over, so the load
is served from cache at full speed.</p>
That's the whole difference between 5.5 ns and 1.7 ns: not the copy, not the API, just when</em> the stores that feed the
copy were issued. Same load instruction, same bytes, the only thing that changed is whether a half-finished store
was still sitting under it.</p>
The benchmark, in other words, was never measuring the Publish</code> API. It was measuring a construct-then-copy
anti-pattern</em>, one that happens to be really easy to write, which is exactly why the number still matters
even though the blame was in the wrong place.</p>
Can't the library just fix this?#</span></a>
</h2>
The stall doesn't happen in the ring buffer's memory. It happens in the caller's stack frame, before the value ever
reaches the ring.</strong> Here's Publish</code>:</p>
func</span> (</span>r </span>*</span>RingBuffer</span>[</span>T</span>])</span> Publish</span>(</span>payload</span> </span>T</span>) {</span></span>
    ...</span></span>
    r.buffer[nextSequence</span>&</span>r.mask]</span> =</span> payload</span>   // copy the value INTO the slot</span></span>
    ...</span></span>
}</span></span></code></pre>
By the time that line runs, it's already over. Trace the caller instead:</p>
o</span> :=</span> </span>object</span>{}</span>       // zeroing stores  -> caller's stack</span></span>
o.x[</span>0</span>]</span> =</span> seq</span>        // 1-byte store    -> caller's stack</span></span>
rb.</span>Publish</span>(o)</span>       // argument copy: read o back off the stack  <- the stall</span></span></code></pre>
The straddling load is the argument copy</em>, the CPU reading o</code> back out of your stack to pass it by value.
Both the stores and the load live in your frame, and they all finish before the ring buffer's own store
into the slot even executes. The slot is just the innocent final destination. You cannot fix a stall
in someone else's stack frame by rearranging your data structure.</p>
But the library does</em> have a structural answer. The sibling method hands you a pointer straight to the slot:</p>
func</span> (</span>r </span>*</span>RingBuffer</span>[</span>T</span>])</span> PublishFunc</span>(</span>f</span> func</span>(</span>*</span>T</span>)) {</span></span>
    ...</span></span>
    f</span>(</span>&</span>r.buffer[nextSequence</span>&</span>r.mask])</span>   // you fill the slot in place</span></span>
    ...</span></span>
}</span></span></code></pre>
There's no intermediate o</code>, so there's nothing to copy and nothing to read back. You build the event where it will
live.</p>
What to actually do in production#</span></a>
</h2>
Strip away the microbenchmark scaffolding, the rotating buffers were only ever a way to simulate</em> "a value whose
stores already retired", and the guidance is a three-line decision:</p>


Constructing the event at publish time?</strong> Fill it in the slot. Use PublishFunc</code> for one, PublishBatchFunc</code> for
many (which also amortizes the cursor store across the batch):</p>
rb.</span>PublishFunc</span>(</span>func</span>(</span>slot</span> *</span>object</span>) {</span></span>
    *</span>slot</span> =</span> </span>object</span>{}</span>   // or just overwrite every field</span></span>
    slot.x[</span>0</span>]</span> =</span> seq</span></span>
})</span></span></code></pre>
No argument copy, no stall, and it stays fast even when every field differs every time (FullWrite</code>, 1.61 ns).</p>
</li>

Already holding a finished value</strong>: one that arrived from a channel, a decoded frame, a slice you're forwarding?
Then Publish(v)</code> is fine. Passing an existing value by value was never the stall; that's exactly what Prepared</code>
measured, and it ran near the floor. The trap is construct-then-copy in the same breath</em>, not by-value itself.</p>
for</span> ev</span> := range</span> incoming {</span> // ev built elsewhere, earlier</span></span>
    rb.</span>Publish</span>(ev)</span>         // no fresh stores under the copy</span></span>
}</span></span></code></pre></li>

Never</strong> write o := T{}; fill(o); rb.Publish(o)</code> back to back in a hot loop. That one shape pays the full stall,
and no API change can save it.</p>
</li>
</ul>
One more twist: your CPU may disagree#</span></a>
</h2>
Running on an Apple M4 did not render the same results. On arm64 the gap mostly evaporates, the by-value construction
path is roughly on par with the rest, and in go-spmc-ring's full benchmark it's actually the faster</em> option.
The store-forwarding stall that dominates on x86 simply doesn't reproduce there. (Apple silicon has its own,
entirely different stall lurking in that code, but that's a story for another post.)</p>

The full standalone reproduction (four benchmarks, the disassembly, and the arm64 note) is on GitHub</a>. It runs in about 15 seconds and needs nothing but a Go toolchain. Clone it and watch your own CPU stall.</em></p>


go-spmc-ring
Mauro Pinto — Tue, 30 Jun 2026 00:00:00 +0000
A single-writer, multiple-reader (SPMC) ring buffer in Go, inspired by the LMAX Disruptor pattern. It focuses on
cache-line awareness, false-sharing avoidance, and lock-free reader lifecycle management: unlike the classic
Disruptor (where the consumer graph is wired once before the first event) readers here join and leave a
live stream at runtime without locks, allocations, or writer stalls.</strong></p>
This is useful for runtimes and actor model-like systems where you need some entering and reentering readers.</p>
We also preview the compatibility and usage with the preview SIMD package in Go 1.26, which allows for vectorized
operations on the ring buffer.</p>
On a Ryzen 5 9600X (Linux, Go 1.26) a single publish takes ~5.2 ns (≈193 M ops/s), batched publishes reach ~2.1 ns/item
(≈476 M items/s), and eight concurrent readers cost the writer less than 7%.</p>
Check the repo</a>.</p>
Mauro Pinto - go

A Benchmark Smelled Funny

go-spmc-ring
