Why does this x86_64 self-modifying loop work on bare metal / QEMU, but Segfaults only on native Linux under high CPU load?

[studioframes]

🏷️ Discussion Type

Question

💬 Feature/Topic Area

Assembly / x86_64 Architecture

Body

Hey everyone,

I'm writing a lightweight x86_64 JIT engine, and I ran into a bizarre hardware/kernel-level issue that I can't wrap my head around.

I have a small routine that dynamically patches its own immediate values inside a loop:

section .textglobal_start_start:movrbx,10 ; Loop counter.loop: ; Self-modifying instruction targetmovrax,0x1122334455667788patch_target equ $ -8 ; Modify the immediate value dynamically for next iterationadd qword [rel patch_target],1decrbxjnz .loop ; Exit syscallmovrax,60xorrdi,rdisyscall

The Problem:

1. I marked the .text segment as readable, writable, and executable (-z execstack / mprotect(PROT_READ | PROT_WRITE | PROT_EXEC)).
2. When running under QEMU or single-core VMs, it executes to completion perfectly.
3. On a native multi-core x86_64 Linux machine (Ubuntu 24.04), it randomly throws Segmentation fault (core dumped) or executes with stale data inside RAX on every ~5th run.

Is this an L1 Instruction Cache (i-cache) pipeline coherency issue where the CPU prefetches the unpatched instruction before add qword commits to memory, or am I violating alignment requirements for RIP-relative addressing on 64-bit ModR/M bytes?

Do I need an explicit MFENCE / SERIALIZE instruction here, or is Linux handling page fault permissions asynchronously?

Guidelines

• I have read and understood this category's guidelines before making this post.

[AntonAzer]

Hi @studioframes!

This issue comes down to two hardware-level phenomena on modern x86_64 CPUs: Store Buffer vs. IFU race conditions and Cache-Line Tearing.

1. Store Buffer vs. IFU (Stale Data)

The Instruction Fetch Unit (IFU) fetches from L1i and does not snoop the CPU's local Store Buffer. Under heavy CPU load, the updated immediate value written by add qword sits in the Store Buffer long enough for the speculatively prefetched mov rax to retire with stale data before the store drains into L1d to trigger an SMC Machine Clear.

2. Cache-Line Tearing (Segfaults)

Your 8-byte immediate (patch_target) is likely unaligned and straddles a 64-byte cache line boundary. Writes across cache lines are non-atomic. If the IFU fetches the instruction mid-store, it reads a corrupted ("Frankenstein") value into RAX. Dereferencing or jumping to this corrupted value triggers the Segmentation Fault.

Why QEMU works: QEMU (TCG) translates blocks sequentially and invalidates Translation Blocks synchronously, bypassing store buffer mechanics entirely.

---

How to Fix It

Option A: Avoid SMC in Tight Loops (Recommended)
Self-modifying code triggers severe hardware pipeline flushes. Instead of modifying instructions on every iteration, mutate the value in a register:

movrax,0x1122334455667788.loop: ; ... do work ...addrax,1decrbxjnz .loop

Option B: Alignment + Serialization (If SMC is required)

1. Align instructions with NOPs so patch_target never crosses a 64-byte boundary.
2. Drain the Store Buffer and serialize the pipeline using mfence + serialize (or cpuid on older CPUs):

.loop:movrax,0x1122334455667788patch_target equ $ -8add qword [rel patch_target],1mfenceserialize ; or cpuid on older CPUsdecrbxjnz .loop

[willsun0]

Hi!

Your self-modifying code crashes because it crosses cache boundaries. Non-atomic cross-line writes split your instruction in half.

Fix 1: Modify Registers

Avoid memory writes completely by updating a register.

movrax,0x1122334455667788.loop:addrax,1 ; Modify register directlydecrbxjnz .loop

Fix 2: Align and Serialize

Force cache alignment and flush the CPU pipeline.

align64 ; Keep instruction inside one cache line.loop:movrax,0x1122334455667788 patch_target equ \$ -8add qword [rel patch_target],1mfence ; Flush store bufferserialize ; Clear pipeline desyncdecrbxjnz .loop

[CristianOlivera1]

Hi @studioframes!

Both @AntonAzer and @willsun0 nailed the root cause, but there is an additional subtle layer regarding how the instruction pointer (RIP) behaves during cross-cache-line fetches that explains why you get an actual SIGSEGV instead of just a bad value inside RAX.

The "Mystery" Segfault Explained

Looking closely at your assembly snippet:

.loop:movrax,0x1122334455667788patch_target equ \$ -8add qword [rel patch_target],1

If the 8-byte immediate value (0x1122334455667788) crosses a 64-byte cache line boundary, the write is non-atomic. Under high CPU load and multi-core context switching, the Instruction Fetch Unit (IFU) can read a partially written, corrupted immediate.

However, notice that your mov rax, imm64 instruction does not dereference memory or perform a branch—it just loads a constant into a register. Loading a garbage value into RAX alone cannot cause a Segmentation Fault.

The SIGSEGV happens because the ModR/M byte or the opcode itself is straddling the cache line boundary, or the execution pipeline is getting completely desynchronized. If the CPU fetches a corrupted, illegal opcode or an invalid ModR/M byte due to the asynchronous split-store, it will decode a completely different instruction (e.g., a garbage memory reference or an invalid branch), triggering an immediate page fault or general protection fault (#GP).

Why Linux vs. QEMU?

• QEMU (TCG): Translates guest code into host instructions in a highly synchronized Basic Block manner. It lacks a true hardware Store Buffer / IFU desync simulation.
• Native Linux (High Load): Cross-core cache coherency traffic (cross-modifying code) forces intensive cache invalidation probes. When multiple cores are heavily hammering memory, the window of time where a cache line remains "torn" or un-synced widens dramatically.

The Correct Fixes

Option A: Avoid SMC entirely (Best Performance)

Modifying instructions in a tight loop forces an SMC Machine Clear, which destroys pipeline performance on modern x86_64 chips. Mutate the register instead:

movrax,0x1122334455667788.loop: ; ... do your JIT work here ...addrax,1decrbxjnz .loop

Option B: Alignment + Pipeline Serialization (If SMC is strictly required)

If your JIT compiler must test self-modifying code architectures, you have to strictly enforce that the instruction stays within a single cache line, and explicitly serialize execution.

On Intel/AMD, a simple JMP instruction immediately following the modification is historically guaranteed to flush the prefetch queue (Intel SDM Vol 3, Section 11.6). For modern architectures, combining it with SERIALIZE (or CPUID on older chips) is safest:

section .textglobal_start_start:movrbx,10 ; Align loop entry point to clear cache boundaries .balign 64.loop:movrax,0x1122334455667788patch_target equ \$ -8add qword [rel patch_target],1 ; Serialize instruction stream & clear prefetch queuejmp .serialize_target.serialize_target: ; serialize ; Use if compiling for newer targets supporting Intel 'SERIALIZE'decrbxjnz .loop

[studioframes]

Thanks!