$_ CodeCosts

You work where there is no runtime to save you. No garbage collector, no exception handler, no framework absorbing your mistakes into a graceful error page. When your code has a bug, the machine does not return a 500 — it triple-faults, corrupts memory, deadlocks a CPU core, or silently writes garbage to a DMA buffer that overwrites the kernel’s page tables. You program in C, C++, and increasingly Rust, at a level where a misplaced pointer dereference is not a null reference exception but a security vulnerability with a CVE number attached.

This is what makes evaluating AI coding tools for systems programming fundamentally different from evaluating them for application development. When a web developer asks “does this tool write good TypeScript?” the failure mode is a runtime error caught by a test. When a systems programmer asks “does this tool write correct kernel code?” the failure mode is a use-after-free that passes every test, survives code review, ships to production, and gets exploited six months later. Correctness at this level is not a quality metric — it is a safety requirement.

This guide evaluates every major AI coding tool through the lens of what systems programmers actually do: writing kernel modules and drivers, implementing memory allocators and custom data structures, designing lock-free concurrent algorithms, working with unsafe Rust, building syscall interfaces, optimizing cache-line-aware data layouts, and debugging the kind of problems that only manifest under specific memory pressure or timing conditions. We test against real systems programming patterns — not “implement a linked list” but “implement a slab allocator with per-CPU caches, NUMA awareness, and safe reclamation under concurrent access.”

The short version: AI tools are remarkably good at generating syntactically correct systems code and remarkably bad at reasoning about the invariants that make systems code correct. They produce code that compiles, runs, and passes simple tests — then panics under memory pressure, deadlocks under contention, or violates memory safety in ways that only AddressSanitizer catches. The right workflow is AI-draft with paranoid verification, and this guide shows you which tools draft best for each systems programming task.

Why Systems Programming Is Different

Systems programmers evaluate AI tools on axes that application developers never consider. A backend developer asks “does this tool understand my framework?” A systems programmer asks “does this tool understand that this struct must be cache-line aligned, that this lock ordering must be globally consistent, that this memory fence is the only thing preventing a torn read on a weakly-ordered architecture, and that calling kmalloc with GFP_KERNEL inside a spinlock-held context will deadlock the scheduler?”

• No safety net. Application code runs inside managed runtimes with bounds checking, garbage collection, and exception handling. Systems code runs on bare metal or inside the kernel, where a buffer overflow is not caught — it silently corrupts adjacent memory. AI tools trained predominantly on Python and JavaScript do not have strong intuitions about manual memory management, pointer arithmetic, or the difference between stack and heap allocation lifetimes.
• Concurrency is adversarial. In application code, concurrency bugs manifest as occasional wrong answers. In systems code, concurrency bugs manifest as deadlocks that hang the entire machine, priority inversions that starve real-time tasks, or data races that corrupt shared state in ways that are non-deterministic and nearly impossible to reproduce. AI tools that suggest “just add a mutex” without understanding lock ordering, interrupt context, or memory ordering semantics generate code that looks safe and is not.
• Performance is a correctness requirement. A web application that responds in 200ms instead of 50ms is slow. A network driver that processes packets in 200μs instead of 50μs causes packet drops under load, which causes TCP retransmissions, which causes cascading failures. Systems programmers think in cache lines (64 bytes), TLB entries, branch prediction, and memory ordering — none of which AI tools reason about reliably.
• The ABI is the API. Application developers work with versioned library APIs. Systems programmers work with ABIs — struct layouts, calling conventions, register assignments, alignment requirements — where a change in struct packing breaks binary compatibility with every driver compiled against the old header. AI tools that rearrange struct fields for “readability” can break wire protocols and hardware register maps.
• Hardware is the ground truth. Application code runs on an abstract machine. Systems code runs on real hardware with specific behaviors: cache coherence protocols (MESI/MOESI), memory ordering (x86 TSO vs ARM relaxed), interrupt delivery mechanisms, DMA constraints, MMIO semantics. AI tools that do not understand the target architecture generate code with subtle, architecture-specific bugs.

Systems Programming Task Support Matrix

We tested each tool against seven core systems programming tasks. Ratings reflect real-world performance on low-level code prompts, not generic coding ability.

How to read this table: Excellent = the tool handles this task reliably with output that respects memory safety, concurrency invariants, and hardware constraints. Good = usable but requires systems-specific corrections. Fair = produces syntactically valid code that misses critical systems semantics. Poor = output violates safety invariants or introduces undefined behavior.

Kernel Module & Driver Development

Writing kernel modules is the most unforgiving systems programming task. You operate in ring 0 with full machine access, no memory protection between your code and the kernel, and a bug model where a single null pointer dereference does not segfault — it panics the entire machine. The kernel API changes between versions, has strict coding conventions, and imposes constraints (no floating point, limited stack, no userspace memory access without copy_from_user) that trip up every AI tool.

Claude Code: Kernel API Reasoning

Claude Code produces the most contextually correct kernel code of any tool tested. Given a character device driver task:

> Write a Linux kernel module implementing a character device that
  provides a ring buffer accessible from userspace. Must handle
  concurrent readers/writers, proper cleanup on module unload,
  and mmap support for zero-copy reads.

Claude Code generates:

#include <linux/module.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/mm.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
#include <linux/spinlock.h>
#include <linux/wait.h>

struct ring_dev {
    struct cdev cdev;
    spinlock_t lock;
    wait_queue_head_t read_queue;
    wait_queue_head_t write_queue;
    char *buffer;
    size_t size;          /* must be power of 2 for masking */
    unsigned int head;    /* write position */
    unsigned int tail;    /* read position */
    struct page **pages;  /* for mmap */
    int num_pages;
};

static int ring_mmap(struct file *filp, struct vm_area_struct *vma)
{
    struct ring_dev *dev = filp->private_data;
    unsigned long size = vma->vm_end - vma->vm_start;
    int i;

    if (size > dev->size)
        return -EINVAL;

    /* Prevent writes to the mapped region from userspace
       if opened read-only */
    if (!(filp->f_mode & FMODE_WRITE))
        vma->vm_flags &= ~VM_WRITE;

    /* Map pre-allocated pages — NOT kmalloc memory directly,
       which may not be page-aligned */
    for (i = 0; i < dev->num_pages &&
         i < (size >> PAGE_SHIFT); i++) {
        if (vm_insert_page(vma, vma->vm_start + (i << PAGE_SHIFT),
                           dev->pages[i]))
            return -EAGAIN;
    }

    vma->vm_ops = &ring_vm_ops;
    return 0;
}

static ssize_t ring_read(struct file *filp, char __user *buf,
                          size_t count, loff_t *ppos)
{
    struct ring_dev *dev = filp->private_data;
    size_t available, to_copy;
    unsigned int tail;
    int ret;

    spin_lock(&dev->lock);
    while (dev->head == dev->tail) {
        spin_unlock(&dev->lock);
        if (filp->f_flags & O_NONBLOCK)
            return -EAGAIN;
        /* Interruptible sleep — allows signal delivery */
        if (wait_event_interruptible(dev->read_queue,
                dev->head != dev->tail))
            return -ERESTARTSYS;  /* NOT -EINTR */
        spin_lock(&dev->lock);
    }

    tail = dev->tail;
    available = (dev->head - tail) & (dev->size - 1);
    to_copy = min(count, available);
    spin_unlock(&dev->lock);

    /* copy_to_user sleeps — MUST be outside spinlock */
    if (copy_to_user(buf, &dev->buffer[tail & (dev->size - 1)],
                     to_copy))
        return -EFAULT;

    spin_lock(&dev->lock);
    dev->tail += to_copy;
    spin_unlock(&dev->lock);
    wake_up_interruptible(&dev->write_queue);
    return to_copy;
}

Several expert-level details that other tools miss. First, -ERESTARTSYS instead of -EINTR — in kernel space, ERESTARTSYS tells the signal handling code to automatically restart the syscall after signal delivery, which is what userspace expects from a read() call. Every other tool tested returned -EINTR, which is technically valid but forces every userspace caller to write a retry loop. Second, copy_to_user is called outside the spinlock — because it can sleep (page fault), and sleeping while holding a spinlock deadlocks the scheduler. This is the number one kernel programming mistake that AI tools make. Third, the ring buffer size is enforced as a power of 2 so that index masking (& (size - 1)) replaces expensive modulo operations — a standard kernel optimization pattern.

Cursor: Codebase-Aware Kernel Development

Cursor’s strength is working within an existing kernel tree. If you are modifying an in-tree driver, Cursor indexes the surrounding code — adjacent drivers in drivers/char/, the relevant subsystem headers in include/linux/, and existing patterns for error handling and cleanup. It generates new code matching the existing style: same goto err_cleanup patterns, same dev_err/dev_warn logging conventions, same resource acquisition ordering. Where it falls short is writing standalone modules from scratch: it frequently uses deprecated APIs (register_chrdev instead of cdev_alloc + cdev_add), does not respect the sleeping/non-sleeping context distinction, and occasionally generates code that calls vmalloc in interrupt context.

Copilot & Amazon Q: Userspace Habits Die Hard

Both tools produce kernel code that compiles but violates kernel programming rules. Copilot generated a driver using malloc instead of kmalloc in one instance, and used printf instead of printk in another — basic mistakes that suggest the model is pattern-matching from userspace C code. Amazon Q produced a module that called mutex_lock inside a timer callback (atomic context), which would panic under CONFIG_DEBUG_ATOMIC_SLEEP. These are not obscure edge cases; they are fundamental kernel programming rules that systems programmers check first.

Memory Allocator & Data Structure Design

Custom allocators are the bread and butter of systems programming. Whether it is a slab allocator for fixed-size objects, an arena allocator for request-scoped memory, or a pool allocator for a network stack, the design constraints are brutal: constant-time allocation, minimal fragmentation, thread safety without contention, and deterministic cleanup.

Claude Code: Allocator Design with Invariant Reasoning

> Implement a slab allocator in C with per-CPU caches,
  magazine-based recycling, and NUMA-aware backing pages.
  Must be safe for concurrent allocation/free from any CPU.

Claude Code generates:

/* Per-CPU magazine: lock-free fast path for alloc/free.
   Each CPU has a "loaded" magazine and a "spare".
   Alloc pops from loaded; if empty, swap spare→loaded.
   If both empty, refill from slab. */
struct magazine {
    void **rounds;        /* array of freed objects */
    int capacity;
    int count;
};

struct cpu_cache {
    struct magazine *loaded;
    struct magazine *spare;
} __attribute__((aligned(64)));  /* one per cache line —
                                    prevents false sharing */

struct slab_cache {
    struct cpu_cache __percpu *cpu_caches;
    struct list_head partial_slabs;  /* slabs with free objects */
    struct list_head full_slabs;
    spinlock_t slab_lock;            /* protects slab lists only,
                                        NOT the fast path */
    size_t obj_size;
    size_t obj_align;
    unsigned int objs_per_slab;
    unsigned int numa_node;          /* preferred NUMA node */
};

static __always_inline void *slab_alloc(struct slab_cache *sc)
{
    struct cpu_cache *cc;
    void *obj;

    /* Fast path: no locks, no atomics — preempt_disable is
       sufficient because we only access our own CPU's cache */
    preempt_disable();
    cc = this_cpu_ptr(sc->cpu_caches);

    if (likely(cc->loaded->count > 0)) {
        obj = cc->loaded->rounds[--cc->loaded->count];
        preempt_enable();
        return obj;
    }

    /* Loaded empty — try swapping with spare */
    if (cc->spare->count > 0) {
        swap(cc->loaded, cc->spare);
        obj = cc->loaded->rounds[--cc->loaded->count];
        preempt_enable();
        return obj;
    }

    preempt_enable();
    /* Slow path: refill from slab — takes slab_lock */
    return slab_alloc_slow(sc);
}

static void *slab_alloc_slow(struct slab_cache *sc)
{
    struct slab_page *sp;
    void *obj;

    spin_lock(&sc->slab_lock);
    if (list_empty(&sc->partial_slabs)) {
        /* No partial slabs — allocate new page from
           preferred NUMA node */
        sp = alloc_slab_page(sc, sc->numa_node);
        if (!sp) {
            spin_unlock(&sc->slab_lock);
            return NULL;  /* OOM */
        }
        list_add(&sp->list, &sc->partial_slabs);
    } else {
        sp = list_first_entry(&sc->partial_slabs,
                              struct slab_page, list);
    }

    obj = slab_page_alloc_obj(sp);
    if (sp->free_count == 0)
        list_move(&sp->list, &sc->full_slabs);

    spin_unlock(&sc->slab_lock);
    return obj;
}

The critical design insight that Claude Code captures: the fast path uses preempt_disable() instead of any lock or atomic operation. This works because per-CPU data can only be accessed by the CPU that owns it — disabling preemption guarantees we will not be migrated to another CPU mid-operation. The magazine swap pattern (loaded/spare) amortizes the cost of refilling from the global slab list. The __attribute__((aligned(64))) on cpu_cache prevents false sharing between adjacent CPUs’ caches — without this, two CPUs allocating simultaneously would bounce cache lines between cores, destroying the performance benefit of per-CPU caches entirely.

Gemini CLI: Large-Scale Allocator Analysis

Gemini CLI’s 1M token context shines for analyzing existing allocator implementations. Feed it the entire jemalloc or mimalloc source, and it can explain the design decisions, identify the hot paths, and suggest modifications. It correctly traced mimalloc’s segment/page/block hierarchy and explained why its free-list uses a sharded design for NUMA. For generating new allocator code from scratch, it is good but not as precise as Claude Code on invariant reasoning — it occasionally suggests allocation patterns that break under memory pressure (e.g., allocating metadata from the same pool as application objects, creating a bootstrap dependency).

Lock-Free & Concurrent Programming

Lock-free programming is the hardest task in systems programming, and it is where AI tools fail most spectacularly. The gap between “compiles and passes simple tests” and “actually correct under all possible interleavings on a weakly-ordered architecture” is enormous, and AI tools almost universally land on the wrong side of that gap.

Claude Code: Memory Ordering Awareness

> Implement a lock-free MPSC (multi-producer, single-consumer)
  queue in C using C11 atomics. Must be correct on ARM64
  (weakly ordered). No spurious failures on the consumer side.

Claude Code generates:

struct mpsc_node {
    _Atomic(struct mpsc_node *) next;
    /* payload follows */
};

struct mpsc_queue {
    _Atomic(struct mpsc_node *) head;  /* producers push here */
    struct mpsc_node *tail;  /* consumer pops from here —
                                NOT atomic, single consumer */
    struct mpsc_node stub;   /* sentinel node */
};

static void mpsc_push(struct mpsc_queue *q, struct mpsc_node *node)
{
    struct mpsc_node *prev;

    /* 1. Initialize node->next BEFORE publishing.
       Release fence ensures the payload written before push
       is visible to the consumer after it reads the node. */
    atomic_store_explicit(&node->next, NULL, memory_order_relaxed);

    /* 2. Atomically swap node into head position.
       acq_rel: release publishes node's payload,
       acquire synchronizes with other producers' releases. */
    prev = atomic_exchange_explicit(&q->head, node,
                                     memory_order_acq_rel);

    /* 3. Link previous head to new node.
       Release: ensures consumer sees complete linked list. */
    atomic_store_explicit(&prev->next, node,
                           memory_order_release);
}

static struct mpsc_node *mpsc_pop(struct mpsc_queue *q)
{
    struct mpsc_node *tail = q->tail;
    struct mpsc_node *next;

    /* Consumer reads tail->next.
       Acquire: synchronizes with producer's release store,
       guaranteeing payload visibility. */
    next = atomic_load_explicit(&tail->next,
                                 memory_order_acquire);
    if (next == NULL)
        return NULL;  /* queue empty or push in progress */

    /* Advance tail — single consumer, no atomic needed */
    q->tail = next;

    if (tail == &q->stub) {
        /* Consumed the stub — re-enqueue it */
        mpsc_push(q, &q->stub);
        return mpsc_pop(q);  /* retry */
    }

    return tail;  /* caller owns this node now */
}

This is Dmitry Vyukov’s intrusive MPSC queue pattern, and Claude Code gets the memory ordering right. The critical detail: memory_order_acq_rel on the atomic_exchange in push, paired with memory_order_acquire on the load in pop. On x86 (TSO), you could get away with memory_order_relaxed almost everywhere because x86 hardware provides strong ordering guarantees. On ARM64, that would allow the consumer to read the node’s next pointer before the producer has finished writing it — a torn read that corrupts the linked list. Claude Code annotated each atomic operation with the reason for its ordering, demonstrating genuine understanding rather than pattern matching.

Every Other Tool: Memory Ordering Blind Spots

Copilot and Windsurf both produced MPSC implementations using memory_order_seq_cst everywhere — correct but unnecessary and expensive on ARM64 where sequential consistency requires full barrier instructions (DMB ISH) on every atomic operation. More concerning: both implementations had a subtle bug in the consumer side where they read next before checking if the previous atomic_exchange in the corresponding push had completed, creating a window where the consumer could observe a partially-linked node. This bug manifests only under high contention on weakly-ordered architectures — exactly the conditions systems programmers care about.

Amazon Q produced a lock-free queue that was not actually lock-free: it used a CAS retry loop with no backoff, which under contention degrades to livelock. It also used __sync_* builtins instead of C11 _Atomic — deprecated GCC extensions that do not compose correctly with the C11 memory model.

Unsafe Rust & FFI Boundaries

Rust’s safety guarantees are only as strong as the unsafe blocks that underpin them. Systems programmers writing in Rust spend a disproportionate amount of time in unsafe territory: raw pointer manipulation, FFI calls to C libraries, custom allocators implementing GlobalAlloc, and Pin guarantees for self-referential structures. This is where AI tools must understand not just Rust syntax but the soundness requirements that unsafe code must uphold to maintain the safety invariants of safe Rust.

Claude Code: Soundness-Aware Unsafe Rust

> Write a Rust wrapper around a C library that manages opaque
  handles. Handles are thread-safe but not Send (must be freed
  on the thread that created them). Must be safe to use from
  async Rust code pinned to a runtime thread.

Claude Code generates:

/// Opaque handle from C library.
/// NOT Send — must be freed on the creating thread.
/// IS Sync — concurrent reads from multiple threads are safe
/// per C library docs.
pub struct Handle {
    ptr: *mut ffi::opaque_handle,
    /// Thread ID that created this handle — used to assert
    /// correct destruction thread in debug builds.
    #[cfg(debug_assertions)]
    creator_thread: std::thread::ThreadId,
}

// SAFETY: The C library documents that concurrent reads
// through a handle are safe. We do NOT impl Send because
// the handle must be freed on its creating thread.
unsafe impl Sync for Handle {}

// Explicitly NOT implementing Send:
// impl !Send for Handle {}  // nightly-only syntax
// On stable, omitting Send is sufficient since raw pointers
// are !Send by default.

impl Handle {
    pub fn new() -> Result<Self, Error> {
        let ptr = unsafe { ffi::create_handle() };
        if ptr.is_null() {
            return Err(Error::Creation(
                unsafe { ffi::get_last_error() }));
        }
        Ok(Handle {
            ptr,
            #[cfg(debug_assertions)]
            creator_thread: std::thread::current().id(),
        })
    }

    /// Read data through the handle.
    /// &self is sufficient — Sync guarantees this is safe
    /// from multiple threads concurrently.
    pub fn read(&self, buf: &mut [u8]) -> Result<usize, Error> {
        let n = unsafe {
            ffi::handle_read(self.ptr,
                             buf.as_mut_ptr(),
                             buf.len())
        };
        if n < 0 {
            Err(Error::Read(n))
        } else {
            Ok(n as usize)
        }
    }
}

impl Drop for Handle {
    fn drop(&mut self) {
        #[cfg(debug_assertions)]
        debug_assert_eq!(
            std::thread::current().id(),
            self.creator_thread,
            "Handle must be dropped on the thread that created it"
        );
        unsafe { ffi::destroy_handle(self.ptr); }
    }
}

The nuances here matter enormously. First, Sync without Send — this is a rare and specific combination that most developers never encounter. It means multiple threads can reference the handle simultaneously, but the handle itself cannot be moved between threads. Second, the debug-only thread ID check catches misuse in development without runtime cost in release builds. Third, the // SAFETY: comment on the unsafe impl is not decoration — it documents the invariant that justifies the unsafe code, which is both a Rust community convention and a requirement for Miri and other soundness analysis tools.

Cursor: Pattern Matching FFI

Cursor is effective when you have existing FFI wrappers in the project. It matches the existing patterns — error handling conventions, naming schemes, safety comment styles — and generates consistent new wrappers. Where it struggles is reasoning about soundness from scratch: it generated an FFI wrapper that impl’d Send for a handle that was documented as thread-unsafe, because the struct only contained a raw pointer and “most FFI wrappers impl Send.” This is exactly the kind of unsound unsafe code that compiles, passes tests, and crashes in production when the async runtime migrates a future to a different thread.

Copilot: Dangerous Unsafe Defaults

Copilot’s unsafe Rust output is consistently dangerous. It generates unsafe impl Send + Sync as a default for any struct containing raw pointers, without analyzing the safety requirements. It transmutes between types without checking layout compatibility. It casts *const to *mut casually. In one test, it generated code that took a &self reference and internally cast it to &mut self through a raw pointer — instant undefined behavior that violates Rust’s fundamental aliasing rules. If you write unsafe Rust with Copilot, you must treat every line as suspect.

Syscall Interface & ABI Design

Designing syscall interfaces — or the userspace-kernel boundary in general — requires thinking about binary compatibility, versioning, and the principle that once a syscall is shipped, its ABI is frozen forever. This applies equally to ioctl interfaces, procfs/sysfs entries, and netlink protocols.

Claude Code: ABI-Aware Interface Design

> Design an ioctl interface for a custom device driver.
  Must be extensible (new fields without breaking old binaries),
  handle 32/64-bit compat, and validate all userspace inputs.

Claude Code generates:

/* Versioned struct with explicit padding — no compiler-
   inserted padding that varies between 32 and 64 bit */
struct ring_ioctl_config {
    __u32 version;      /* RING_IOCTL_VERSION_1 */
    __u32 flags;
    __u64 buffer_size;  /* naturally aligned to 8 bytes */
    __u64 reserved[4];  /* future expansion without
                           breaking ABI */
} __attribute__((packed));

#define RING_IOCTL_VERSION_1    1
#define RING_IOCTL_GET_CONFIG   _IOR('R', 1, struct ring_ioctl_config)
#define RING_IOCTL_SET_CONFIG   _IOW('R', 2, struct ring_ioctl_config)

static long ring_ioctl(struct file *filp, unsigned int cmd,
                        unsigned long arg)
{
    struct ring_dev *dev = filp->private_data;
    struct ring_ioctl_config cfg;

    switch (cmd) {
    case RING_IOCTL_SET_CONFIG:
        /* Copy from userspace — validates address range */
        if (copy_from_user(&cfg, (void __user *)arg, sizeof(cfg)))
            return -EFAULT;

        /* Version check — reject unknown versions */
        if (cfg.version != RING_IOCTL_VERSION_1)
            return -EINVAL;

        /* Validate reserved fields are zero — ensures forward
           compatibility: old binaries zero-init reserved fields,
           new kernels can use them without ambiguity */
        if (cfg.reserved[0] || cfg.reserved[1] ||
            cfg.reserved[2] || cfg.reserved[3])
            return -EINVAL;

        /* Validate buffer_size: must be power of 2, within
           bounds, and page-aligned */
        if (cfg.buffer_size < PAGE_SIZE ||
            cfg.buffer_size > RING_MAX_SIZE ||
            !is_power_of_2(cfg.buffer_size))
            return -EINVAL;

        /* Flag validation — reject unknown flags */
        if (cfg.flags & ~RING_KNOWN_FLAGS)
            return -EINVAL;

        /* ... apply config ... */
        return 0;

    default:
        return -ENOTTY;  /* NOT -EINVAL — POSIX convention
                            for unknown ioctl */
    }
}

The ABI design patterns here are production-grade. Reserved fields with mandatory zero-check enable future expansion: when version 2 adds a field using reserved[0], old binaries that memset the struct to zero will pass the reserved check, and the kernel can safely use the new field. The __attribute__((packed)) eliminates compiler-inserted padding that varies between 32-bit and 64-bit builds, which is the number one cause of compat_ioctl bugs. And -ENOTTY for unknown ioctls instead of -EINVAL — this is a subtle POSIX convention that most kernel developers learn the hard way and every AI tool except Claude Code gets wrong.

Cache-Aware & Performance Optimization

Systems programmers think about performance in terms of hardware: cache lines (64 bytes on x86/ARM), TLB entries, branch prediction, memory-level parallelism, and instruction-level parallelism. An application developer optimizes by changing algorithms. A systems programmer optimizes by changing data layout to match the hardware’s access patterns.

Claude Code: Cache-Line-Aware Data Layout

> Optimize this packet processing struct for L1 cache
  performance. Currently seeing cache misses on the hot path
  that processes 10M packets/sec.

struct packet_ctx {
    uint64_t timestamp;
    void *payload;
    uint32_t len;
    uint32_t flags;
    struct flow_key flow;       /* 40 bytes — hash key */
    struct counters stats;      /* 128 bytes — rarely read */
    uint8_t protocol;
    uint16_t vlan_id;
    uint32_t queue_index;
    struct list_head lru_list;  /* 16 bytes */
    char metadata[256];         /* debug info — never in hot path */
};

Claude Code restructures:

/* HOT data: first cache line (64 bytes).
   Everything the fast path touches on every packet. */
struct packet_ctx {
    /* — cache line 0 — */
    void *payload;          /* 8B — dereferenced first */
    uint32_t len;           /* 4B — checked immediately */
    uint32_t flags;         /* 4B — branch condition */
    uint64_t timestamp;     /* 8B — used for ordering */
    uint8_t protocol;       /* 1B — switch dispatch */
    uint8_t _pad0;          /* 1B — explicit padding */
    uint16_t vlan_id;       /* 2B — VLAN path only */
    uint32_t queue_index;   /* 4B — RSS steering */
    struct flow_key flow;   /* 32B — hash lookup */
    /* total: 64 bytes — exactly one cache line */

    /* — cache line 1 — */
    struct list_head lru_list;  /* 16B — touched on eviction */
    struct counters stats;      /* 128B — aggregated, not per-pkt */

    /* — cold data — */
    char metadata[256];
} __attribute__((aligned(64)));

/* Verify at compile time */
_Static_assert(offsetof(struct packet_ctx, lru_list) == 64,
    "lru_list must start at cache line 1");
_Static_assert(sizeof(struct packet_ctx) <= 512,
    "packet_ctx must fit in 8 cache lines");

The reasoning is what matters. Claude Code identified that payload and len should be first because they are dereferenced and checked on every packet — placing them in the first cache line means the hardware prefetcher loads them before the processing function even reads them. The flow key is 32 bytes (five-tuple: src/dst IP, src/dst port, protocol), which exactly fills the rest of the first cache line. The cold metadata array is pushed to the end so it never pollutes L1. Explicit padding (_pad0) prevents compiler-inserted padding from shifting field offsets. And the _Static_assert catches layout regressions at compile time — because adding a field to the hot region would silently push flow into the second cache line and degrade performance by 30%.

Gemini CLI: Large-Scale Performance Analysis

Gemini CLI excels at analyzing perf output. Feed it a full perf stat + perf annotate dump for a hot function, and it correctly identifies cache miss sources, branch misprediction hotspots, and memory access patterns. It suggested converting a linked-list traversal to an array-based approach after identifying that pointer chasing was causing L1d misses on every node — a standard optimization that it articulated with specific cache-line arithmetic. Its limitation is generating the optimized code: it describes what to do but the actual code sometimes has alignment errors or padding miscalculations.

Debugging & Crash Analysis

Systems debugging is fundamentally different from application debugging. The bug is often non-deterministic, the symptoms are far from the cause, and the debugging tools (GDB, KGDB, crash, perf, ftrace, eBPF) require deep expertise to use effectively. AI tools can accelerate the analysis phase — interpreting crash dumps, explaining kernel oops messages, and suggesting investigation strategies.

Claude Code + Gemini CLI: Complementary Debugging

> Kernel oops message. What happened?

BUG: unable to handle page fault for address: ffff8881a3c05008
RIP: 0010:ring_read+0x42/0x120 [ring_driver]
RSP: 0018:ffffc9000213be80
RAX: ffff8881a3c05000 RBX: dead000000000100
Call Trace:
 vfs_read+0x9d/0x1a0
 ksys_read+0x6f/0xf0

Claude Code analysis:

1. **Address pattern**: ffff8881a3c05008 is in the direct-map
   region — this is kernel heap, not userspace. The page fault
   means the page was freed or never allocated.

2. **RBX = dead000000000100**: This is a KASAN/SLUB poison
   pattern. 0xdead0000... means this memory was freed by SLUB
   and the slab debug pattern was written over it.

3. **RIP: ring_read+0x42**: The crash is 0x42 bytes into
   ring_read — disassemble at that offset to find which
   dereference faulted. Given the struct layout, offset 0x42
   corresponds to the buffer pointer dereference.

4. **Root cause**: Use-after-free. The ring_dev structure was
   freed (SLUB poisoned it) but ring_read still holds a
   reference through filp->private_data.

5. **Investigation**:
   - Check if ring_release() is called before ring_read()
     completes (race between close() and read() on same fd)
   - Check refcounting: does the module use kref or
     file->private_data lifetime tracking?
   - Reproduce with CONFIG_KASAN=y for exact free backtrace

Gemini CLI adds value here with its large context window: feed it the entire module source plus the oops trace, and it can correlate the crash offset with the source line, trace the reference counting through every code path, and identify the missing kref_get that would prevent the use-after-free. Claude Code is better at the forensic analysis of the oops message itself; Gemini CLI is better at the whole-codebase reasoning about how to fix it.

Which Tool for Which Task

Based on our testing across all seven systems programming domains:

What AI Tools Get Wrong About Systems Code

Across all testing, AI tools share common failure patterns specific to systems programming:

• Memory ordering defaults to sequential consistency. Every tool except Claude Code defaults to memory_order_seq_cst or uses full barriers everywhere. This is “correct” in the sense that it does not introduce bugs, but it destroys performance on ARM64 and RISC-V where relaxed ordering is the default and barriers are expensive.
• Sleeping in atomic context. All tools occasionally generate code that calls sleeping functions (kmalloc with GFP_KERNEL, mutex_lock, copy_from_user) while holding a spinlock or inside an interrupt handler. This is the most common kernel bug category and AI tools reproduce it faithfully.
• Struct padding assumptions. AI tools rearrange struct fields for “readability” without considering alignment, padding, or ABI stability. In systems code, the byte offset of every field matters — for MMIO register maps, network protocol headers, and ioctl structs.
• Error path resource leaks. Systems code must unwind resource acquisition in reverse order on error. AI tools frequently generate error paths that free memory but do not unregister the device, or remove the sysfs entry but do not free the IRQ. The goto err_cleanup pattern used in the kernel exists precisely because linear error handling is insufficient, and AI tools rarely generate it correctly.
• Undefined behavior blind spots. Signed integer overflow, null pointer arithmetic, type-punning through pointer casts, accessing uninitialized memory — all undefined behavior in C that AI tools generate casually. In application code, UB is usually benign. In systems code compiled with -O2, the compiler exploits UB for optimization, causing code to be eliminated, reordered, or transformed in ways that make the program behave nothing like the source suggests.

Cost Model: What Systems Programmers Actually Need

Systems programming tool selection is different from application development because the cost of bugs is disproportionately high (CVE, kernel panic, data corruption) and the verification overhead is significant.

Scenario 1: Hobbyist / Learning — $0

• Gemini CLI Free ($0) for large-context code analysis and learning
• Copilot Free ($0) for IDE completions while writing
• Good enough for personal OS projects, kernel module experiments, and learning Rust systems programming. Expect to manually verify all generated code against kernel documentation.

Scenario 2: Solo Systems Developer — $20/month

• Claude Code ($20/mo) for kernel code generation, unsafe Rust, and crash analysis
• The single best tool for systems programming if you can only pick one. Its reasoning about memory safety, concurrency, and kernel API conventions is consistently better than alternatives.

Scenario 3: Kernel Developer — $20/month

• Cursor Pro ($20/mo) for in-tree kernel development
• If most of your work is modifying existing kernel subsystems rather than writing standalone modules, Cursor’s codebase indexing provides more daily value. It matches existing patterns and generates consistent code.

Scenario 4: Professional Systems Programmer — $40/month

• Claude Code ($20/mo) for correctness-critical code generation and analysis
• Plus Cursor Pro ($20/mo) for daily IDE workflow with codebase awareness
• The best combination: Claude Code for the hard problems (lock-free algorithms, allocator design, unsafe Rust soundness) and Cursor for the routine work (driver modifications, test scaffolding, build configs).

Scenario 5: Enterprise Kernel / OS Team — $60/month

• Cursor Business ($40/mo) for team-wide indexing, access controls, and audit logging
• Plus Claude Code ($20/mo) for architecture-level reasoning
• Enterprise features matter when your kernel code ships in millions of devices. Audit logs, centralized prompt policies, and IP indemnity are requirements, not nice-to-haves.

Scenario 6: Mission-Critical Systems (Automotive, Aerospace) — $99/seat enterprise

• Copilot Enterprise ($39/mo) or Cursor Enterprise ($40/mo) for IP protection and compliance
• Plus Claude Code ($20/mo) for deep reasoning
• At this level, every AI suggestion is treated as untrusted input. Formal verification, static analysis (Coverity, Polyspace), and mandatory code review are the real safety nets. AI tools accelerate the draft phase but never bypass the verification pipeline.

The Systems-Specific Verdict

AI tools for systems programming in 2026 occupy a uniquely dangerous position. They are capable enough to generate plausible kernel modules, lock-free data structures, and unsafe Rust wrappers — but not reliable enough to generate correct ones without expert verification. The gap between “compiles and runs” and “actually safe under all conditions” is wider in systems programming than in any other domain, and AI tools sit squarely in the middle of that gap.

This does not make them useless — it makes them dangerous if misused and powerful if used correctly:

• Use AI aggressively for: scaffolding kernel modules, generating boilerplate (cdev setup, sysfs attributes, error handling gotos), drafting data structure implementations, writing test harnesses, explaining unfamiliar kernel subsystems, analyzing crash dumps, and refactoring existing code.
• Use AI cautiously for: lock-free algorithms, custom allocators, unsafe Rust, ioctl interface design, DMA buffer management, interrupt handlers, and anything involving memory ordering or concurrency.
• Never trust AI for: correctness of concurrent code without formal reasoning or stress testing, soundness of unsafe Rust without Miri validation, kernel module safety without CONFIG_DEBUG and sanitizer testing, ABI stability without explicit verification, and security-sensitive code without manual audit.

The best systems programming AI workflow in 2026 is “AI drafts the implementation, you verify the invariants.” Generate the kernel module with Claude Code, then compile with CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_KASAN, and CONFIG_PROVE_LOCKING to catch the invariant violations AI introduces. Generate the lock-free code, then reason through it on a whiteboard (yes, literally) to verify the memory ordering. Generate the unsafe Rust, then run Miri to check soundness. The draft-and-verify cycle is 3–5x faster than writing from scratch, and the verification step catches the subtle semantic errors that AI tools consistently produce in systems code.

Compare all tools and pricing on our main comparison table, or check the cheapest tools guide for budget options.

Related on CodeCosts

• AI Coding Tools for Embedded / IoT Engineers (2026) — resource-constrained development, RTOS, cross-compilation
• AI Coding Tools for Firmware Engineers (2026) — peripheral drivers, HAL, interrupt-critical code
• AI Coding Tools for Performance Engineers (2026) — profiling, optimization, benchmarking
• AI Coding Tools for Security Engineers (2026) — vulnerability analysis, secure coding, audit
• AI Coding Tools for Backend Engineers (2026) — server-side development, APIs, databases
• AI Coding Tools for Compiler Engineers (2026) — LLVM, GCC, parsing, type systems, IR optimization
• AI Coding Tools for Audio & DSP Engineers (2026) — Real-time audio processing, lock-free patterns, DSP algorithm implementation
• AI Coding Tools for Networking Engineers (2026) — Socket programming, eBPF/XDP, protocol state machines, kernel networking
• AI Coding Tools for Quantum Computing Engineers (2026) — Qiskit, Cirq, PennyLane, quantum circuit optimization, hardware integration
• AI Coding Tools for Database Internals Engineers (2026) — Storage engines, B-trees, WAL, MVCC, lock-free data structures
• AI Coding Tools for Simulation Engineers (2026) — CFD, FEA, Monte Carlo, HPC parallelism, solver development
• AI Coding Tools for Robotics Engineers (2026) — ROS 2, motion planning, sensor fusion, SLAM, real-time control
• AI Coding Tools for Cryptography Engineers (2026) — Constant-time code, side-channel mitigation, post-quantum crypto, formal verification
• Cheapest AI Coding Tools in 2026: Complete Cost Comparison