Building a 7,800+ tok/s LLM Engine in Rust + CUDA

The Void LLM Architecture

Void LLM (32,000+ LOC across 87 Rust + CUDA source files, 8 model presets from 6.9M seed to 3B frontier, and 242 passing tests) is an independent Transformer and Mixture-of-Experts (MoE) training and inference engine built from scratch without PyTorch, Triton, or LibTorch. Decoupling LLM execution from Python unlocks 7,800+ to 9,500 tokens/sec throughput with custom FlashAttention SRAM-tiled kernels and FP16/INT8 quantization.

Zero-Copy Tensor Loading via memmap2

Rather than reading gigabytes of weights into host RAM and serializing them into tensors, Void uses memory-mapped I/O (memmap2). The OS pages weights directly from the NVMe drive into the CUDA host-pinned staging buffer:

use memmap2::MmapOptions;
use std::fs::File;

pub struct SafetensorsLoader {
    mmap: memmap2::Mmap,
}

impl SafetensorsLoader {
    pub fn open(path: &str) -> std::io::Result {
        let file = File::open(path)?;
        let mmap = unsafe { MmapOptions::new().map(&file)? };
        Ok(Self { mmap })
    }

    pub fn get_tensor_slice(&self, offset: usize, len: usize) -> Result<&[f32], &'static str> {
        let byte_slice = &self.mmap[offset..offset + len * 4];
        let ptr = byte_slice.as_ptr();
        // Strict Invariant: f32 pointer dereference requires 4-byte alignment
        if ptr as usize % std::mem::align_of::() != 0 {
            return Err("Tensor data is not 4-byte aligned for f32 slice");
        }
        unsafe {
            Ok(std::slice::from_raw_parts(ptr as *const f32, len))
        }
    }
}