Building a 7,800+ tok/s LLM Engine in Rust + CUDA

The Void LLM Architecture

Void LLM is an independent transformer training and inference engine built from scratch in pure Rust with custom CUDA C++ kernels. Void proves that decoupling LLM runtimes from massive Python dependency stacks (PyTorch, Triton, LibTorch) unlocks 7,800+ tokens/sec throughput on consumer and enterprise GPUs.

Zero-Copy Tensor Loading via memmap2

Rather than reading gigabytes of weights into host RAM and serializing them into tensors, Void uses memory-mapped I/O (memmap2). The OS pages weights directly from the NVMe drive into the CUDA host-pinned staging buffer:

use memmap2::MmapOptions;
use std::fs::File;

pub struct SafetensorsLoader {
    mmap: memmap2::Mmap,
}

impl SafetensorsLoader {
    pub fn open(path: &str) -> std::io::Result {
        let file = File::open(path)?;
        let mmap = unsafe { MmapOptions::new().map(&file)? };
        Ok(Self { mmap })
    }

    pub fn get_tensor_slice(&self, offset: usize, len: usize) -> Result<&[f32], &'static str> {
        let byte_slice = &self.mmap[offset..offset + len * 4];
        let ptr = byte_slice.as_ptr();
        // Strict Invariant: f32 pointer dereference requires 4-byte alignment
        if ptr as usize % std::mem::align_of::() != 0 {
            return Err("Tensor data is not 4-byte aligned for f32 slice");
        }
        unsafe {
            Ok(std::slice::from_raw_parts(ptr as *const f32, len))
        }
    }
}