Writing Custom CUDA Kernels via Rust FFI (cudarc & PTX)

Bypassing Framework Overhead with Direct CUDA FFI

While frameworks like Candle provide rich high-level tensor abstractions, maximizing inference throughput often requires custom operator fusion. On NVIDIA hardware, the cudarc crate (v0.19, supporting CUDA 11.4 through CUDA 13.4 via CudaContext and CudaDevice) provides safe, zero-cost bindings to the CUDA Driver API. For cross-vendor Vulkan, DirectX 12, and Metal portability, pure-Rust engines like Infinity's HARNESS use wgpu v30.0.1 (MSRV Rust 1.87+) with native WGSL subgroup intrinsics (enable subgroups;).

Operator Fusion: RMSNorm + Rotary Position Embedding (RoPE)

In standard transformer inference, applying RMSNorm followed by Rotary Position Embedding (RoPE)—which rotates each 2D head pair (x2i, x2i+1) at token index m by angle mθi (θi = base-2i/d) via x'2i = x2icos(mθi) - x2i+1sin(mθi) and x'2i+1 = x2isin(mθi) + x2i+1cos(mθi)—requires two separate GPU kernel launches and HBM round-trips. Fusing both into a single kernel keeps intermediate activations inside registers and SRAM, cutting HBM traffic by 50%.

use cudarc::driver::{CudaDevice, LaunchAsync, LaunchConfig};
use std::sync::Arc;

pub struct FusedOperatorHarness {
    dev: Arc,
}

impl FusedOperatorHarness {
    pub fn new(device_id: usize, ptx_path: &str) -> anyhow::Result {
        let dev = CudaDevice::new(device_id)?;
        dev.load_ptx(ptx_path.into(), "fused_ops", &["fused_rmsnorm_rope"])?;
        Ok(Self { dev })
    }

    pub fn launch(&self, hidden_dim: u32, num_tokens: u32) -> anyhow::Result<()> {
        let f = self.dev.get_func("fused_ops", "fused_rmsnorm_rope").unwrap();
        let cfg = LaunchConfig {
            grid_dim: (num_tokens, 1, 1),
            block_dim: (hidden_dim.min(1024), 1, 1),
            shared_mem_bytes: 0,
        };
        // Zero-copy async kernel launch directly on hardware stream
        unsafe { f.launch(cfg, ()) }?;
        Ok(())
    }
}