Writing Custom CUDA Kernels via Rust FFI (cudarc & PTX)

Bypassing Framework Overhead with Direct CUDA FFI

While frameworks like Candle provide rich high-level tensor abstractions, maximizing inference throughput often requires custom operator fusion. In high-performance Rust harnesses, the cudarc crate provides safe, zero-cost bindings to the CUDA Driver API, allowing you to load compiled PTX (Parallel Thread Execution) assembly and launch kernels directly into asynchronous CUDA streams.

Operator Fusion: RMSNorm + Rotary Position Embedding (RoPE)

In standard transformer inference, applying RMSNorm followed by RoPE requires two separate GPU kernel launches and round-trips to high-bandwidth memory (HBM). By fusing both operations into a single CUDA kernel in Rust, intermediate activations remain inside the GPU's ultra-fast register file and SRAM, cutting memory bandwidth consumption by 50%.

use cudarc::driver::{CudaDevice, LaunchAsync, LaunchConfig};
use std::sync::Arc;

pub struct FusedOperatorHarness {
    dev: Arc,
}

impl FusedOperatorHarness {
    pub fn new(device_id: usize, ptx_path: &str) -> anyhow::Result {
        let dev = CudaDevice::new(device_id)?;
        dev.load_ptx(ptx_path.into(), "fused_ops", &["fused_rmsnorm_rope"])?;
        Ok(Self { dev })
    }

    pub fn launch(&self, hidden_dim: u32, num_tokens: u32) -> anyhow::Result<()> {
        let f = self.dev.get_func("fused_ops", "fused_rmsnorm_rope").unwrap();
        let cfg = LaunchConfig {
            grid_dim: (num_tokens, 1, 1),
            block_dim: (hidden_dim.min(1024), 1, 1),
            shared_mem_bytes: 0,
        };
        // Zero-copy async kernel launch directly on hardware stream
        unsafe { f.launch(cfg, ()) }?;
        Ok(())
    }
}