Hugging Face Candle: Serverless No-Python Inference

Why Candle Eliminates Python in Production

Deploying PyTorch models in containerized production requires 8GB to 15GB Docker images containing the Python interpreter, CUDA toolkits, and hundreds of C++ shared libraries. Candle (developed by Hugging Face in pure Rust) compiles into a single standalone 35MB binary that boots in sub-50 milliseconds with zero Python runtime dependencies.

Loading and Evaluating a Model with Candle

use candle_core::{Device, Tensor};
use candle_nn::{Linear, Module};

pub fn run_candle_linear() -> Result<(), candle_core::Error> {
    // Select CUDA Device 0 if available, else fallback to CPU
    let device = Device::new_cuda(0)?;

    // Create tensors directly in VRAM
    let weights = Tensor::randn(0f32, 1.0, (1024, 4096), &device)?;
    let bias = Tensor::zeros((1024,), candle_core::DType::F32, &device)?;
    let layer = Linear::new(weights, Some(bias));

    let input = Tensor::randn(0f32, 1.0, (8, 4096), &device)?;
    let output = layer.forward(&input)?;

    println!("Candle Forward Pass complete: shape = {:?}", output.shape());
    Ok(())
}