VOID 1B MIXTURE OF EXPERTS

From-Scratch BPE Tokenizer Engine

Custom Byte-Pair Encoding implementation hand-written in Rust. Builds 32,000 token vocabularies from raw corpus, computes subword merge frequency tables, and handles byte-level fallback for zero out-of-vocab errors.

Gradient-Checked 12-Expert MoE Router

DeepSeek-style sparse activation engine with 12 specialized experts. Uses auxiliary-loss-free load balancing and Top-k gating to route tokens dynamically without compute waste.

FlashAttention & Gemma-3 QK-Norm

Grouped Query Attention (GQA) with Gemma-3 style QK-Normalization to bound attention logit growth, paired with RoPE positional encodings and custom CUDA FlashAttention kernels.

Spiking Neural Network (SNN) FFN

Experimental Leaky Integrate-and-Fire (LIF) spiking neuron FFN block with surrogate gradient backward passes for ultra-low power neuromorphic computing research.

Architecture Growth & Weight Transfer

Expands trained checkpoints into larger model topologies (e.g. 125M -> 300M -> 1B MoE) by inserting near-identity initialized layers, preserving existing learned representations.

Dynamic CUDA NVRTC Kernel Dispatch

Safe Rust bindings via cudarc to dynamically compile CUDA C kernels into PTX bytecode at runtime, featuring PTX caching and automatic cuBLAS SGEMM/HGEMM dispatch.

Hyperparameters

GPU Telemetry

• ▸N-dimensional storage with contiguous/strided layouts
• ▸Shape broadcasting & automatic reshape
• ▸CPU ↔ CUDA device transfer
• ▸In-place and out-of-place operations
• ▸Lazy computation with fused kernels

• ▸MatMul, BatchMatMul, BMM with transpose
• ▸Softmax, LogSoftmax, GELU, SiLU/SwiGLU
• ▸LayerNorm, RMSNorm, Dropout
• ▸RoPE positional encoding
• ▸Cross-entropy loss with label smoothing

• ▸Reverse-mode automatic differentiation
• ▸Dynamic computation graph
• ▸Gradient accumulation & clipping
• ▸Memory-efficient checkpointing
• ▸Custom backward for attention & FFN

• ▸cudarc 0.19 — safe Rust bindings
• ▸cuBLAS SGEMM/DGEMM for matmul
• ▸Custom NVRTC-compiled kernels
• ▸Async memory copies & streams
• ▸Pinned host memory for transfers

• ▸Router + 12 domain expert selection, gradient-checked end-to-end
• ▸DeepSeek-style auxiliary-loss-free load balancing
• ▸Configurable drop-in alternative to the dense FFN block
• ▸Huge capacity with modest active compute per token

• ▸Real cuBLAS Hgemm fp16 GPU compute, verified against f32 CPU reference
• ▸Confirmed 2-bytes-vs-4 memory savings on real hardware
• ▸fp32 master weights with fp16 forward/backward for stability
• ▸Full training-loop storage integration

• ▸Round-trip quantize/dequantize verified for correctness
• ▸~4x storage reduction for serving trained checkpoints
• ▸GPU-accelerated INT8 GEMM kernel integration

• ▸Numerical finite-difference check against every hand-written analytical gradient
• ▸Catches silent correctness bugs a loss curve alone would miss
• ▸Extended to cover MoE and mixed-precision paths

• ▸A trained checkpoint expands into a larger model without retraining
• ▸New layers initialize near-identity preserving existing knowledge
• ▸Seamless path from 125M to 1B MoE

• ▸Isolated, gradient-checked spiking FFN block (LIF neurons)
• ▸Benchmarked head-to-head against dense equivalents
• ▸Spiking efficiency benchmarked for neuromorphic hardware

• ▸Intent classification and task decomposition
• ▸Multi-step execution plan generation
• ▸Dynamic routing based on model confidence

• ▸Sandboxed execution with timeout guards
• ▸Output validation against expected schemas
• ▸Automatic retry with corrective prompting

• ▸Critique-and-Train learning from execution outcomes
• ▸DPO-style preference optimization from harness feedback
• ▸Continuous self-improvement without human annotation

• ▸Teacher-student knowledge transfer
• ▸Dynamic growth policy for model scaling
• ▸Checkpoint expansion with preserved learned weights

Zero-Cost Abstractions

Rust's type system and ownership model produce code that compiles to the same machine instructions as hand-tuned C — with full memory safety guarantees. No garbage collector pauses during training.

Fearless Concurrency

Data races are compile-time errors in Rust. Rayon parallelism for CPU-bound data loading, async CUDA stream management, and lock-free metric reporting — all verified at compile time.

Single Binary Deployment

Void compiles to a single static binary. No conda environments, no pip dependencies, no virtualenvs, no CUDA version mismatches. Just run the binary.

Native CUDA Integration

cudarc provides safe Rust bindings to the CUDA driver API — device management, memory allocation, kernel launches, cuBLAS — without unsafe blocks leaking into application code.

How to Train an LLM From Scratch

Void's full stack — BPE tokenizer, transformer, autograd, optimizer, checkpointing — is hand-written in Rust, making every stage of the training pipeline inspectable rather than hidden behind a framework abstraction.

Rust vs Python for Machine Learning

No garbage collector pauses during training, compile-time data-race safety for parallel data loading, and a single static binary with no conda/pip/CUDA version conflicts to manage.

Self-Evolving Neural Architecture Search

An autonomous mutation-and-fitness loop that adds/removes layers, adjusts attention heads, and resizes the FFN, then evaluates and selects the best-performing variant with isolated rollback safety.

GPU LLM Training Without PyTorch

cudarc's safe Rust bindings drive cuBLAS matmul and custom NVRTC-compiled kernels directly — proving GPU-accelerated training doesn't require a Python ML framework in the loop.

Mixture-of-Experts vs Dense Transformers

Void's gradient-checked MoE layer routes each token through 12 specialized experts instead of the full network — the same sparse-activation principle DeepSeek published at 671B scale.