What 30B–32B Coder and 70B performance has been measured on an 8GB RTX 5060?

On a bare-metal NVIDIA GeForce RTX 5060 (8,151 MiB GDDR7, 128-bit @ 28 Gbps = 448 GB/s) + Intel Core i5-10400F + 32 GB DDR4-2133, HARNESS achieves 16.37 tok/s pooled (16.53 tok/s peak on Rust, 78.0 tok/s prefill) on 30.5B Qwen3-Coder-30B-A3B-Instruct (Q4_K_M, Apache 2.0) via Hybrid Sparse MoE Partitioning (-ngl 99 --n-cpu-moe 32) — a 14.11x speedup over the 1.16 tok/s 32B baseline and 5.37x faster than standard Ollama (3.05 tok/s). On dense 32.76B Qwen2.5-Coder-32B-Instruct (Q4_K_M) with a 100% GPU-resident 0.5B drafter, it reaches 3.90 tok/s pooled (4.80 tok/s peak at 96.2% draft acceptance), and 15.89 tok/s pooled (19.17 tok/s peak) on Qwen2.5-Coder-14B. All 8 configurations pass 24/24 (100%) compiler and unit test assertions (rustc --test -O and Python DP). Historical baselines also include 77.6 tok/s on 7B Q4_K_M and 0.99 - 1.68 tok/s on 70B Q2_K.

What are the Three Empirical Hardware Laws discovered in the Skeptic's Trench benchmark?

1) Hybrid Sparse MoE Partitioning (-ngl 99 --n-cpu-moe 32): Offloading 100% of all 48 layers' shared 8:1 GQA attention (32Q/4KV, 256K native context, 32K active KV), KV cache, router gates, and 16 full MoE layers into GDDR7 while streaming only 8/128 (6.25% = ~0.70 GB/token) of active experts from DDR4 eliminates 95.2% of the DDR4 bus bottleneck (16.37 tok/s). 2) The Sparse MoE Expert-Union Batching Law: Speculative batching (M=7) speeds up Dense 32B by 3.36x (3.90 tok/s), but in 128-expert Top-8 MoE, the union of 7 tokens' Top-8 selections activates ~35-45 experts per CPU layer (4-5x more DDR4 traffic), so zero-draft MoE (16.37 tok/s) beats drafted MoE (11.59 tok/s). 3) The Windows WDDM ~6,650 MiB GDDR7 Spillover Cliff: Exceeding ~6,650 MiB on an 8,151 MiB GPU causes WDDM to silently page CUDA buffers over PCIe Gen 3 x8, dropping throughput by 13%-20%.

What native GPU kernels and quantization formats are implemented in HARNESS?

HARNESS implements direct wgpu (Vulkan/DirectX 12) WGSL compute shaders (respecting WebGPU baseline maxComputeInvocationsPerWorkgroup = 256 / @workgroup_size(256) and 16x16 tiles) benchmarked with explicit hardware synchronization (Maintain::Wait) on the RTX 5060: Activation-Sparse INT4 GEMV (sparse_gemv_q4.wgsl, 215.05 μs at 75% sparsity vs 311.10 μs dense Q4 and 752.03 μs FP32 — 313.9 GB/s effective BW = 70.1% of 448 GB/s GDDR7 peak), Batched Speculative Q4 GEMM (gemm_q4_batched.wgsl), Native GPU Top-8 MoE Softmax Router (moe_router_topk.wgsl), 5:1 & 8:1 GQA FlashAttention, 32B SwiGLU (435.08 μs, 8.13B elem/s), and RMSNorm (766.89 μs, 1.34M tok/s). Quantization primitives include bit-exact 256-weight GGUF Q4_K super-blocks (144 bytes = 4.50 bpw; ~4.80–4.85 bpw in mixed Q4_K_M, imatrix weighted least-squares refinement, fused dot_f32), OCP FP8 E4M3FN, 64-group Voronoi NF4, and real GGUF v2/v3 tensor mmap extraction.

How do OnlineDraftCalibrator and Pushdown/DFA Code Verification work?

OnlineDraftCalibrator (crates/harness-pipeline/src/draft_aligner.rs) performs online cross-entropy gradient updates to contextual bigram and unigram logit biases whenever the 32B target model rejects a 0.5B/1.5B draft token (+1.0 on target truth, -1.0 on rejected draft with cosine-annealed learning rate), paired with a zero-VRAM O(1) RepoSuffixDrafter. CodeQualityEngine and ConstrainedDecoder enforce 78.4 ns DFA token masks plus a pushdown brace/bracket/string automaton that distinguishes Rust lifetime annotations ('a, 'static) and comments from string literals.

How does Bio-Inspired LIF Spiking Attention work?

Standard transformer attention computes dense O(N²) floating-point matrix multiplications. HARNESS converts token activations into discrete Leaky Integrate-and-Fire (LIF) spike events. When membrane potential fails to breach dynamic threshold theta >= 0.35, attention evaluation for that key is bypassed entirely, pruning 50.0% of attention FLOPs in the repository report.

What is Hippocampal Fast-Slow Dual Memory (CLS Theory)?

Based on Complementary Learning Systems neuroscience, HARNESS splits memory into a volatile episodic buffer (recent high-resolution FP8/Paged KV blocks) and a cortical consolidator. During context pressure, the consolidator projects multi-layer KV states into low-rank sparse engram vectors, saving 93.8% of context in the repository report.

Can I use HARNESS with Cursor, Antigravity, or VS Code?

Yes. HARNESS includes a native Model Context Protocol crate (harness-mcp) that runs over stdio, plus an OpenAI-compatible Axum HTTP/SSE server (prioritizing port 8089 bare-metal 30B-32B Coder runner and port 8080 orchestration API) for Cursor, Antigravity, Windsurf, or VS Code.

Is HARNESS completely free and open-source?

Yes, 100% free and open-source under the permissive MIT license (75 passing workspace tests across 11 crates, 15,714 LOC, 0 clippy warnings). The full repository is available at https://github.com/ModernOps888/harness.

HARNESS v0.1.0 · PURE RUST 1.85+ (11 CRATES · 15.7K LOC · 75 TESTS)30B SPARSE MOE (QWEN3-CODER-30B-A3B): 16.37 TOK/S POOLED (14.11× SPEEDUP)14B + 0.5B SPECULATIVE: 15.89 TOK/S POOLED (19.17 PEAK · 98.7% DRAFT ACC)32B DENSE + 0.5B SPECULATIVE: 3.90 TOK/S POOLED (4.80 PEAK · 96.2% DRAFT ACC)WGPU VULKAN/DX12 TILED GEMM: 948.59 GFLOP/S SYNCED (4.67 TFLOP/S ASYNC)SKEPTIC'S TRENCH TORTURE SUITE: 24/24 (100%) COMPILER-VERIFIED PASSREAL 70B Q2_K ON RTX 5060 8GB: 0.99 - 1.68 TOK/S · 7B Q4_K_M: 77.6 TOK/SDFA MASK: 78.4 NS · PAGED KV: 17.3M BLOCKS/S (0 LEAKS)HARNESS v0.1.0 · PURE RUST 1.85+ (11 CRATES · 15.7K LOC · 75 TESTS)30B SPARSE MOE (QWEN3-CODER-30B-A3B): 16.37 TOK/S POOLED (14.11× SPEEDUP)14B + 0.5B SPECULATIVE: 15.89 TOK/S POOLED (19.17 PEAK · 98.7% DRAFT ACC)32B DENSE + 0.5B SPECULATIVE: 3.90 TOK/S POOLED (4.80 PEAK · 96.2% DRAFT ACC)WGPU VULKAN/DX12 TILED GEMM: 948.59 GFLOP/S SYNCED (4.67 TFLOP/S ASYNC)SKEPTIC'S TRENCH TORTURE SUITE: 24/24 (100%) COMPILER-VERIFIED PASSREAL 70B Q2_K ON RTX 5060 8GB: 0.99 - 1.68 TOK/S · 7B Q4_K_M: 77.6 TOK/SDFA MASK: 78.4 NS · PAGED KV: 17.3M BLOCKS/S (0 LEAKS)

← Infinity Tech Stack/FREE DEVELOPER TOOL: HARNESS

● 100% FREE & OPEN SOURCE (MIT)★ Star on GitHub

AUTONOMOUS PURE-RUST INFERENCE & SAFETY MIDDLEWARE30B SPARSE MOE @ 16.37 TOK/S · 24/24 COMPILER PASSWGPU VULKAN/DX12 + BIO-SNN & CLS

Break the Memory Wall.
30B–70B on an 8GB GPU. Measured, not claimed.

HARNESS is an autonomous pure-Rust LLM inference orchestration, custom wgpu (Vulkan/DX12/Metal) WGSL GPU kernel, and safety middleware fusing high-throughput systems programming with biological primitives. On a single consumer NVIDIA RTX 5060 (8 GB GDDR7 @ 448 GB/s) over PCIe Gen 3 x8, HARNESS drives 30.5B Sparse MoE (Qwen3-Coder-30B-A3B) at 16.37 tok/s pooled (16.53 peak — 14.11× speedup over 32B baseline, 5.37× faster than Ollama), 14B + 0.5B Speculative at 15.89 tok/s pooled (19.17 peak, 98.7% draft acceptance), 32B Dense + 0.5B Speculative at 3.90 tok/s pooled (4.80 peak, 3.36× speedup), 70B Q2_K at 0.99 – 1.68 tok/s, and 7B Q4_K_M at 77.6 tok/s — achieving a 24/24 (100%) compiler-verified pass rate (rustc --test -O & python) across the Skeptic's Trench torture suite.

⚡ Open Interactive Studio 📂 View GitHub Repository (11 Crates · 15.7k LOC)

30B Sparse MoE Coder

16.37 tok/s

14.11× vs 32B base · Qwen3-Coder-30B-A3B · 16.53 peak

14B + 0.5B Speculative

15.89 tok/s

19.17 tok/s peak · 98.7% draft acceptance (88.8% pool)

32B Dense + 0.5B Draft

3.90 tok/s

3.36× speedup · 4.80 peak · 96.2% draft acceptance

wgpu Tiled GEMM & Q4 BW

948.6 GFLOP/s

4.67 TFLOP/s async · 313.9 GB/s Q4_K GEMV on RTX 5060

Skeptic's Trench Gate

24/24 PASS

100% compiler-verified (rustc --test -O & Python DP)

7B Resident / 70B Stream

77.6 / 1.68

77.6 tok/s (7B Q4_K_M) · 1.68 tok/s (70B Q2_K + 1B)

⚡

HARNESS Interactive Studio & Telemetry Console

SIMULATION: interactive demo of layer streaming and SNN dynamics. Numbers in this console are illustrative, not measurements. Verified results come from benchmarks/runs/RESULTS.md in the repository.

Select Model Architecture & Quantization ModeZero-Copy Mmap DMA Active

Throughput

16.4 tok/s

TTFT Latency

24.5 ms

VRAM Allocation

6.3 / 8 GB

Paged KV Fragmentation

1.8%

Zero Memory Leaks

LIF Spiking Sparsity

50%

FLOPs Skipped

Confidence Score

98.3%

Entropy Calibrated

🌊Temporal Layer Streaming Architecture (scheduling simulation)

Double-Buffered Ping-Pong DMA

Slot 0: Active GPU ComputeLayer #14

Executing FlashAttention-3 + SwiGLU forward pass in a simulated 4.8GB VRAM budget.

Slot 1: Async DMA PrefetchLayer #15

Non-blocking PCIe stream transfer from pinned host RAM directly into VRAM buffer.

80-Layer Execution TimelineLayer 15 / 80

🧬Bio-Evolutionary & Anti-Hallucination Control Suite

LIF Spiking Threshold (θ)0.35

Skips sub-threshold key attention evaluation (50% FLOP savings).

Lateral Inhibition (γ)1.8

Biological winner-take-all: suppresses tail noise and hallucination drift.

Constrained Logit Masking (DFA)

Calculates token validity mask in 78.4 ns (<50 µs), ensuring zero syntax errors.

Observation Compactor85%

Masks verbose tool returns, saving 40-60% agent context window costs.

SIMULATED DEMO — NOT A REAL MODEL. Replies are canned/scripted text and every number shown (tok/s, TTFT, entropy, confidence, sparsity) is illustrative. No 70B (or any) inference runs on this page. For measured results see Verified Benchmarks.

HARNESS STUDIO — SIMULATED DEMO (no live model)Qwen3-Coder: 30B-A3B Sparse MoE (Verified m5)

BENCHMARK SUITE:

👤

Developer•17:00

Implement a generic thread-safe bounded lock-free Single-Producer Single-Consumer (SPSC) RingBuffer<T> in Rust with unit tests compiled via rustc --test -O.

⚡

Qwen3-Coder: 30B-A3B Sparse MoE (Verified m5)•17:00•16.4 tok/s (simulated)•24.5ms TTFT (simulated)

🧠 Scripted Reasoning Trace (simulated, Sparse-MoE)Hide Trace ▲

Scripted demo trace: no model is running; this text is canned.

### Skeptic's Trench Gate #2: Verified Lock-Free Bounded SPSC RingBuffer<T> in Pure Rust

```rust
use std::cell::UnsafeCell;
use std::mem::MaybeUninit;
use std::sync::atomic::{AtomicUsize, Ordering};

pub struct RingBuffer<T> {
    buffer: Box<[UnsafeCell<MaybeUninit<T>>]>,
    capacity: usize,
    head: AtomicUsize, // Consumer read index
    tail: AtomicUsize, // Producer write index
}

unsafe impl<T: Send> Send for RingBuffer<T> {}
unsafe impl<T: Send> Sync for RingBuffer<T> {}

impl<T> RingBuffer<T> {
    pub fn new(capacity: usize) -> Self {
        let cap = capacity.max(1) + 1; // +1 sentinel slot to distinguish full vs empty
        let mut vec = Vec::with_capacity(cap);
        for _ in 0..cap {
            vec.push(UnsafeCell::new(MaybeUninit::uninit()));
        }
        Self {
            buffer: vec.into_boxed_slice(),
            capacity: cap,
            head: AtomicUsize::new(0),
            tail: AtomicUsize::new(0),
        }
    }

    pub fn push(&self, item: T) -> Result<(), T> {
        let tail = self.tail.load(Ordering::Relaxed);
        let next_tail = (tail + 1) % self.capacity;
        if next_tail == self.head.load(Ordering::Acquire) {
            return Err(item); // Buffer full
        }
        unsafe { (*self.buffer[tail].get()).write(item); }
        self.tail.store(next_tail, Ordering::Release);
        Ok(())
    }

    pub fn pop(&self) -> Option<T> {
        let head = self.head.load(Ordering::Relaxed);
        if head == self.tail.load(Ordering::Acquire) {
            return None; // Buffer empty
        }
        let item = unsafe { (*self.buffer[head].get()).assume_init_read() };
        self.head.store((head + 1) % self.capacity, Ordering::Release);
        Some(item)
    }
}
```

#### Bare-Metal Verification Telemetry (`m5_qwen3_coder_30b_moe_ncmoe32_pure`):
1. **Measured Throughput (RTX 5060 8GB @ 448 GB/s GDDR7)**: **16.37 tok/s pooled** (**16.53 tok/s peak on Rust**, **78.0 tok/s prefill**) — **14.11× speedup** over 32B baseline (`1.16 tok/s`).
2. **Hybrid Sparse MoE Partitioning**: 100% of all 48 layers' shared 8:1 GQA attention (32Q / 4KV heads), KV cache (`78 MiB` @ 32K of 256K native context), router gates, plus 16 full MoE layers reside in GDDR7 (`6,463.4 MiB`), while 32 MoE layers stream only `8/128 = 6.25%` active experts (`~0.70 GB/token`) from DDR4.
3. **Compiler Gate**: Compiled and passed all multi-threaded SPSC assertions via `rustc --test -O` (`3/3 PASS`).

⚡ Pure Rust 1.85+ (No GC)🧬 Bio-SNN LIF Spiking: 50% FLOPs Skipped (illustrative)🛡️ DFA Constrained: json_schema

Press Enter to Send

HARDWARE ARCHITECTURE

Memory Tiering & Apple Silicon UMA Physics

Calculate the optimal inference strategy and model sizing for your physical workstation.

Workstation Spec Calculator

Operating Architecture

GPU Dedicated VRAM8 GB

4 GB (Edge)8 GB (RTX 5060, measured)16 GB (4080)24 GB (3090/4090)32 GB (5090 Blackwell)

Measured: RTX 5060 8GB GDDR7 (448 GB/s, 128-bit @ 28 Gbps) + 32GB DDR4-2133 (PCIe 3.0 x8)

30B Sparse MoE (Qwen3-Coder-30B-A3B) · 32B Dense + 0.5B Draft · 14B + 0.5B Draft · 70B Q2_K · 7B Q4_K_M

Execution Strategy: Sparse MoE --n-cpu-moe 32 (6,463 MiB CUDA) / Speculative Decoding -ngl 20..38 + -ngld 99 + --prio 2 / Partial GPU offload

Expected Speed:16.37 tok/s (30B MoE) · 15.89 tok/s (14B Spec) · 3.90 tok/s (32B Spec) · 77.6 tok/s (7B) · 1.68 tok/s (70B)

Bus Efficiency:448 GB/s GDDR7 (313.9 GB/s eff Q4_K GEMV = 70.1% util) + PCIe 3.0 x8 (~7.9 GB/s)

The Physics of Memory Bandwidth

On the measured PC, weights cross a PCIe 3.0 x8 link (~7.9 GB/s theoretical) from DDR4-2133 dual-channel RAM (34.1 GB/s theoretical peak). A 70B Q2_K model is 26.4 GB, so most layers live in host RAM and speed is bounded by memory bandwidth (tok/s ≤ bandwidth / active footprint). The real 70B numbers on this page come from llama.cpp partial GPU offload; the HARNESS ping-pong DMA scheduler is a simulation of the overlap.

On Apple Silicon, unified memory removes the PCIe hop and offers much higher bandwidth. The repository gives only theoretical bandwidth sizing for those machines (bandwidth / model size), not measurements, so no Mac throughput claim is made here.

PCIe 3.0 x8 (measured PC):~7.9 GB/s (theoretical)

Host Dual-Channel DDR4-2133 (measured PC):34.1 GB/s (theoretical)

NVIDIA RTX 5060 GDDR7 (128-bit @ 28 Gbps):448 GB/s (313.9 GB/s eff Q4_K)

Apple M4 Max:546 GB/s (spec, theoretical)

Apple M2/M3/M4 Ultra:800+ GB/s (spec, theoretical)

VERIFIED · SKEPTIC'S TRENCH TORTURE SUITE & RAW LOGS IN REPOSITORY

Measured on NVIDIA RTX 5060 8GB (GDDR7 · PCIe 3.0 x8)

Every figure below is computed from bare-metal hardware runs on an NVIDIA GeForce RTX 5060 (8,151 MiB GDDR7 @ 448 GB/s, 128-bit @ 28 Gbps) paired with an Intel Core i5-10400F (6C/12T) and 32 GB DDR4-2133 over PCIe Gen 3 x8 on Windows 11. All 8 elite coder configurations passed 24/24 (100%) automated compiler & execution checks (rustc --test -O & python) across 3 torture tasks (python_dp_and_intervals, rust_lockfree_spsc, and async_deadlock_audit).

1. 30B–32B & 14B Elite Coder Torture-Test Ledger (8 Configurations · 24/24 Compiler Verified)

24 / 24 PASS (100% Exact)

Config ID	Model & Offload Topology	CUDA VRAM / Splits	Pooled tok/s	Peak (Rust) / Prefill	Speedup vs 32B Base	Compiler Gate
m5_qwen3_coder_30b_moe_ncmoe32_pure	Qwen3-Coder-30B-A3B-Q4_K_M (30.5B / 3.3B active, -ngl 99 --n-cpu-moe 32)	6,463.4 MiB / 162	16.37 tok/s	16.53 / 78.0 tok/s	14.11× (5.37× vs Ollama)	3/3 PASS
d1_qwen25_coder_14b_spec_0_5b	Qwen2.5-Coder-14B-Q4_K_S (-ngl 38/49) + 0.5B Draft (-ngld 99)	6,074.8 MiB / 76	15.89 tok/s	19.17 tok/s (98.7% acc)	13.70× (88.8% pool acc)	3/3 PASS
m2_qwen3_coder_30b_moe_ncmoe32_ngram	Qwen3-Coder-30B-A3B-Q4_K_M (--n-cpu-moe 32 + ngram-simple)	6,463.4 MiB / 162	15.37 tok/s	15.79 / 77.4 tok/s	13.25×	3/3 PASS
m1_qwen3_coder_30b_moe_ncmoe35	Qwen3-Coder-30B-A3B-Q4_K_M (-ngl 99 --n-cpu-moe 35)	5,491.3 MiB / 168	14.92 tok/s	15.07 / 72.5 tok/s	12.86×	3/3 PASS
m4_qwen3_coder_30b_moe_ncmoe31_ultra	Qwen3-Coder-30B-A3B-Q4_K_M (--n-cpu-moe 31 · WDDM spillover test)	6,787.4 MiB / 160	13.04 tok/s	13.21 / 66.8 tok/s	11.24×	3/3 PASS
m3_qwen3_coder_30b_moe_spec_0_6b	Qwen3-Coder-30B-A3B (--n-cpu-moe 36) + Qwen3-0.6B Draft (-ngld 99)	5,689.2 MiB / 170	11.59 tok/s	15.81 tok/s (76.9% acc)	9.99×	3/3 PASS
c5_dense32b_0_5b_ngl20_prio	Qwen2.5-Coder-32B-Q4_K_M (-ngl 20/65) + 0.5B Draft (-ngld 99 --prio 2)	6,013.2 MiB / 40	3.90 tok/s	4.80 tok/s (96.2% acc)	3.36× (85.4% pool acc)	3/3 PASS
c6_dense32b_0_5b_ngl21_ultra	Qwen2.5-Coder-32B-Q4_K_M (-ngl 21/65 + 0.5B · WDDM spillover test)	6,274.8 MiB / 42	3.39 tok/s	4.28 tok/s (94.9% acc)	2.92×	3/3 PASS

EMPIRICAL LAW I · SPARSE MOE DOMINANCE

100% Attention on GDDR7 + Selective CPU MoE

Offloading all 48 self-attention layers + KV cache + 16 MoE layers to RTX 5060 GDDR7 (-ngl 99 --n-cpu-moe 32, 6,463.4 MiB) while streaming only 3.3B active parameters across 32 CPU MoE layers achieves 16.37 tok/s pooled (16.53 tok/s Rust peak, 78.0 tok/s prefill) — 14.11× faster than the 32B dense baseline.

EMPIRICAL LAW II · PARALLEL VERIFICATION AMORTIZES DDR4

Speculative Drafter Helps Dense 32B/14B, Hurts MoE

For dense 32B (13 GB in DDR4-2133), verifying K + 1 draft tokens in a single parallel pass amortizes the ~550 ms RAM sweep across 5–6 tokens (3.90 tok/s pooled, 4.80 peak at 96.2% acceptance). Conversely, on 30B Sparse MoE, multi-token verification triggers additional expert union routing on CPU, reducing speed from 16.37 → 11.59 tok/s.

EMPIRICAL LAW III · WINDOWS 11 WDDM 6.5 GB THRESHOLD

Preventing Silent PCIe Gen 3 x8 Shared-RAM Paging

On an 8,151 MiB Windows 11 GPU, desktop compositor + WDDM reserves ~1.35 GB. Pushing CUDA model buffers from 6,463 MiB (m5) to 6,787 MiB (m4) or 6,013 MiB (c5) to 6,275 MiB (c6) crosses the WDDM paging cliff, silently stalling CUDA kernels over PCIe 3.0 x8 (16.37 → 13.04 tok/s and 3.90 → 3.39 tok/s).

2. Dense 32B Speculative Ablation Progression (c0 → c5 · Qwen2.5-Coder-32B-Q4_K_M)

Stage	Offload & Draft Strategy	Pooled	Peak / Speedup
c0_baseline	18/65 GPU layers, no draft, defaultKV	1.16 tok/s	1.18 tok/s (1.00×)
c1_tuned_q8kv	22/65 GPU layers, Q8_0 KV, ubatch 512	1.26 tok/s	1.28 tok/s (1.08×)
c2_ngram_spec	22/65 GPU layers + zero-VRAM ngram-mod	1.46 tok/s	1.78 tok/s (1.26×)
c3_spec_0_5b	20/65 GPU + Qwen2.5-Coder-0.5B (-ngld 99)	3.58 tok/s	4.36 tok/s (3.08×)
c4_spec_1_5b	17/65 GPU + Qwen2.5-Coder-1.5B (-ngld 99)	3.37 tok/s	4.24 tok/s (2.90×)
c5_0_5b_prio2	20/65 GPU + 0.5B Draft + --prio 2 + draft-max 10	3.90 tok/s	4.80 tok/s (3.36×)

3. Historical 70B Q2_K & 7B Q4_K_M Baseline Ledger (Llama 3.1 70B · 144 Tokens)

Config	GPU layers	tok/s (pooled)	Draft accepted	Output vs baseline
c0_default_fit (baseline)	16 / 81	0.986	n/a	reference
c1_tuned_fit	19 / 81	1.036	n/a	identical
c2_ngram_simple	n/a	1.164	62% (35/56)	diverged on 1 of 3 prompts
c3_draft1b_k4 (1B draft model)	11 / 81	1.678	79% (106/134)	identical
c4_draft1b_plus_ngram	11 / 81	1.411	71% (108/152)	identical
c5_tuned_threads4	n/a	0.799	n/a	identical
m7b_qwen25coder (7B Q4_K_M, different model)	81 / 81	77.610	n/a	n/a

Stress microbenchmark (same RTX 5060 PC): wgpu 2D Tiled GEMM 948.59 GFLOP/s synced (4.67 TFLOP/s async); Dense Q4 GEMV 311.10 μs (313.9 GB/s effective BW); Sparse MoE Q4 GEMV 215.05 μs; AVX2 CPU GEMM 21.32 GFLOP/s; PagedAttention 17.3 M blocks/s with 1.29% peak fragmentation and 0 leaked blocks; DFA mask 78.4 ns. Raw JSON & Markdown captures on GitHub ↗

HARDWARE-SYNCHRONIZED WGPU VULKAN/DX12 KERNELS & MIDDLEWARE TELEMETRY

Audited Systems & Custom WGSL GPU Kernel Matrix

Every GPU kernel measurement in crates/harness-core (wgpu v30 Vulkan/DX12 backend respecting WebGPU baseline maxComputeInvocationsPerWorkgroup = 256 / @workgroup_size(256) and 16×16 tiles) uses strict device-synchronized readback fences (poll(Maintain::Wait)) on the NVIDIA GeForce RTX 5060 (448 GB/s GDDR7) alongside CPU SIMD and safety microbenchmarks.

⚙️ Custom wgpu WGSL Compute Kernels948.6 GFLOP/s Synced

2D Shared-Memory Tiled GEMM:948.59 GFLOP/s (4.67 TFLOP/s async)

Dense Q4 GEMV (16,384 × 4,096):311.10 μs (313.9 GB/s · 70.1% BW)

Activation-Sparse INT4 GEMV (75% sparse):215.05 μs (1.45× over dense Q4)

🧬 Fused 32B SwiGLU, RMSNorm & GGUF Q4_KReal GGUF Verified

Fused 32B SwiGLU FFN (27,648 dim):435.08 μs (gate × SiLU × up)

Fused Workgroup RMSNorm (5,120 dim):766.89 μs (@workgroup_size(256))

Real GGUF Q4_K Tensor Extraction:144B/256w (4.50 bpw) · 290/291 tensors

🧠 Tri-Core Mathematical Subsystems75 Automated Tests

Q4_K Importance-Weighted Quantizer:7.11× compression · imatrix grid search

Online Draft Calibrator (E[N]/T):Dynamic K* = 0..10 (EMA α_k + H(p))

Skeptic's Trench Compiler Verifier:rustc --test -O & Python AST/DP

🛡️ Constrained Decoding & Safety (DFA)100% Valid JSON

JSON Schema Compliance:100% by construction (DFA mask)

DFA Logit Mask Latency:78.4 ns per mask (stress microbenchmark)

Shannon Entropy Trigger:>0.40 nats (Anti-Hallucination Gate)

⚡ Adaptive Speculative Waste (EAGLE)-94.8% Waste Cut

Speculative Token Waste Cut:-94.8% (Entropy-Gated Guard)

Draft Token Acceptance:98.7% (14B) · 96.2% (32B) · 79% (70B)

Effective Decode Speedup:3.36x (32B + 0.5B) · 1.70x (70B + 1B)

📦 Paged KV Block ManagerZero Leaks

Block Allocation Throughput:17.3 Million blocks/s

KV Cache Fragmentation:1.29% peak (stress test)

Memory Leak Invariant:50,000 / 50,000 blocks freed after churn

🧮 Pure-Rust SIMD & Tensor Kernels21.32 GFLOP/s

AVX2 GEMM 1024x1024 FP32:21.32 GFLOP/s (Rayon, 4-wide AVX2)

Zero-Copy Loader Speed:Memory-Mapped SafeTensors / GGUF

Runtime Python/GIL Overhead:No Python runtime (compiled Rust)

💾 Temporal Layer Streaming (PoC Pipeline Simulation)PoC Simulation

70B Peak VRAM Allocation:Scheduling model only; measured 70B uses 7.4 - 7.6 GB via llama.cpp

PCIe DMA Ping-Pong Overlap:Not measured (simulation)

70B Streaming Decode:0.99 - 1.68 tok/s (llama.cpp, measured)

🚀 Resident & Hybrid MoE In-VRAM Inference77.6 / 16.37 tok/s

30B Sparse MoE / 7B Q4_K_M:16.37 tok/s (30B MoE) · 77.6 tok/s (7B)

30B MoE Prompt Prefill:78.0 tok/s (-ngl 99 --n-cpu-moe 32)

LIF Spiking Attention Sparsity:50.0% FLOPs pruned (θ ≥ 0.35, report)

PURE RUST 1.85+ (2024 EDITION) MODULAR WORKSPACE (15.7K LOC · 75 AUTOMATED TESTS)

The 11-Crate Systems Architecture

No garbage collector (Rust). Bounds-checked, compiler-verified, and leak-checked by the repository stress suite.

crates/harness-core4.4k LOC

Tensor abstractions, wgpu v30 (Vulkan/DX12/Metal) WGSL compute kernels (@workgroup_size(256) <= maxComputeInvocationsPerWorkgroup: Sparse Q4 GEMV 215.05 μs, Dense Q4 GEMV, Batched Q4 GEMM, Top-8 MoE Router, 5:1 & 8:1 GQA FlashAttention, RoPE, SwiGLU, RMSNorm), and multi-OS hardware profiles (plan_coder_offload)

crates/harness-loader0.7k LOC

GGUF v2/v3 binary header, metadata & tensor unpacker (extract_tensor_f32 for F32/F16/Q8_0/Q4_K super-blocks) plus zero-copy mmap SafeTensors loader with path traversal guards

crates/harness-quant1.4k LOC

Bit-exact 256-weight GGUF Q4_K (BlockQ4K, 144B = 4.50 bpw; ~4.80–4.85 bpw in mixed Q4_K_M, imatrix WLS refinement, fused dot_f32), OCP FP8 E4M3FN (32-group), 64-group Voronoi NF4, and imatrix Q4_0

crates/harness-attention1.2k LOC

Decode-aware FlashAttention-3, Quantized Int8 (Q8_0) & Int4 (Q4_0) Paged KV Cache (1.29% peak fragmentation, 0 leaks), LIF Spiking Attention, DeepSeek MLA (512d latent + 64d RoPE), and RadixPrefixCache

crates/harness-models0.6k LOC

Qwen3-Coder-30B-A3B Sparse MoE (48L, 32Q/4KV 8:1 GQA, 128x8 experts, 256K ctx), Qwen2.5-Coder-32B/14B/0.5B (64L/48L, 40Q/8KV 5:1 GQA, 128K ctx), DeepSeek-V3/R1 (MLA + 256+1E sigmoid MoE), and Llama 3.3 70B / Llama 4

crates/harness-pipeline2.1k LOC

OnlineDraftCalibrator (cross-entropy SGD logit bias distillation), RepoSuffixDrafter (O(1) trigram tree), CodeQualityEngine (Rust lifetime 'a aware AST validator), TestTimeCompute (Best-of-N), and layer streaming

crates/harness-safety0.7k LOC

Pushdown automaton & precompiled DFA constrained decoding (78.4 ns/mask), Cortical lateral inhibition, Shannon entropy monitoring (>0.40 nats), AgentCircuitBreaker, and observation compactor

crates/harness-rag0.2k LOC

Hippocampal fast-slow dual memory (CLS theory): volatile episodic buffer + cortical engram consolidation (93.8% context saved in report)

crates/harness-server1.6k LOC

Axum HTTP/SSE OpenAI-compatible API server (/v1/chat/completions, /v1/models, /metrics, /proof) prioritizing port 8089 bare-metal 30B–32B Coder runner

crates/harness-cli1.8k LOC

Unified CLI toolchain: coder32b, gpu-bench, code-check, reason, moe-bench, speculative, stress, tune, doctor, verify, compare7b, stream70b, serve, chat, and mcp

crates/harness-mcp0.9k LOC

Model Context Protocol (MCP) stdio server for native tool calling, code verification, and context in Cursor, Antigravity, Windsurf, and VS Code

🔌

Model Context Protocol (MCP) Setup

Connect HARNESS directly to Cursor, Antigravity, Windsurf, or VS Code via harness-mcp over stdio for local offline coding and agentic execution:

.cursor/mcp.json

{
  "mcpServers": {
    "harness": {
      "command": "cargo",
      "args": ["run", "-p", "harness-mcp", "--release"],
      "env": {
        "RUST_LOG": "info"
      }
    }
  }
}

🚀

Developer Quickstart CLI

Clone and run the complete pure-Rust toolchain, wgpu GPU kernel suite, and 30B–32B Coder launcher:

terminal bash / powershell

# 1. Clone & run all 75 workspace tests
git clone https://github.com/ModernOps888/harness.git && cd harness
cargo test --workspace

# 2. Run 9-stage hardware-synchronized wgpu GPU kernel benchmark
cargo run --release -p harness-cli -- gpu-bench

# 3. Launch 30B Sparse MoE Coder (16.37 tok/s) or 32B Speculative
powershell -ExecutionPolicy Bypass -File scripts/launch_qwen32b_coder.ps1 -Mode MoE30B

# 4. Run 32B/30B topology planner & Skeptic's Trench compiler gate
cargo run --release -p harness-cli -- coder32b
cargo run --release -p harness-cli -- code-check

# 5. Run hardware stress suite, invariants & 70B DMA simulation
cargo run -p harness-cli -- stress
cargo run -p harness-cli -- verify
cargo run -p harness-cli -- stream70b

DEEP ARCHITECTURAL ANSWERS

Frequently Asked Technical Questions

How does HARNESS achieve 16.37 tok/s on a 30.5B-parameter Coder model with an 8GB RTX 5060?

By pairing Qwen3-Coder-30B-A3B-Instruct (Q4_K_M, Apache 2.0) — which has 30.5B total parameters across 48 layers (32Q/4KV = 8:1 GQA, 256K native context) but activates only 8 of 128 routed experts (3.3B active parameters) per token — with selective hybrid MoE offloading (-ngl 99 --n-cpu-moe 32). All 48 self-attention layers, the 32K active KV cache (78 MiB), router gates, and 16 MoE layers remain 100% resident in RTX 5060 GDDR7 VRAM (6,463.4 MiB @ 448 GB/s, safely below the 6.5 GB Windows 11 WDDM spillover cliff), while only the active 3.3B expert slices stream across 32 CPU MoE layers. This yields 16.37 tok/s pooled (16.53 tok/s Rust peak, 78.0 tok/s prefill) — a 14.11× speedup over the 32B baseline and 5.37× faster than default Ollama.

Why does speculative decoding accelerate Dense 32B by 3.36× (3.90 tok/s) but slow down 30B Sparse MoE?

In a dense 32B model (Qwen2.5-Coder-32B-Instruct Q4_K_M, 64 transformer layers / 65 offloadable blocks, 40Q/8KV = 5:1 GQA), 45 of 65 blocks (~13 GB) sit in DDR4-2133 host RAM and must be read on every forward pass (~550 ms). Using a 100% GPU-resident 0.5B drafter (-ngld 99 --prio 2) lets the 32B target verify K + 1 draft tokens in a single parallel batch pass, amortizing the DDR4 read across 5–6 accepted tokens (96.2% Rust draft acceptance, 4.80 tok/s peak, 3.90 tok/s pooled). Conversely, in a Sparse MoE model, verifying multiple draft tokens in parallel activates the union of distinct top-8 expert sets across tokens, multiplying CPU memory reads and reducing throughput from 16.37 tok/s to 11.59 tok/s.

What is the Tri-Core Architecture (Q4_K imatrix, OnlineDraftCalibrator, and Skeptic's Trench)?

HARNESS integrates three mathematical subsystems in pure Rust: (1) Q4_K Importance-Weighted Super-Block Quantizer (crates/harness-quant, BlockQ4K) packing 256 FP32 weights into 144-byte GGUF super-blocks (4.50 bpw, 7.11× compression vs FP32; ~4.80–4.85 bpw in mixed Q4_K_M) using activation importance I_i = E[x_i²] and alternating weighted least-squares scale search; (2) Online Draft Calibrator (crates/harness-pipeline/src/draft_aligner.rs) solving K* = argmax_K E[tokens(K)] / T(K) online via cross-entropy SGD bigram logit-bias updates, per-position EMA acceptance tracking, and Shannon entropy gating; and (3) Skeptic's Trench Compiler Verifier (crates/harness-pipeline & benchmarks/bench_elite_coder.py) executing real rustc --test -O and Python subprocess assertions across all 24/24 benchmark outputs.

How does 70B Temporal Layer Streaming work on consumer GPUs?

HARNESS implements a double-buffered ping-pong DMA pipeline architecture for layer streaming. While layer Ln executes in GPU VRAM (Slot 0), layer Ln+1 is prefetched asynchronously from pinned host RAM (Slot 1) over PCIe Gen4. Because only two layer buffers reside on the device at any time, this is a scheduling model (simulation); real 70B throughput on the measured PC is 0.99 - 1.68 tok/s via llama.cpp partial offload (see Verified Benchmarks).

What are the biological mechanics behind LIF Spiking Attention?

Standard attention evaluates all-to-all dense dot products (O(N²) FLOPs). HARNESS models each token key activation as a biological membrane potential with Leaky Integrate-and-Fire (LIF) dynamics. Keys whose membrane potential fails to breach dynamic threshold θ ≥ 0.35 emit zero spikes and are bypassed completely during matrix multiplication, pruning 50.0% of dense attention FLOPs in the repository report.

How does Hippocampal Fast-Slow Dual Memory (CLS Theory) solve the KV Cache Wall?

In standard inference, KV cache size grows linearly with sequence length (O(N) memory), exhausting VRAM on long documents. Based on Complementary Learning Systems (CLS) neuroscience, HARNESS pairs a volatile episodic buffer (recent tokens in high-resolution FP8/Paged KV blocks) with a cortical consolidator. During context pauses, multi-layer KV states are projected into low-rank sparse engrams, saving 93.8% of context memory in the repository report across 128k context windows (complementing DeepSeek MLA's 93.3% latent KV compression).

Why is pure Rust superior to PyTorch or C++ for LLM runtime execution?

PyTorch incurs Python interpreter GIL overhead, GC pauses, and expensive cross-language pointer wrapping. Pure C++ runtimes are vulnerable to memory unsafety and buffer overflows. Rust 1.85+ (2024 Edition, compatible through Rust 1.99) enforces compile-time ownership, deterministic zero-cost abstractions, portable wgpu v30 Vulkan/DX12/Metal shader execution, zero-copy SafeTensors/GGUF memory mapping via mmap, and Rayon parallel concurrency with zero data races.

How does local sovereign execution on HARNESS compare to 2026 frontier cloud API costs?

Frontier cloud coding APIs typically bill $3–$15 per 1M input tokens and $15–$75 per 1M output tokens while transmitting proprietary source code over external networks. Running Apache 2.0 Qwen3-Coder-30B-A3B-Instruct locally via HARNESS on an 8 GB RTX 5060 delivers 16.37 tok/s (~58,900 output tokens/hour) at £0 / $0 marginal token cost with 100% air-gapped IP sovereignty, deterministic 78.4 ns DFA schema enforcement, and local rustc --test -O verification.

100% Free & Open Source. Zero Sign-Up Required.

HARNESS is licensed under the MIT License. Run it locally, inspect the 11 crates, integrate it into your IDE via MCP, or deploy it into production with Docker.

🛡️ Sovereign AI Architecture & Audits →★ Fork & Star on GitHub 🏗️ Explore Harness Engineering Academy 🦀 Explore Rust Systems Academy

Break the Memory Wall.30B–70B on an 8GB GPU. Measured, not claimed.