Nvidia Rubin Delivers 4.2x Cost Efficiency in Inference

SemiAnalysis benchmark data reveals major throughput gains on a 180-billion-parameter MoE model running SGLang on new HBM4 silicon.

Nvidia Rubin hardware delivers 4.23 times more performance per dollar than previous-generation Blackwell GB300 NVL72 systems during AI inference workloads. The figures were published on Wednesday (October 14, 2026) by independent semiconductor research firm SemiAnalysis, based on preliminary InferenceX benchmarks using the open-source SGLang runtime.

Benchmarking the 180B MoE Model

The benchmark evaluated Qwen3.8 Flash Next, a Mixture of Experts (MoE) architecture with 180 billion total parameters that activates only 6 billion parameters per token. Engineers from Nvidia, the SGLang project, and RadixArk developed the underlying software optimizations, crediting Nick Comly, Alex Sette, and Kedar Potdar.

To establish the 4.23x cost-performance multiplier, SemiAnalysis modeled total cost of ownership (TCO) across complete Rubin NVL72 racks against Blackwell GB300 NVL72 systems, holding per-user token generation speeds constant. Under this setup, the system maintains target response times while substantially cutting the operational cost per request.

How the Nvidia Rubin Architecture Accelerates Inference

Software tuning played as critical a role as the new silicon itself. The SGLang engineering team documented that native FP8 precision kernels with Multi-head Latent Attention (MLA) execute up to 20 percent faster at batch size 1 across long 128k context windows.

The team also accelerated speculative decoding verification in KDA by 20 percent and introduced tail kernel fusion for MoE layers, utilizing Rubin’s 327 KiB of shared memory per Cooperative Thread Array (CTA). At the hardware level, the platform integrates HBM4 high-bandwidth memory, NVLink 6 interconnects, and Nvidia Dynamo disaggregated scheduling, which isolates prefill (prompt processing) from decode (token generation) on dedicated nodes.

Production Numbers and Technical Caveats

Field results from production environments showed similar gains. Cloud deployment testing by Cognition revealed roughly 4.8 times higher token throughput per GPU when running its Devin SWE-2 software engineering agent on CoreWeave infrastructure, measured against earlier GB200 systems at identical decode speeds.

The report also emphasizes important caveats. The 4.23x improvement represents a preview test on a sparse MoE model with a modest active parameter count. Benchmark submissions for dense enterprise models and larger MoE variants, such as the GLM-5.3 series, remain pending. Furthermore, commercial purchase pricing and cloud rental rates for Rubin racks are not yet finalized, while rival runtimes like vLLM and TensorRT-LLM have yet to release comparable numbers on early hardware.

The Race for Token Economics

As data center operators confront tight thermal limits and power availability constraints, raw compute output is no longer the sole purchasing metric. For providers supporting autonomous agents that process tens of thousands of context tokens continuously, the financial and energy cost per million generated tokens now determines whether commercial deployments are viable.