loading
loading
humans& Beats FlashInfer's Best MoE Kernel by 25% on Blackwell GPUs Humans& rebuilt the NVFP4 decode kernel on Blackwell by keeping weights at 4 bits while promoting activations to BF16, gaining both speed and accuracy. - Humans& accelerated NVFP4 MoE decode by up to 25% while increasing activation precision from 4 to 16 bits. - End-to-end: 10% more throughput and 7% faster TPOT on GLM-5.2 across eight B300 GPUs. - Weights stay in NVFP4, activations stay BF16, removing activation quantization from the critical path. - Weight dequantization runs ahead using Blackwell TMEM, async MMA, and Programmatic Dependent Launch. - Training–rollout mismatch drops to BF16 level, stabilizing low-precision RL on Miles. - Shipped in FlashInfer, SGLang, and Miles with a single env-var toggle. Engineers at humans& accelerated Mixture-of-Experts decode on NVIDIA Blackwell GPUs by keeping weights in 4-bit NVFP4 while retaining activations in 16-bit BF16. Their W4A16 kernel outperformed FlashInfer’s optimized W4A4 MegaMoE path by up to 25%, showing that higher activation precision can reduce execution time when weight traffic dominates the workload. The reported percentages apply to this workload and baseline. The advantage is largest at low concurrency and narrows as each loaded weight serves more tokens. Decode rewards compact weights Autoregressive decode generates one token at a time. In a Mixture-of-Experts model, each token is routed to a small subset of expert networks, whose weight matrices must be fetched from high-bandwidth memory, or HBM. Those matrices are much larger than the activation vector, so memory transfers often determine latency while the Tensor Cores retain unused arithmetic capacity. Low precision provides two separate benefits: compact data requires less memory bandwidth, and low-precision arithmetic can run at higher throughput. Decode still benefits from compact weights, but its idle arithmetic capacity leaves little value in performing the matrix multiplication with 4-bit activations. On a GLM-5-shaped layer, the team measured BF16 activations at less than 1% of combined activation and weight traffic. Retaining BF16 activations changes the dependency graph in two useful ways: - Activation quantization leaves the critical path. The next matrix multiplication can consume the previous layer’s BF16 output directly instead of waiting for NVFP4 packing. - Weight dequantization starts earlier. Stored weights are static, so the kernel can begin converting them before the preceding computation finishes producing activations. Blackwell hides the conversion work The kernel overlaps weight dequantization with data movement and Tensor Core computation by combining four Blackwell capabilities: - Native NVFP4 conversion instructions convert packed E2M1 values and E4M3 scales without a long software instruction sequence. - Tensor Memory, or TMEM, holds FP32 accumulators and dequantized weights outside the conventional register file. - Single-thread instruction issue for Tensor Memory Accelerator transfers and matrix multiply-accumulate operations leaves more warps available for dequantization. - Asynchronous execution allows transfers, dequantization, and Tensor Core work to proceed concurrently. Programmatic Dependent Launch lets the second expert linear layer, FC2, allocate TMEM and begin dequantizing weights before the first layer, FC1, completes. This scheduling removes weight preparation from part of the serial path between the two matrix multiplications. L2 cache hints protect the data needed across that boundary. The kernel applies evict_first to streaming FC1 weights that have already been consumed and evict_last to activations that FC2 will soon read. The design depends on Blackwell-specific features such as TMEM, native conversions, and dependent launch scheduling. BF16 preserves each local product NVFP4 stores blocks of 16 E2M1 values with a shared E4M3 scale. Dequantization multiplies each low-precision value by that scale. An MXFP8 intermediate could offer greater Tensor Core throughput, but E4M3 cannot represent every resulting product exactly because it lacks enough significant bits. The product 1.5 × 1.125, for example, equals 1.6875. BF16 represents that value exactly, while E4M3 must round it to a nearby value. The BF16 path therefore avoids additional rounding during local weight dequantization. Reinforcement learning rollouts are sensitive to these differences because rounded values can alter logits, action probabilities, and the effective sampling policy. The team reports that using BF16 activations with the same NVFP4 weight recipe as training reduces training-rollout disagreement to the BF16 level. Dedicated warps keep the pipeline busy A CUDA warp contains 32 threads, and a warpgroup combines four warps. The split MoE kernel assigns four warpgroups, or 16 warps, to each cooperative thread array. Two warpgroups handle weight dequantization exclusively, processing complete 16-value NVFP4 blocks and converting each shared E4M3 scale once. The MegaMoE variant adds a fifth warpgroup for token dispatch, bringing the total to 20 warps. Its persistent kernel combines token delivery, FC1, FC2, and the final top-K expert output merge. Despite dequantizing weights during execution, the implementation sustains more than 98% utilization of the FC1 Tensor Core pipeline. The packed PTX instructions, expressed in NVIDIA’s low-level GPU instruction language, perform the conversion and multiplication directly: cvt.rn.bf16x2.e4m3x2 scale_bf16x2, scale_pair; cvt.rn.bf16x2.e2m1x2 weight_bf16x2, packed_pair; mul.rn.bf16x2 dequantized_bf16x2, weight_bf16x2, scale_bf16x2; The lead narrows as concurrency grows The serving test used an eight-GPU NVIDIA B300 node running GLM-5.2 with a SemiAnalysis InferenceX-style workload of roughly 1,000 prompt tokens and 8,000 generated tokens. At eight concurrent requests, W4A16 increased generation throughput by about 10% and reduced time per output token, or TPOT, by 7%. Low-concurrency decode gains the most because each fetched expert weight serves relatively few tokens, leaving the workload strongly memory-bound. Higher concurrency reuses each loaded weight across more tokens and increases arithmetic work per transfer. Under those conditions, the higher FP4 compute throughput of W4A4 narrows the W4A16 lead. One layout supports two serving paths Disaggregated deployments can assign precision by serving phase because both paths consume the same prepared NVFP4 weight layout. Prefill workers can use W4A4 while processing prompts in larger, more compute-intensive batches. Decode workers can enable the W4A16 path with one SGLang environment variable: SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16=1 The shared layout removes the need for weight repacking or separate checkpoints. Reinforcement learning systems can also use the same NVFP4 weight preparation during training and rollout while retaining BF16 activations where numerical agreement matters. Where the code landed The optimized split and MegaMoE kernels have been contributed through a FlashInfer pull request, while serving support landed through an SGLang pull request. The Miles training path includes fused NVFP4 quantize-dequantize kernels and the bit-exact 4over6 scale-selection algorithm. The same kernels support reinforcement learning for the Ampersand collaborator model and production inference for the Boba product. Their design assigns low precision per operand: NVFP4 weights preserve the bandwidth savings that decode needs, while BF16 activations remove quantization dependencies and retain exact local dequantization products.
document fetched 2026-10-24 15:22