loading
loading
Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA DGX Spark, profiling showed that decoding was memory-bound: reading model weights drove execution time. Two operations stood out—the mixture-of-experts (MoE) computation and the output-layer projection that produces scores across the model’s vocabulary. Those findings pointed to two ways to improve performance: reduce the weight traffic in each decode step and generate more tokens per verification round. This post examines three techniques that address those costs: - Humming MXFP4 MoE kernels to improve execution of the routed-expert computation. - Online FP8 quantization of the output layer, or LM head, to reduce its weight-read volume. - Multi-token prediction (MTP) speculative decoding with NEXTN to draft and verify multiple tokens together. This post explains how each optimization works, measures its effect, and examines an integration issue that can make the combined configuration slower than the baseline. Ling-3.0-flash serves as the implementation example, with code and deployment instructions in the Ling cookbook. Model and baseline configuration The evaluation used Ling-3.0-flash-fp4, a mixture-of-experts model with 124 billion total parameters and 5.1 billion active parameters per token. The model has 42 layers: 35 Kimi Delta Attention (KDA) linear-attention layers and seven multi-head latent attention (MLA) layers. Each MoE layer routes each token to eight of 512 routed experts, alongside one shared expert. For agentic workflows, Ling-3.0-flash provides a native 256K-token context window, expandable to 1M tokens. Compared with the previous Ling flash model, it improves long-horizon task stability, tool-calling accuracy, and harness compatibility. Developers can enable reasoning for complex agent tasks or disable it for latency-sensitive interactions. Improved spatial understanding supports constructing physical-scene grids and reasoning about relative positions in coding tasks. The baseline followed the official cookbook’s W4A8 configuration: FlashInfer MXFP4 MoE kernels, NVIDIA CUTLASS dense GEMM, and the –enable-fp32-lm-head option. The LM head projects a hidden dimension of 2,560 to a vocabulary of 157,184 tokens. For standard MXFP4 deployment instructions, follow the MXFP4 deployment notebook. Optimization 1: Humming MXFP4 MoE backend The first optimization replaces the flashinfer_mxfp4 MoE execution backend with humming. Humming is an independent kernel library available through the humming-kernels pip package. Unlike the FlashInfer CUTLASS path, Humming uses NVIDIA CUDA Runtime Compilation (NVRTC) to generate and compile CUBINs at runtime for the actual M/N/K dimensions, data types, and GPU architecture. It caches compiled binaries in ~/.humming/cache. This specializes kernels for Ling-3.0-flash shapes, including its relatively narrow moe_intermediate_size of 768. It also compiles at a finer granularity: an individual kernel completes in seconds, whereas the FlashInfer CUTLASS path uses ninja to compile 97 targets and takes on the order of tens of minutes. Performance is reported as time per output token (TPOT) and decode throughput in tokens per second. Optimization 2: Online FP8 LM head quantization In the checkpoint, the LM head is stored in BF16 and listed in modules_to_not_convert, so it is not quantized during loading. Its vocabulary GEMM is 157,184 × 2,560, and the entire weight block must be read for each generated token. Under a bandwidth bottleneck, it is therefore a high-value target for compression. The optimization applies a dynamic FP8 quantization configuration only when constructing the LM head. The rest of the model continues to load with the checkpoint’s FP8/MXFP4 configuration. Moving the LM-head weights from BF16 to FP8 halves the read volume for this layer. The configuration is enabled with SGLANG_ENABLE_FP8_LM_HEAD=1. This setting is mutually exclusive with the baseline –enable-fp32-lm-head option, which outputs logits in FP32 and uses a slower vocabulary-GEMM path. Optimization 3: MTP speculative decoding with NEXTN The first two optimizations reduce the cost of each decode step. Speculative decoding follows a different path: it reduces the number of decode steps. Ling-3.0-flash includes a NEXTN prediction layer (num_nextn_predict_layers: 1), which can be used directly as a draft model without additional training or downloads. The configuration uses three draft steps and a four-token verification window. In each round, the draft predicts three tokens in sequence, and the target verifies four positions at once, accepting the correct prefix. The benefit depends on two quantities: accept length, the average number of accepted tokens per round, and the verification-cost ratio, the cost of one verification step relative to a normal decode step. The measured ratio was approximately 2.26. The principal source of verification cost was not attention computation; at batch size 1, KV reads were small relative to weight reads. Instead, the cost came from MoE expert activation: a normal decode activates eight experts for one token, while verification processes four tokens and can activate up to 32 experts. Note: When online FP8 LM-head quantization and MTP are enabled together, the draft model must share the entire LM-head module with the target, rather than only its weight tensor. The original Ling NEXTN implementation didn’t expose this interface. Without it, the framework silently falls back: the draft receives FP8 weights but interprets them as BF16, accept length drops to 1.00, and throughput becomes lower than a configuration with no optimizations. Sharing the full LM-head module through this interface restores normal token acceptance. Follow the Ling cookbook to enable FP8 LM-head quantization and MTP together. End-to-end results on GSM8K The component-level results above use a synthetic 32K-input/2K-output workload. Synthetic workloads help isolate individual gains, but their repetitive outputs and simple sampling systematically overestimate speculative-decoding benefits. The evaluation also included a strict end-to-end comparison across all 1,319 GSM8K questions. The evaluation used five-shot prompting, with demonstrations from the training set and the complete test set for evaluation, temperature 0.6, top_p 0.95, top_k 29, max_tokens 32,768, and chain-of-thought enabled. Apart from the three optimizations, both evaluations aligned all variables: sampling parameters, parser, five-shot protocol, random seed, and memory allocation. Time to first token (TTFT) measures the delay before the first output token. Why synthetic and real workloads differ - Synthetic outputs are more repetitive, making draft predictions easier to accept. Accept length was 3.90 for the synthetic workload, compared with 2.68 for GSM8K. - Sampling parameters differ. The synthetic test used greedy decoding, while the real evaluation used temperature=0.6, top_p=0.95, and top_k=29. - Output-length distributions differ. The synthetic workload fixes output at 2K tokens, while the GSM8K median output is approximately 310 tokens. This spreads the fixed overhead reductions from the first two optimizations over a shorter output. When evaluating these optimizations on DGX Spark, use a workload that reflects your intended deployment. The 185.1% throughput gain measured in the synthetic test does not establish the gain for other workloads. Conclusion Across the full GSM8K evaluation, the three optimizations increased decode throughput by 42.5%, reduced mean end-to-end latency by 24.5%, and produced no statistically detectable accuracy change. Their mechanisms do not overlap: Humming and the FP8 LM head reduce the cost of each step, while MTP reduces the number of steps, so the techniques can be combined. Stacking, however, is not automatic. Each optimization worked independently, but enabling all three required the draft model to share the full LM-head module. Otherwise, the system silently degrades to a configuration slower than the unoptimized baseline. During deployment, validate that every optimization is active at startup; this is more reliable than inferring configuration state from later performance numbers.
document fetched 2026-10-28 02:51 · canonical https://developer.nvidia.com/blog/3-ways-to-boost-moe-inference-on-nvidia-dgx-spark