loading
loading
AI developers aren’t just running a single model anymore—they’re orchestrating multiple models in parallel, chaining inference calls, and experimenting with frontier-class architectures that demand hundreds of gigabytes of memory. NVIDIA DGX Spark was built to bring AI development to the desk, and now you’ll be able to link even more Spark nodes together. At GTC Berlin, NVIDIA announced official support for eight-node DGX Spark clusters, scaling unified memory to 1 TB and opening the door to frontier-class models, multi-agent workloads, and near-linear pre-training—all running locally. NVIDIA Sync makes it easy to cluster. Its Cluster Assistant handles setup and device management. Support for eight-node clusters in NVIDIA Sync is coming by the end of the year. Read on for benchmark data on how inference, orchestration, and training scale as the cluster grows. Run larger, newer frontier-class models The biggest benefit of an eight-node cluster is memory. A single DGX Spark provides 128 GB of unified memory; four nodes reach 512 GB; eight nodes reach 1 TB. That capacity opens the door to a class of models that previously wouldn’t start on a four-node system, including the full GLM 5.3 (753B) model from Z.ai. Beyond simply fitting these models, the cluster is designed to serve them interactively. As an early example, we ran GLM 5.3 Flash across an eight-node cluster, and measured roughly 80 tokens per second (TPS) in generation, suitable for interactive local development against a frontier-class mixture-of-experts model. The full GLM 5.3 (753B) checkpoint runs across the 8x 128 GB cluster, making it one of the largest open models you can serve locally. Faster collectives with a Spark-optimized backend Tensor-parallel inference spends a meaningful fraction of its time in collective operations such as all-reduce. On switch-connected clusters, we have been evaluating a Spark-optimized collective backend, RoCEnante, that reduces collective latency for the small-message, low-concurrency, modest-rank-count pattern typical of tensor-parallel decode on local AI hardware. This backend implements an all-to-all exchange followed by local reduction. Each rank shares its input with every other rank, then adds the inputs locally. A single GPU kernel coordinates the exchange and reduction. The main benefit is fewer communication steps. Ring all-reduce passes data between neighboring ranks over multiple rounds. This approach exchanges data directly with all peers, avoiding that chain of forwarding. For small messages and a modest number of ranks, this can reduce latency. The tradeoff is more data movement. Each rank sends its full input to every peer and reduces all the inputs itself. As messages or number of ranks (nodes) grow, that extra traffic becomes more expensive, making ring all-reduce’s bandwidth efficiency increasingly attractive. If you need to support high concurrency, larger clusters, or ring topologies, NVIDIA Collective Communications Library (NCCL) would perform better. In practice we see roughly 5% higher prefill and 21% higher decode throughput at four nodes (NVFP4), and about 18% higher decode throughput at eight nodes on the non-quantized model, consistent with the latency-versus-bandwidth tradeoff above. The backend is merged into FlashInfer and falls back transparently to NCCL if its startup test fails. The backend applies only to switch-connected topologies, because it relies on direct point-to-point exchange between every node. It is not a replacement for NCCL, which remains the supported default and the better choice as cluster size and concurrency grow. The original one-shot all-reduce approach was contributed by the community (Jason Cook/el8). Two, four, eight: Scaling the cluster Low-latency NVIDIA ConnectX-7 NICs and RDMA on Converged Ethernet (RoCE) communication let DGX Spark nodes exchange tensors efficiently, so you can scale a model across more nodes to fit it in memory and to improve throughput. To isolate cluster scaling behavior and remove context-dependent behavior, we benchmarked a dense Qwen3.8-27B (NVFP4) model without speculative decoding/MTP across one to eight nodes: Two patterns stand out. First, decode throughput scales strongly with node count, from about 12 TPS on one node to 52 TPS on eight nodes. Second, the best topology depends on the workload: A physical ring delivers excellent decode throughput for a single model spread across all nodes. A switch is needed once you want to split the cluster into independent groups, or run workloads that require direct any-to-any connectivity between nodes. To try this yourself, this networking guide walks through cluster setup starting from a two-node configuration and scaling up. Switchless ring topology and setup simplicity Not every cluster may need a switch. DGX Spark nodes can be wired directly together in a ring, which lowers cost and simplifies setup—an attractive option for enthusiasts and small deployments. Within a four-node DGX Spark deployment, NCCL defaults to ring-based algorithms for tensor-parallel communication, allowing a switchless ring topology to align perfectly with traffic flows and offer impressive throughput at lower expense. As Table 2 shows, the four-node ring performs well in prefill and decode throughput for this workload. The picture changes as you scale or specialize: - Eight nodes or split allocations: If you want to split a cluster into, say, two groups of four, a ring cannot express that topology; you need a switch. - All-to-all and traditional HPC workloads: Anything that depends on point-to-point patterns (for example, MPI-based clustering) requires a switch; a ring would pay a hop penalty. - The optimized collective backend: The Spark-optimized backend described above requires direct point-to-point connectivity between every node, so it applies to switched configurations only. In short: Reach for a switchless ring when you want the simplest, lowest-cost four-node inference box, and choose a switch when you need to scale up to eight nodes, split the cluster, or run communication patterns that a ring cannot serve. Multi-model orchestration with Switchyard Not every request needs your largest model. On a shared eight-node cluster, you can get substantially more aggregate throughput by routing easy requests to a small, fast model and reserving a larger model for the hard ones. NVIDIA NeMo Switchyard makes this practical: A lightweight classifier inspects each request (or session) and routes it to the appropriate model pool, with a load balancer distributing traffic across replicas. Because this layout splits the eight nodes into independent model pools, Switchyard runs on a switch-connected cluster, which provides the any-to-any connectivity needed to route requests across replicas. In our prototype, we used OpenBMB’s MiniCPM5 2B as the classifier and execution model for simple tasks and Qwen3.8 Flash Next for complex ones. On a standard-plus-hard tool-use benchmark, the routed agent matched the quality of the Qwen-only configuration while freeing capacity on the cluster. On the 92-case tool-use benchmark (including hard cases), the routed agent scored 91/100, matching the Qwen-only configuration. Because MiniCPM serves roughly an order of magnitude more requests per second than Qwen per replica, shifting part of the traffic to the smaller model raises the throughput ceiling of the whole cluster. Allocating three Qwen replicas (two Spark nodes each) and two MiniCPM replicas (one Spark each) across the eight nodes, we get: - ~1.5x aggregate throughput over a Qwen-only baseline when half of requests route to Qwen. - ~3.0x aggregate throughput when a quarter of requests route to Qwen and the rest to MiniCPM. A routing policy is a simple YAML file, and NVIDIA provides a sample policy as a starting point; developers can tune the split and classifier to their own workload. Pre-training from scratch: near-linear scaling with nanochat Training behaves differently from inference. When each node holds a full copy of the model, the only cross-node communication is gradient synchronization once per step, so clustering scales close to linearly. To demonstrate, we pre-trained a nanochat model from scratch (about 896.5M parameters, 20 transformer layers, hidden dimension 1,280- and 2,048-token sequences) and measured training throughput as we added nodes. Eight nodes deliver a 7.79x speedup at 97.4% efficiency, the kind of near-linear scaling that makes an eight-node Spark cluster a credible platform for training small models from scratch, not just serving large ones. It is also a natural on-ramp workflow: Prototype and pre-train locally, then scale out as the model grows. To reproduce the nanochat pre-training experiment or adapt it to your own model, the full code is available on GitHub. Get started with multi-system DGX Spark Eight-node DGX Spark clusters give you the memory to load frontier-class models, the flexibility to orchestrate several models at once, and the efficiency to train small models from scratch, all on the desk. To learn more, check out:
document fetched 2026-10-24 11:20 · canonical https://developer.nvidia.com/blog/running-frontier-class-ai-models-across-8-nvidia-dgx-spark-nodes