NVIDIA/Megatron-LM core_v0.19.0: NVIDIA Megatron Core 0.19.0
extract
#### MoE * **Quantile Balancing router:** adds auxiliary-loss-free load balancing using per-expert routing biases derived from quantile estimates ([\#5349](https://github.com/NVIDIA/Megatron-LM/pull/5349)). * **Fused shared-expert MLP:** enables shared experts through grouped GEMM, providing the path for fused GEMM, SwiGLU, and quantized execution ([\#5604](https://github.com/NVIDIA/Megatron-LM/pull/5604)). * **Packed-sequence MoE dispatch:** adds THD sequence-packing support to the DeepEP and HybridEP dispatcher backends ([\#5008](https://github.com/NVIDIA/Megatron-LM/pull/5008)). * **MoE routing analysis:** adds training/inference routing traces plus expert-concentration and one-layer-ahead predictability analysis tools ([\#5220](https://github.com/NVIDIA/Megatron-LM/pull/5220)). #### Model Architecture and Support * **HybridModel:** next-gen model class. * `GPTModel` to `HybridModel` guide [published](https://docs.nvidia.com/megatron-core/developer-guide/nightly/user-guide/hybrid-model-migration.html). * Added ability to directly load `GPTModel` checkpoints ([\#5792](https://github.com/NVIDIA/Megatron-LM/pull/5792)). * Part 1 of the [design doc](https://github.com/NVIDIA/Megatron-LM/issues/4620) is complete, part 2 will be completed for 0.20.0. * Deepseek v4 migrated to HybridModel on the `dev` branch ([\#5042](https://github.com/NVIDIA/Megatron-LM/pull/5042), [\#5485](https://github.com/NVIDIA/Megatron-LM/pull/5485), [\#4949](https://github.com/NVIDIA/Megatron-LM/pull/4949)). * **GPTModel:** [deprecation announcement](https://github.com/NVIDIA/Megatron-LM/issues/6071). * **DeepSeek Sparse Attention (DSA):** adds backend-neutral CP and THD support, pipeline-aware indexer-loss scaling, IndexShare, and fused cuDNN and TileLang implementations ([\#5244](https://github.com/NVIDIA/Megatron-LM/pull/5244), [\#5246](https://github.com/NVIDIA/Megatron-LM/pull/5246), [\#5099](https://github.com/NVIDIA/Megatron-LM/pull/5099), [\#5049](https://github.com/NVIDIA/Megatron-LM/pull/5049)). * **Multi-Latent Attention in HybridModel:** adds MLA layer support, Q/K norm spec resolution, and fused MLA projection handling to HybridModel ([\#4452](https://github.com/NVIDIA/Megatron-LM/pull/4452)). #### Parallelism * **FSDP:** MFSDP v2 is under development. See [the design doc](https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/distributed/fsdp/src/docs/mfsdp_design.md) for links to the project tracker and implementation code. * **NCCL EP support:** adds NCCL EP token dispatcher backend ([\#5129](https://github.com/NVIDIA/Megatron-LM/pull/5129), [\#5735](https://github.com/NVIDIA/Megatron-LM/pull/5735)). * **Fuse per-sequence AlltoAll:** creates a unified AlltoAll ([\#4913](https://github.com/NVIDIA/Megatron-LM/pull/4913)). #### Performance and Memory * **CUDA Graph-compatible activation offloading:** enables fine-grained activation offloading during capture and replay, including deferred transfers and fused grouped-MLP offloading ([\#3697](https://github.com/NVIDIA/Megatron-LM/pull/3697), [\#5082](https://github.com/NVIDIA/Megatron-LM/pull/5082)). * **Streaming quantized checkpoint loading:** dequantizes FP8, MXFP8, blockwise-FP8, and NVFP4 tensors incrementally instead of allocating BF16 scratch space for the entire model ([\#4451](https://github.com/NVIDIA/Megatron-LM/pull/4451)). * **Partial CUDA Graph memory optimization:** moves graph surfaces into the global CUDA Graph memory pool and removes references and copies that retained excess buffers ([\#5451](https://github.com/NVIDIA/Megatron-LM/pull/5451)). #### Precision and Optimizers * **MXFP8 training paths:** take a look at our [Nemotron 3 Super MXFP8 example](https://github.com/NVIDIA/Megatron-LM/blob/core_r0.19.0/tests/functional_tests/test_cases/nemotron/nemotron3_super_release_gb200/model_config.yaml) * **Lion distributed-optimizer support:** routes Lion through `DistributedOptimizer` and adds single-moment checkpoint support, including Muon configurations that use Lion for scalar parameters ([\#5742](https://github.com/NVIDIA/Megatron-LM/pull/5742)). #### Inference * **Disaggregated-inference foundations:** adds heterogeneous inference-shard specifications, multi-destination refit, KV/Mamba-state resharding, local deterministic planning, and a NIXL copy backend ([\#5186](https://github.com/NVIDIA/Megatron-LM/pull/5186), [\#5187](https://github.com/NVIDIA/Megatron-LM/pull/5187), [\#5188](https://github.com/NVIDIA/Megatron-LM/pull/5188), [\#5872](https://github.com/NVIDIA/Megatron-LM/pull/5872)). * **Full-model CUDA Graphs for MTP inference:** enables full-model graph execution for Multi-Token Prediction inference ([\#4950](https://github.com/NVIDIA/Megatron-LM/pull/4950)). * **Load-aware prefix-cache routing:** routes requsts across inference workers using prefix-cache load information ([\#5607](https://github.com/NVIDIA/Megatron-LM/pull/5607)). #### Datasets * **Inter-document attention masking:** `GPTDataset` now emits document-boundary metadata, resets positions per document, and uses THD attention to prevent cross-document attention within packed samples, including TP and CP support ([\#5298](https://github.com/NVIDIA/Megatron-LM/pull/5298)). #### Multimodal * **MIMO heterogeneous training:** expands the current MIMO bullet with end-to-end non-colocated execution, stock training-loop integration, encoder prefetch, DDP overlap, and Nemotron MoE VLM/RADIO examples ([\#5602](https://github.com/NVIDIA/Megatron-LM/pull/5602), [\#5833](https://github.com/NVIDIA/Megatron-LM/pull/5833), [\#5979](https://github.com/NVIDIA/Megatron-LM/pull/5979)). #### Known Issues * **TE** **fused cross-entropy may** **degrade** **stability:** setting `cross_entropy_fusion_impl` to `te` causes convergence issues on some models. Use `native` instead, or upgrade Transformer Engine to 2.18, which carries the fix ([TE PR \#3193](https://github.com/NVIDIA/TransformerEngine/pull/3193)). * **SPCX grouped all-to-all payload corruption at high expert parallelism**: in the affected NCCL 2.30.5/SPCX stack, stale or uninitialized `ncclIbRequest.send.onlyWriteImm` state can misclassify sends in grouped variable-split BF16 all-to-all operations, causing payload loss or corruption. MCore's MoE `alltoall` dispatcher can expose this issue at high EP configurations, but the fault originates in the SPCX transport rather than the model or router. As a workaround, disable the external NCCL network plugin and use NCCL's internal IB transport (`NCCL_NET_PLUGIN=none NCCL_NET=IB`), or use the HybridEP dispatcher ([Megatron-Bridge issue \#5462](https://github.com/NVIDIA-NeMo/Megatron-Bridge/issues/5462)). <details><summary>Changelog Details</summary> - Add separate mtp_grad_scale_func for MTP loss scaling by @yfw :: PR: #3459 - [training migration] Migrate GPT builder by @maanug-nv :: PR: #4741 - Make Mamba conv params direct mixer params by @wujingyue :: PR: #4899 - Update oncall reviewer assignment by @Phlip79 :: PR: #5093 - Pass explicit process groups to hybrid logging by @yashaswikarnati :: PR: #4781 - Clean up top-level repository files by @Phlip79 :: PR: #5097 - [main] fix(moe): Fix several bugs for DSA rope and spec. by @yuzhongw-nvidia :: PR: #3026 - Move MIMO unit tests into models/mimo by @yashaswikarnati :: PR: #5063 - test: update DeepSeek FSDP2 GB200 memory golden by @wujingyue :: PR: #5094 - ci: make CI resilient to pip/uv network timeouts by @ko3n1g :: PR: #5118 - ci: treat docker container-removal conflict as flaky by @ko3n1g :: PR: #5120 - Fix GDN DTensor splitting for FSDP checkpointing by @conver334 :: PR: #4843 - Remove DeepEP hardware limit check by @janEbert :: PR: #4846 - Update transformer-engine dependency to revision 4220403 by @balasaajay :: PR: #5112 - Fix MoE aux_loss / z_loss gradient scaling with TP > 1 by @deepakn94 :: PR: #5047 - Update Claude copy workflow to enforce user restrictions and improve error messages by @balasaajay :: PR: #5117 - Add advisory process group guidance to Claude reviews by @yashaswikarnati :: PR: #5111 - build: cap pydantic<2.14 in transformer-engine dependency metadata by @cuichenx :: PR: #5125 - fix(test): skip scalar-less tensorboard event files in resume checks by @ko3n1g :: PR: #5121 - docs: fix contributor guide typo by @LeSingh1 :: PR: #4858 - ci(unit-tests): split slow unit-test buckets over 15min SLA by @ko3n1g :: PR: #5133 - Update MINOR version to 19 by @balasaajay :: PR: #5096 - Fix Muon QKV split for gated attention by @Moozy23232 :: PR: #4728 - Fix DSA indexer loss not averaged across micro-batches by @kaimo455 :: PR: #4070 - Roll input IDs for MTP labels by @yfw :: PR: #3457 - Adding blackwell tests by @shanmugamr1992 :: PR: #5113 - Refactor: Move paged stashing Triton kernels by @Phlip79 :: PR: #5003 - Relax atol for test_router_gating_linear router_dtype=torch.float32 by @adityasingh2400 :: PR: #4915 - Fix incorrect inference metadata tensor dtypes by @santhnm2 :: PR: #4855 - Disable TE cross entropy loss fusion by @mchrzanowski :: PR: #5115 - fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather by @ko3n1g :: PR: #4982 - Pass TP group to unfused cross entropy by @yashaswikarnati :: PR: #5128 - test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev by @ko3n1g :: PR: #5156 - ci(notify): mention mcore-oncall and philipp on critical CI events by @ko3n1g :: PR: #5152 - Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill by @mathemakitten :: PR: #3509 - ci: Disable a few gb200 test cases to support 2 branches. by @balasaajay :: PR: #5151 - build: Switch DSv3 on H100 to HybridEP by @balasaajay :: PR: #5164 - Add MTP acceptance rate metrics by @yfw :: PR: #3458 - Nemotron Ultra config for ModelOpt examples by @jenchen13 :: PR: #5159 - Make MTP / prefix cache stats persist for engine lifetime by @santhnm2 :: PR: #4101 - Restore Greptile configuration by @Phlip79 :: PR: #5166 - chore: bump `_code_freeze` workflow to `v1.4.2` by @ko3n1g :: PR: #5132 - ci: Remove docs build test in favor of release test by @chtruong814 :: PR: #5182 - Move TE cross entropy guard to training args by @yaoyu-33 :: PR: #5162 - Fix error in deepseek parser by @tdene :: PR: #5136 - Clamp `max_new_tokens` in MInf to mirror vllm by @tdene :: PR: #5181 - Fix logprob slicing for 0 generated token case by @santhnm2 :: PR: #5167 - [Perf] Fold frozen linear dgrad matmul by @cuichenx :: PR: #5092 - build: add managed = true to [tool.uv] by @kajalj22 :: PR: #5190 - Stabilize GB200 inference perf tests against cold-start noise by @shanmugamr1992 :: PR: #5171 - nvidia style guide audit for getting started folder by @megnvidia :: PR: #5168 - AI aided audit for Nvidia Style guidance by @megnvidia :: PR: #5141 - Avoid stat syscall in rerun result validation by @dimapihtar :: PR: #5107 - fix(combined-1f1b): release loss-node input storage after combined backward by @Wohox :: PR: #4909 - Enable selective recompute for `norm_out` in GDN layers by @xuantengh :: PR: #4715 - fix(elastification): align with get_batch + utils refactors by @balasaajay :: PR: #5194 - docs: Update Latest News in README.md by @sbhavani :: PR: #3790 - Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. by @cspades :: PR: #4802 - ci: add smoke tests by @balasaajay :: PR: #5143 - Add mtp_detach_heads config to detach MTP head inputs by @yfw :: PR: #3456 - docs: fix install guide NGC container anchor by @Connor-XY :: PR: #5224 - Fuse per-sequence AlltoAll into a unified one in GDN forward by @xuantengh :: PR: #4913 - Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path by @yashaswikarnati :: PR: #5150 - Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool by @wujingyue :: PR: #4810 - Add named layouts to HyperCommGrid for heterogeneous parallelism by @yashaswikarnati :: PR: #5148 - Fix wgrad race condition when using double buffers. by @cspades :: PR: #5222 - Move uneven DTensor distributed fixture to conftest by @wujingyue :: PR: #5237 - Route bridge communicator cross-grid P2P through a dedicated process group by @yashaswikarnati :: PR: #5234 - Fix test_split_tensor_along_last_dim to actually assert correctness by @lichenlu :: PR: #4710 - Add optional group= to common_utils model/data-parallel reduction helpers by @yashaswikarnati :: PR: #5251 - ci: Allow DCO check in merge queue and add DCO requirement to Contribution guide by @chtruong814 :: PR: #5278 - Add MIMO hetero topology + distributed bootstrap (examples/mimo training-loop folder) by @yashaswikarnati :: PR: #5260 - Remove checkpoint-time GPU cache reclaim workaround by @shurkat-nvidia :: PR: #5170 - Remove duplicate nccl_allocator import by @returnL :: PR: #5057 - fix(ci): resolve t5 dataloader stall + GRPO cudagraph-memory regression (CI-validated) by @Connor-XY :: PR: #5280 - Fix Dockerfile warnings by @janEbert :: PR: #4856 - Fix fused MLA delayed weight grad hooks by @sraman-rgb :: PR: #5273 - ci: limit retries on unsuccessful test launches by @balasaajay :: PR: #5275 - Thread pg_collection into get_model DDP bucket sizing by @yashaswikarnati :: PR: #5250 - Enable non-deterministic results in model configuration for nemotron tests by @balasaajay :: PR: #5239 - Stabilize hybrid nanov3 gb200 perf by @shanmugamr1992 :: PR: #5295 - Clip mtp grads separately when mtp_detach_heads=True by @yfw :: PR: #4116 - Thread pg_collection into train_step reductions by @yashaswikarnati :: PR: #5259 - Offline Logits-Based Knowledge Distillation by @AAnoosheh :: PR: #5019 - Handle None values in sampling parameters by @tdene :: PR: #5300 - Allow for pre-bound socket to be passed in server by @tdene :: PR: #5301 - Add moe loss normalization for RL SFT by @pthombre :: PR: #3956 - Add code owners for optimizer-related files by @janEbert :: PR: #5297 - Fix EP=1 inference by allocating buffers anyway by @mathemakitten :: PR: #5233 - Fix crash due to tool call at sequence length by @tdene :: PR: #5302 - Inference: Cudagraph-aware admission gating in prefill scheduler by @mathemakitten :: PR: #4870 - Account for reasoning token stripping by @tdene :: PR: #5313 - Thread pg_collection through wrap_model_chunks_with_ddp by @yashaswikarnati :: PR: #5328 - Fix LatentMoE theoretical memory estimate by @Wong4j :: PR: #5145 - Add zstandard package to Docker LTS requirements. Fix nightly failures by @balasaajay :: PR: #5347 - Thread MIMO support through the stock training loop (schedule + optimizer) by @yashaswikarnati :: PR: #5333 - ci: default functional test time limit to 4h for release/weekly scopes by @ko3n1g :: PR: #5360 - Fix memory leak with log_max_attention_logit (#4699) by @asolergi-nv :: PR: #5067 - Clean up pretrain_gpt.py and pretrain_hybrid.py formatting and remove module globals by @ilml :: PR: #5351 - Add full model cuda graph support for MTP inference by @santhnm2 :: PR: #4950 - Expand the Mamba prefix caching memory safety check to include scratch space buffers by @santhnm2 :: PR: #5348 - Make Megatron RL only materialize last token logit by @tdene :: PR: #4551 - Profiling by @jalbericiola :: PR: #3110 - Support fused MLA QKV checkpoint reload by @sraman-rgb :: PR: #5310 - Add minimal DBuffer implementation by @wujingyue :: PR: #4835 - [split 1/4] Fix packed THD RoPE under CP by @HollowMan6 :: PR: #5243 - Document agent PR commit sign-off and signing by @wujingyue :: PR: #5381 - Remove unused distributed pytest markers by @wujingyue :: PR: #5380 - [feat] Support fine-grained activation offloading in fused group mlp by @lhb8125 :: PR: #5082 - Thread tensor-parallel group into the RADIO patch embedder by @yashaswikarnati :: PR: #5371 - Add MimoModel.zero_grad_buffer delegating to active DDP submodules by @yashaswikarnati :: PR: #5372 - [split 3/4] Refactor absorbed MLA projection handling by @HollowMan6 :: PR: #5245 - ci: Remove sync skills workflow by @chtruong814 :: PR: #5091 - Add flaky marker to fine-grained activation offloading test (#5350) by @balasaajay :: PR: #5368 - Revert "Remove checkpoint-time GPU cache reclaim workaround (#5170)" by @balasaajay :: PR: #5366 - Update goldens for weekly tests after pytorch and TE bumps. by @balasaajay :: PR: #5399 - Add MIMO runtime setup: per-role RNG seeding and DDP wrapping by @yashaswikarnati :: PR: #5285 - Add --mamba-training-ssm-states-dtype argument by @tdene :: PR: #5309 - Fix Mamba prefix match for chunked prefill by @lmcafee-nvidia :: PR: #4758 - Disag MR2: Refit into multiple destination pools and tied-embedding + UVM fixes by @wdykas :: PR: #5187 - Disag MR1: Add inference shard specs and pg-collection building by @wdykas :: PR: #5186 - Support the MIMO cross-grid path in training loop by @yashaswikarnati :: PR: #5373 - Stabilize hybrid_2b GB200 perf test against run-to-run noise by @shanmugamr1992 :: PR: #5364 - Consistent oncall schedule by @Phlip79 :: PR: #5404 - Disag MR3: Add heterogeneous KV/Mamba reshard planners by @wdykas :: PR: #5188 - Add RADIO vision encoder wrapper for MIMO example by @yashaswikarnati :: PR: #5397 - Clean up MTP inference control flow by @santhnm2 :: PR: #5418 - Add MIMO dual gradient finalization (colocated + non-colocated) by @yashaswikarnati :: PR: #5286 - Add RL rollout submission and consumption granularity controls by @lauradang :: PR: #5306 - Add --functional-test-name to trigger_internal_ci by @ko3n1g :: PR: #5449 - Rename CP batch helpers to describe balancing granularity by @deepakn94 :: PR: #5403 - build: point flash_mla at the nv_dev branch by @ko3n1g :: PR: #5448 - Add logprobs_mode (raw/processed) to inference config by @tdene :: PR: #5419 - ci: Set test_save_verify_integrity_manifest_directly as flaky by @chtruong814 :: PR: #5468 - Remove DBuffer mesh axis validation by @wujingyue :: PR: #5441 - Add hetero grid args and MoE process groups for MIMO example by @yashaswikarnati :: PR: #5375 - Support SWA and sink attention in dynamic inference by @cuichenx :: PR: #5249 - feat(inference): default use_coordinator to True in high-level APIs by @shanmugamr1992 :: PR: #5326 - Support HybridModel feature specs in ModelOpt by @Phlip79 :: PR: #5354 - Add experimental Megatron-FSDP fully_shard implementation by @wujingyue :: PR: #5387 - Add inference functions to support MCore-/MBridge- training refactor and remove legacy modelbuilder functions by @shanmugamr1992 :: PR: #5169 - ci: launch GB200 unit tests via launch_on_gb200 marker by @ko3n1g :: PR: #5477 - build: install flash_mla from source in the CI image by @ko3n1g :: PR: #5481 - [split 2/4] Scale DSA indexer loss in pipeline schedules by @HollowMan6 :: PR: #5244 - ci: check megatron.training imports in installation test by @ko3n1g :: PR: #5458 - Fix merges_file kwarg name in HuggingFaceTokenizer by @muyihao :: PR: #5406 - Clean up training.py module header (dedupe + reorganize imports/globals) by @ilml :: PR: #5469 - Automated community request assignment by @Phlip79 :: PR: #5147 - Thread process groups through training checkpoint paths by @yashaswikarnati :: PR: #5486 - Narrow oncall responsibilities by @Phlip79 :: PR: #5490 - Add MIMO forward step and per-token loss for hetero training by @yashaswikarnati :: PR: #5376 - Add Nemotron6-MoE VLM model provider for MIMO example by @yashaswikarnati :: PR: #5374 - ci: auto-retry test-data download in container-build job by @ko3n1g :: PR: #5498 - Force RL inference to CP=1 by @tdene :: PR: #5423 - Merge cu_seqlens across micro-batch for THD attention by @deepakn94 :: PR: #5454 - [split 4/4] Enable DSA CP and THD hooks by @HollowMan6 :: PR: #5246 - Fix fused MLA down projection with tensor parallelism by @sraman-rgb :: PR: #5383 - Fix NameError in is_flashinfer_min_version when check_equality=False by @adityasingh2
retained text (truncated)
#### MoE * **Quantile Balancing router:** adds auxiliary-loss-free load balancing using per-expert routing biases derived from quantile estimates ([\#5349](https://github.com/NVIDIA/Megatron-LM/pull/5349)). * **Fused shared-expert MLP:** enables shared experts through grouped GEMM, providing the path for fused GEMM, SwiGLU, and quantized execution ([\#5604](https://github.com/NVIDIA/Megatron-LM/pull/5604)). * **Packed-sequence MoE dispatch:** adds THD sequence-packing support to the DeepEP and HybridEP dispatcher backends ([\#5008](https://github.com/NVIDIA/Megatron-LM/pull/5008)). * **MoE routing analysis:** adds training/inference routing traces plus expert-concentration and one-layer-ahead predictability analysis tools ([\#5220](https://github.com/NVIDIA/Megatron-LM/pull/5220)). #### Model Architecture and Support * **HybridModel:** next-gen model class. * `GPTModel` to `HybridModel` guide [published](https://docs.nvidia.com/megatron-core/developer-guide/nightly/user-guide/hybrid-model-migration.html). * Added ability to directly load `GPTModel` checkpoints ([\#5792](https://github.com/NVIDIA/Megatron-LM/pull/5792)). * Part 1 of the [design doc](https://github.com/NVIDIA/Megatron-LM/issues/4620) is complete, part 2 will be completed for 0.20.0. * Deepseek v4 migrated to HybridModel on the `dev` branch ([\#5042](https://github.com/NVIDIA/Megatron-LM/pull/5042), [\#5485](https://github.com/NVIDIA/Megatron-LM/pull/5485), [\#4949](https://github.com/NVIDIA/Megatron-LM/pull/4949)). * **GPTModel:** [deprecation announcement](https://github.com/NVIDIA/Megatron-LM/issues/6071). * **DeepSeek Sparse Attention (DSA):** adds backend-neutral CP and THD support, pipeline-aware indexer-loss scaling, IndexShare, and fused cuDNN and TileLang implementations ([\#5244](https://github.com/NVIDIA/Megatron-LM/pull/5244), [\#5246](https://github.com/NVIDIA/Megatron-LM/pull/5246), [\#5099](https://github.com/NVIDIA/Megatron-LM/pull/5099), [\#5049](https://github.com/NVIDIA/Megatron-LM/pull/5049)). * **Multi-Latent Attention in HybridModel:** adds MLA layer support, Q/K norm spec resolution, and fused MLA projection handling to HybridModel ([\#4452](https://github.com/NVIDIA/Megatron-LM/pull/4452)). #### Parallelism * **FSDP:** MFSDP v2 is under development. See [the design doc](https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/distributed/fsdp/src/docs/mfsdp_design.md) for links to the project tracker and implementation code. * **NCCL EP support:** adds NCCL EP token dispatcher backend ([\#5129](https://github.com/NVIDIA/Megatron-LM/pull/5129), [\#5735](https://github.com/NVIDIA/Megatron-LM/pull/5735)). * **Fuse per-sequence AlltoAll:** creates a unified AlltoAll ([\#4913](https://github.com/NVIDIA/Megatron-LM/pull/4913)). #### Performance and Memory * **CUDA Graph-compatible activation offloading:** enables fine-grained activation offloading during capture and replay, including deferred transfers and fused grouped-MLP offloading ([\#3697](https://github.com/NVIDIA/Megatron-LM/pull/3697), [\#5082](https://github.com/NVIDIA/Megatron-LM/pull/5082)). * **Streaming quantized checkpoint loading:** dequantizes FP8, MXFP8, blockwise-FP8, and NVFP4 tensors incrementally instead of allocating BF16 scratch space for the entire model ([\#4451](https://github.com/NVIDIA/Megatron-LM/pull/4451)). * **Partial CUDA Graph memory optimization:** moves graph surfaces into the global CUDA Graph memory pool and removes references and copies that retained excess buffers ([\#5451](https://github.com/NVIDIA/Megatron-LM/pull/5451)). #### Precision and Optimizers * **MXFP8 training paths:** take a look at our [Nemotron 3 Super MXFP8 example](https://github.com/NVIDIA/Megatron-LM/blob/core_r0.19.0/tests/functional_tests/test_cases/nemotron/nemotron3_super_release_gb200/model_config.yaml) * **Lion distributed-optimizer support:** routes Lion through `DistributedOptimizer` and adds single-moment checkpoint support, including Muon configurations that use Lion for scalar parameters ([\#5742](https://github.com/NVIDIA/Megatron-LM/pull/5742)). #### Inference * **Disaggregated-inference foundations:** adds heterogeneous inference-shard specifications, multi-destination refit, KV/Mamba-state resharding, local deterministic planning, and a NIXL copy backend ([\#5186](https://github.com/NVIDIA/Megatron-LM/pull/5186), [\#5187](https://github.com/NVIDIA/Megatron-LM/pull/5187), [\#5188](https://github.com/NVIDIA/Megatron-LM/pull/5188), [\#5872](https://github.com/NVIDIA/Megatron-LM/pull/5872)). * **Full-model CUDA Graphs for MTP inference:** enables full-model graph execution for Multi-Token Prediction inference ([\#4950](https://github.com/NVIDIA/Megatron-LM/pull/4950)). * **Load-aware prefix-cache routing:** routes requsts across inference workers using prefix-cache load information ([\#5607](https://github.com/NVIDIA/Megatron-LM/pull/5607)). #### Datasets * **Inter-document attention masking:** `GPTDataset` now emits document-boundary metadata, resets positions per document, and uses THD attention to prevent cross-document attention within packed samples, including TP and CP support ([\#5298](https://github.com/NVIDIA/Megatron-LM/pull/5298)). #### Multimodal * **MIMO heterogeneous training:** expands the current MIMO bullet with end-to-end non-colocated execution, stock training-loop integration, encoder prefetch, DDP overlap, and Nemotron MoE VLM/RADIO examples ([\#5602](https://github.com/NVIDIA/Megatron-LM/pull/5602), [\#5833](https://github.com/NVIDIA/Megatron-LM/pull/5833), [\#5979](https://github.com/NVIDIA/Megatron-LM/pull/5979)). #### Known Issues * **TE** **fused cross-entropy may** **degrade** **stability:** setting `cross_entropy_fusion_impl` to `te` causes convergence issues on some models. Use `native` instead, or upgrade Transformer Engine to 2.18, which carries the fix ([TE PR \#3193](https://github.com/NVIDIA/TransformerEngine/pull/3193)). * **SPCX grouped all-to-all payload corruption at high expert parallelism**: in the affected NCCL 2.30.5/SPCX stack, stale or uninitialized `ncclIbRequest.send.onlyWriteImm` state can misclassify sends in grouped variable-split BF16 all-to-all operations, causing payload loss or corruption. MCore's MoE `alltoall` dispatcher can expose this issue at high EP configurations, but the fault originates in the SPCX transport rather than the model or router. As a workaround, disable the external NCCL network plugin and use NCCL's internal IB transport (`NCCL_NET_PLUGIN=none NCCL_NET=IB`), or use the HybridEP dispatcher ([Megatron-Bridge issue \#5462](https://github.com/NVIDIA-NeMo/Megatron-Bridge/issues/5462)). <details><summary>Changelog Details</summary> - Add separate mtp_grad_scale_func for MTP loss scaling by @yfw :: PR: #3459 - [training migration] Migrate GPT builder by @maanug-nv :: PR: #4741 - Make Mamba conv params direct mixer params by @wujingyue :: PR: #4899 - Update oncall reviewer assignment by @Phlip79 :: PR: #5093 - Pass explicit process groups to hybrid logging by @yashaswikarnati :: PR: #4781 - Clean up top-level repository files by @Phlip79 :: PR: #5097 - [main] fix(moe): Fix several bugs for DSA rope and spec. by @yuzhongw-nvidia :: PR: #3026 - Move MIMO unit tests into models/mimo by @yashaswikarnati :: PR: #5063 - test: update DeepSeek FSDP2 GB200 memory golden by @wujingyue :: PR: #5094 - ci: make CI resilient to pip/uv network timeouts by @ko3n1g :: PR: #5118 - ci: treat docker container-removal conflict as flaky by @ko3n1g :: PR: #5120 - Fix GDN DTensor splitting for FSDP checkpointing by @conver334 :: PR: #4843 - Remove DeepEP hardware limit check by @janEbert :: PR: #4846 - Update transformer-engine dependency to revision 4220403 by @balasaajay :: PR: #5112 - Fix MoE aux_loss / z_loss gradient scaling with TP > 1 by @deepakn94 :: PR: #5047 - Update Claude copy workflow to enforce user restrictions and improve error messages by @balasaajay :: PR: #5117 - Add advisory process group guidance to Claude reviews by @yashaswikarnati :: PR: #5111 - build: cap pydantic<2.14 in transformer-engine dependency metadata by @cuichenx :: PR: #5125 - fix(test): skip scalar-less tensorboard event files in resume checks by @ko3n1g :: PR: #5121 - docs: fix contributor guide typo by @LeSingh1 :: PR: #4858 - ci(unit-tests): split slow unit-test buckets over 15min SLA by @ko3n1g :: PR: #5133 - Update MINOR version to 19 by @balasaajay :: PR: #5096 - Fix Muon QKV split for gated attention by @Moozy23232 :: PR: #4728 - Fix DSA indexer loss not averaged across micro-batches by @kaimo455 :: PR: #4070 - Roll input IDs for MTP labels by @yfw :: PR: #3457 - Adding blackwell tests by @shanmugamr1992 :: PR: #5113 - Refactor: Move paged stashing Triton kernels by @Phlip79 :: PR: #5003 - Relax atol for test_router_gating_linear router_dtype=torch.float32 by @adityasingh2400 :: PR: #4915 - Fix incorrect inference metadata tensor dtypes by @santhnm2 :: PR: #4855 - Disable TE cross entropy loss fusion by @mchrzanowski :: PR: #5115 - fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather by @ko3n1g :: PR: #4982 - Pass TP group to unfused cross entropy by @yashaswikarnati :: PR: #5128 - test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev by @ko3n1g :: PR: #5156 - ci(notify): mention mcore-oncall and philipp on critical CI events by @ko3n1g :: PR: #5152 - Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill by @mathemakitten :: PR: #3509 - ci: Disable a few gb200 test cases to support 2 branches. by @balasaajay :: PR: #5151 - build: Switch DSv3 on H100 to HybridEP by @balasaajay :: PR: #5164 - Add MTP acceptance rate metrics by @yfw :: PR: #3458 - Nemotron Ultra config for ModelOpt examples by @jenchen13 :: PR: #5159 - Make MTP / prefix cache stats persist for engine lifetime by @santhnm2 :: PR: #4101 - Restore Greptile configuration by @Phlip79 :: PR: #5166 - chore: bump `_code_freeze` workflow to `v1.4.2` by @ko3n1g :: PR: #5132 - ci: Remove docs build test in favor of release test by @chtruong814 :: PR: #5182 - Move TE cross entropy guard to training args by @yaoyu-33 :: PR: #5162 - Fix error in deepseek parser by @tdene :: PR: #5136 - Clamp `max_new_tokens` in MInf to mirror vllm by @tdene :: PR: #5181 - Fix logprob slicing for 0 generated token case by @santhnm2 :: PR: #5167 - [Perf] Fold frozen linear dgrad matmul by @cuichenx :: PR: #5092 - build: add managed = true to [tool.uv] by @kajalj22 :: PR: #5190 - Stabilize GB200 inference perf tests against cold-start noise by @shanmugamr1992 :: PR: #5171 - nvidia style guide audit for getting started folder by @megnvidia :: PR: #5168 - AI aided audit for Nvidia Style guidance by @megnvidia :: PR: #5141 - Avoid stat syscall in rerun result validation by @dimapihtar :: PR: #5107 - fix(combined-1f1b): release loss-node input storage after combined backward by @Wohox :: PR: #4909 - Enable selective recompute for `norm_out` in GDN layers by @xuantengh :: PR: #4715 - fix(elastification): align with get_batch + utils refactors by @balasaajay :: PR: #5194 - docs: Update Latest News in README.md by @sbhavani :: PR: #3790 - Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. by @cspades :: PR: #4802 - ci: add smoke tests by @balasaajay :: PR: #5143 - Add mtp_detach_heads config to detach MTP head inputs by @yfw :: PR: #3456 - docs: fix install guide NGC container anchor by @Connor-XY :: PR: #5224 - Fuse per-sequence AlltoAll into a unified one in GDN forward by @xuantengh :: PR: #4913 - Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path by @yashaswikarnati :: PR: #5150 - Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool by @wujingyue :: PR: #4810 - Add named layout
document fetched 2026-08-26 05:55