# NVIDIA/Megatron-LM core_v0.19.0: NVIDIA Megatron Core 0.19.0 — retained record

> Retained github record from github:NVIDIA/Megatron-LM.

Canonical: https://highsignal.app/data/records/892f98f11235ac8a

[← sources](https://highsignal.app/data)


source: [github](https://highsignal.app/data/github) (github:NVIDIA/Megatron-LM)  published 2026-08-19 19:02 ingested 2026-08-21 06:29

[NVIDIA Corporation](https://highsignal.app/entities/NVDA)[original source ↗](https://github.com/NVIDIA/Megatron-LM/releases/tag/core_v0.19.0)

## extract

#### MoE

* **Quantile Balancing router:** adds auxiliary-loss-free load balancing using per-expert routing biases derived from quantile estimates ([\#5349](https://github.com/NVIDIA/Megatron-LM/pull/5349)).  
* **Fused shared-expert MLP:** enables shared experts through grouped GEMM, providing the path for fused GEMM, SwiGLU, and quantized execution ([\#5604](https://github.com/NVIDIA/Megatron-LM/pull/5604)).  
* **Packed-sequence MoE dispatch:** adds THD sequence-packing support to the DeepEP and HybridEP dispatcher backends ([\#5008](https://github.com/NVIDIA/Megatron-LM/pull/5008)).  
* **MoE routing analysis:** adds training/inference routing traces plus expert-concentration and one-layer-ahead predictability analysis tools ([\#5220](https://github.com/NVIDIA/Megatron-LM/pull/5220)).

#### Model Architecture and Support

* **HybridModel:** next-gen model class.  
  * `GPTModel` to `HybridModel` guide [published](https://docs.nvidia.com/megatron-core/developer-guide/nightly/user-guide/hybrid-model-migration.html).  
  * Added ability to directly load `GPTModel` checkpoints ([\#5792](https://github.com/NVIDIA/Megatron-LM/pull/5792)).  
  * Part 1 of the [design doc](https://github.com/NVIDIA/Megatron-LM/issues/4620) is complete, part 2 will be completed for 0.20.0.  
  * Deepseek v4 migrated to HybridModel on the `dev` branch ([\#5042](https://github.com/NVIDIA/Megatron-LM/pull/5042), [\#5485](https://github.com/NVIDIA/Megatron-LM/pull/5485), [\#4949](https://github.com/NVIDIA/Megatron-LM/pull/4949)).  
* **GPTModel:** [deprecation announcement](https://github.com/NVIDIA/Megatron-LM/issues/6071).  
* **DeepSeek Sparse Attention (DSA):** adds backend-neutral CP and THD support, pipeline-aware indexer-loss scaling, IndexShare, and fused cuDNN and TileLang implementations ([\#5244](https://github.com/NVIDIA/Megatron-LM/pull/5244), [\#5246](https://github.com/NVIDIA/Megatron-LM/pull/5246), [\#5099](https://github.com/NVIDIA/Megatron-LM/pull/5099), [\#5049](https://github.com/NVIDIA/Megatron-LM/pull/5049)).  
* **Multi-Latent Attention in HybridModel:** adds MLA layer support, Q/K norm spec resolution, and fused MLA projection handling to HybridModel ([\#4452](https://github.com/NVIDIA/Megatron-LM/pull/4452)).

#### Parallelism

* **FSDP:** MFSDP v2 is under development. See [the design doc](https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/distributed/fsdp/src/docs/mfsdp_design.md) for links to the project tracker and implementation code.  
* **NCCL EP support:** adds NCCL EP token dispatcher backend ([\#5129](https://github.com/NVIDIA/Megatron-LM/pull/5129), [\#5735](https://github.com/NVIDIA/Megatron-LM/pull/5735)).  
* **Fuse per-sequence AlltoAll:** creates a unified AlltoAll ([\#4913](https://github.com/NVIDIA/Megatron-LM/pull/4913)).

#### Performance and Memory

* **CUDA Graph-compatible activation offloading:** enables fine-grained activation offloading during capture and replay, including deferred transfers and fused grouped-MLP offloading ([\#3697](https://github.com/NVIDIA/Megatron-LM/pull/3697), [\#5082](https://github.com/NVIDIA/Megatron-LM/pull/5082)).  
* **Streaming quantized checkpoint loading:** dequantizes FP8, MXFP8, blockwise-FP8, and NVFP4 tensors incrementally instead of allocating BF16 scratch space for the entire model ([\#4451](https://github.com/NVIDIA/Megatron-LM/pull/4451)).  
* **Partial CUDA Graph memory optimization:** moves graph surfaces into the global CUDA Graph memory pool and removes references and copies that retained excess buffers ([\#5451](https://github.com/NVIDIA/Megatron-LM/pull/5451)).

#### Precision and Optimizers

* **MXFP8 training paths:** take a look at our [Nemotron 3 Super MXFP8 example](https://github.com/NVIDIA/Megatron-LM/blob/core_r0.19.0/tests/functional_tests/test_cases/nemotron/nemotron3_super_release_gb200/model_config.yaml)  
* **Lion distributed-optimizer support:** routes Lion through `DistributedOptimizer` and adds single-moment checkpoint support, including Muon configurations that use Lion for scalar parameters ([\#5742](https://github.com/NVIDIA/Megatron-LM/pull/5742)).

#### Inference

* **Disaggregated-inference foundations:** adds heterogeneous inference-shard specifications, multi-destination refit, KV/Mamba-state resharding, local deterministic planning, and a NIXL copy backend ([\#5186](https://github.com/NVIDIA/Megatron-LM/pull/5186), [\#5187](https://github.com/NVIDIA/Megatron-LM/pull/5187), [\#5188](https://github.com/NVIDIA/Megatron-LM/pull/5188), [\#5872](https://github.com/NVIDIA/Megatron-LM/pull/5872)).  
* **Full-model CUDA Graphs for MTP inference:** enables full-model graph execution for Multi-Token Prediction inference ([\#4950](https://github.com/NVIDIA/Megatron-LM/pull/4950)).  
* **Load-aware prefix-cache routing:** routes requsts across inference workers using prefix-cache load information ([\#5607](https://github.com/NVIDIA/Megatron-LM/pull/5607)).

#### Datasets

* **Inter-document attention masking:** `GPTDataset` now emits document-boundary metadata, resets positions per document, and uses THD attention to prevent cross-document attention within packed samples, including TP and CP support ([\#5298](https://github.com/NVIDIA/Megatron-LM/pull/5298)).

#### Multimodal

* **MIMO heterogeneous training:** expands the current MIMO bullet with end-to-end non-colocated execution, stock training-loop integration, encoder prefetch, DDP overlap, and Nemotron MoE VLM/RADIO examples ([\#5602](https://github.com/NVIDIA/Megatron-LM/pull/5602), [\#5833](https://github.com/NVIDIA/Megatron-LM/pull/5833), [\#5979](https://github.com/NVIDIA/Megatron-LM/pull/5979)).

#### Known Issues

* **TE** **fused cross-entropy may** **degrade** **stability:** setting `cross_entropy_fusion_impl` to `te` causes convergence issues on some models. Use `native` instead, or upgrade Transformer Engine to 2.18, which carries the fix ([TE PR \#3193](https://github.com/NVIDIA/TransformerEngine/pull/3193)).  
* **SPCX grouped all-to-all payload corruption at high expert parallelism**: in the affected NCCL 2.30.5/SPCX stack, stale or uninitialized `ncclIbRequest.send.onlyWriteImm` state can misclassify sends in grouped variable-split BF16 all-to-all operations, causing payload loss or corruption. MCore's MoE `alltoall` dispatcher can expose this issue at high EP configurations, but the fault originates in the SPCX transport rather than the model or router. As a workaround, disable the external NCCL network plugin and use NCCL's internal IB transport (`NCCL_NET_PLUGIN=none NCCL_NET=IB`), or use the HybridEP dispatcher ([Megatron-Bridge issue \#5462](https://github.com/NVIDIA-NeMo/Megatron-Bridge/issues/5462)).

<details><summary>Changelog Details</summary>

- Add separate mtp_grad_scale_func for MTP loss scaling by @yfw :: PR: #3459
- [training migration] Migrate GPT builder by @maanug-nv :: PR: #4741
- Make Mamba conv params direct mixer params by @wujingyue :: PR: #4899
- Update oncall reviewer assignment by @Phlip79 :: PR: #5093
- Pass explicit process groups to hybrid logging by @yashaswikarnati :: PR: #4781
- Clean up top-level repository files by @Phlip79 :: PR: #5097
- [main] fix(moe): Fix several bugs for DSA rope and spec. by @yuzhongw-nvidia :: PR: #3026
- Move MIMO unit tests into models/mimo by @yashaswikarnati :: PR: #5063
- test: update DeepSeek FSDP2 GB200 memory golden by @wujingyue :: PR: #5094
- ci: make CI resilient to pip/uv network timeouts by @ko3n1g :: PR: #5118
- ci: treat docker container-removal conflict as flaky by @ko3n1g :: PR: #5120
- Fix GDN DTensor splitting for FSDP checkpointing by @conver334 :: PR: #4843
- Remove DeepEP hardware limit check by @janEbert :: PR: #4846
- Update transformer-engine dependency to revision 4220403 by @balasaajay :: PR: #5112
- Fix MoE aux_loss / z_loss gradient scaling with TP > 1 by @deepakn94 :: PR: #5047
- Update Claude copy workflow to enforce user restrictions and improve error messages by @balasaajay :: PR: #5117
- Add advisory process group guidance to Claude reviews by @yashaswikarnati :: PR: #5111
- build: cap pydantic<2.14 in transformer-engine dependency metadata by @cuichenx :: PR: #5125
- fix(test): skip scalar-less tensorboard event files in resume checks by @ko3n1g :: PR: #5121
- docs: fix contributor guide typo by @LeSingh1 :: PR: #4858
- ci(unit-tests): split slow unit-test buckets over 15min SLA by @ko3n1g :: PR: #5133
- Update MINOR version to 19 by @balasaajay :: PR: #5096
- Fix Muon QKV split for gated attention by @Moozy23232 :: PR: #4728
- Fix DSA indexer loss not averaged across micro-batches by @kaimo455 :: PR: #4070
- Roll input IDs for MTP labels by @yfw :: PR: #3457
- Adding blackwell tests by @shanmugamr1992 :: PR: #5113
- Refactor: Move paged stashing Triton kernels by @Phlip79 :: PR: #5003
- Relax atol for test_router_gating_linear router_dtype=torch.float32 by @adityasingh2400 :: PR: #4915
- Fix incorrect inference metadata tensor dtypes by @santhnm2 :: PR: #4855
- Disable TE cross entropy loss fusion by @mchrzanowski :: PR: #5115
- fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather by @ko3n1g :: PR: #4982
- Pass TP group to unfused cross entropy by @yashaswikarnati :: PR: #5128
- test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev by @ko3n1g :: PR: #5156
- ci(notify): mention mcore-oncall and philipp on critical CI events by @ko3n1g :: PR: #5152
- Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill by @mathemakitten :: PR: #3509
- ci: Disable a few gb200 test cases to support 2 branches. by @balasaajay :: PR: #5151
- build: Switch DSv3 on H100 to HybridEP by @balasaajay :: PR: #5164
- Add MTP acceptance rate metrics by @yfw :: PR: #3458
- Nemotron Ultra config for ModelOpt examples by @jenchen13 :: PR: #5159
- Make MTP / prefix cache stats persist for engine lifetime by @santhnm2 :: PR: #4101
- Restore Greptile configuration by @Phlip79 :: PR: #5166
- chore: bump `_code_freeze` workflow to `v1.4.2` by @ko3n1g :: PR: #5132
- ci: Remove docs build test in favor of release test by @chtruong814 :: PR: #5182
- Move TE cross entropy guard to training args by @yaoyu-33 :: PR: #5162
- Fix error in deepseek parser by @tdene :: PR: #5136
- Clamp `max_new_tokens` in MInf to mirror vllm by @tdene :: PR: #5181
- Fix logprob slicing for 0 generated token case by @santhnm2 :: PR: #5167
- [Perf] Fold frozen linear dgrad matmul by @cuichenx :: PR: #5092
- build: add managed = true to [tool.uv] by @kajalj22 :: PR: #5190
- Stabilize GB200 inference perf tests against cold-start noise by @shanmugamr1992 :: PR: #5171
- nvidia style guide audit for getting started folder by @megnvidia :: PR: #5168
- AI aided audit for Nvidia Style guidance by @megnvidia :: PR: #5141
- Avoid stat syscall in rerun result validation by @dimapihtar :: PR: #5107
- fix(combined-1f1b): release loss-node input storage after combined backward by @Wohox :: PR: #4909
- Enable selective recompute for `norm_out` in GDN layers  by @xuantengh :: PR: #4715
- fix(elastification): align with get_batch + utils refactors by @balasaajay :: PR: #5194
- docs: Update Latest News in README.md by @sbhavani :: PR: #3790
- Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. by @cspades :: PR: #4802
- ci: add smoke tests by @balasaajay :: PR: #5143
- Add mtp_detach_heads config to detach MTP head inputs by @yfw :: PR: #3456
- docs: fix install guide NGC container anchor by @Connor-XY :: PR: #5224
- Fuse per-sequence AlltoAll into a unified one in GDN forward by @xuantengh :: PR: #4913
- Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path by @yashaswikarnati :: PR: #5150
- Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool by @wujingyue :: PR: #4810
- Add named layouts to HyperCommGrid for heterogeneous parallelism by @yashaswikarnati :: PR: #5148
- Fix wgrad race condition when using double buffers. by @cspades :: PR: #5222
- Move uneven DTensor distributed fixture to conftest by @wujingyue :: PR: #5237
- Route bridge communicator cross-grid P2P through a dedicated process group by @yashaswikarnati :: PR: #5234
- Fix test_split_tensor_along_last_dim to actually assert correctness by @lichenlu :: PR: #4710
- Add optional group= to common_utils model/data-parallel reduction helpers by @yashaswikarnati :: PR: #5251
- ci: Allow DCO check in merge queue and add DCO requirement to Contribution guide by @chtruong814 :: PR: #5278
- Add MIMO hetero topology + distributed bootstrap (examples/mimo training-loop folder) by @yashaswikarnati :: PR: #5260
- Remove checkpoint-time GPU cache reclaim workaround by @shurkat-nvidia :: PR: #5170
- Remove duplicate nccl_allocator import by @returnL :: PR: #5057
- fix(ci): resolve t5 dataloader stall + GRPO cudagraph-memory regression (CI-validated) by @Connor-XY :: PR: #5280
- Fix Dockerfile warnings by @janEbert :: PR: #4856
- Fix fused MLA delayed weight grad hooks by @sraman-rgb :: PR: #5273
- ci: limit retries on unsuccessful test launches by @balasaajay :: PR: #5275
- Thread pg_collection into get_model DDP bucket sizing by @yashaswikarnati :: PR: #5250
- Enable non-deterministic results in model configuration for nemotron tests by @balasaajay :: PR: #5239
- Stabilize hybrid nanov3 gb200 perf by @shanmugamr1992 :: PR: #5295
- Clip mtp grads separately when mtp_detach_heads=True by @yfw :: PR: #4116
- Thread pg_collection into train_step reductions by @yashaswikarnati :: PR: #5259
- Offline Logits-Based Knowledge Distillation by @AAnoosheh :: PR: #5019
- Handle None values in sampling parameters by @tdene :: PR: #5300
- Allow for pre-bound socket to be passed in server by @tdene :: PR: #5301
- Add moe loss normalization for RL SFT by @pthombre :: PR: #3956
- Add code owners for optimizer-related files by @janEbert :: PR: #5297
- Fix EP=1 inference by allocating buffers anyway by @mathemakitten :: PR: #5233
- Fix crash due to tool call at sequence length by @tdene :: PR: #5302
- Inference: Cudagraph-aware admission gating in prefill scheduler by @mathemakitten :: PR: #4870
- Account for reasoning token stripping by @tdene :: PR: #5313
- Thread pg_collection through wrap_model_chunks_with_ddp by @yashaswikarnati :: PR: #5328
- Fix LatentMoE theoretical memory estimate by @Wong4j :: PR: #5145
- Add zstandard package to Docker LTS requirements. Fix nightly failures by @balasaajay :: PR: #5347
- Thread MIMO support through the stock training loop (schedule + optimizer) by @yashaswikarnati :: PR: #5333
- ci: default functional test time limit to 4h for release/weekly scopes by @ko3n1g :: PR: #5360
- Fix memory leak with log_max_attention_logit (#4699) by @asolergi-nv :: PR: #5067
- Clean up pretrain_gpt.py and pretrain_hybrid.py formatting and remove module globals by @ilml :: PR: #5351
- Add full model cuda graph support for MTP inference by @santhnm2 :: PR: #4950
- Expand the Mamba prefix caching memory safety check to include scratch space buffers by @santhnm2 :: PR: #5348
- Make Megatron RL only materialize last token logit by @tdene :: PR: #4551
- Profiling  by @jalbericiola :: PR: #3110
- Support fused MLA QKV checkpoint reload by @sraman-rgb :: PR: #5310
- Add minimal DBuffer implementation by @wujingyue :: PR: #4835
- [split 1/4] Fix packed THD RoPE under CP by @HollowMan6 :: PR: #5243
- Document agent PR commit sign-off and signing by @wujingyue :: PR: #5381
- Remove unused distributed pytest markers by @wujingyue :: PR: #5380
- [feat] Support fine-grained activation offloading in fused group mlp by @lhb8125 :: PR: #5082
- Thread tensor-parallel group into the RADIO patch embedder by @yashaswikarnati :: PR: #5371
- Add MimoModel.zero_grad_buffer delegating to active DDP submodules by @yashaswikarnati :: PR: #5372
- [split 3/4] Refactor absorbed MLA projection handling by @HollowMan6 :: PR: #5245
- ci: Remove sync skills workflow by @chtruong814 :: PR: #5091
- Add flaky marker to fine-grained activation offloading test (#5350) by @balasaajay :: PR: #5368
- Revert "Remove checkpoint-time GPU cache reclaim workaround (#5170)" by @balasaajay :: PR: #5366
- Update goldens for weekly tests after pytorch and TE bumps. by @balasaajay :: PR: #5399
- Add MIMO runtime setup: per-role RNG seeding and DDP wrapping by @yashaswikarnati :: PR: #5285
- Add --mamba-training-ssm-states-dtype argument by @tdene :: PR: #5309
- Fix Mamba prefix match for chunked prefill by @lmcafee-nvidia :: PR: #4758
- Disag MR2: Refit into multiple destination pools and tied-embedding + UVM fixes by @wdykas :: PR: #5187
- Disag MR1: Add inference shard specs and pg-collection building by @wdykas :: PR: #5186
- Support the MIMO cross-grid path in training loop by @yashaswikarnati :: PR: #5373
- Stabilize hybrid_2b GB200 perf test against run-to-run noise by @shanmugamr1992 :: PR: #5364
- Consistent oncall schedule by @Phlip79 :: PR: #5404
- Disag MR3: Add heterogeneous KV/Mamba reshard planners by @wdykas :: PR: #5188
- Add RADIO vision encoder wrapper for MIMO example by @yashaswikarnati :: PR: #5397
- Clean up MTP inference control flow by @santhnm2 :: PR: #5418
- Add MIMO dual gradient finalization (colocated + non-colocated) by @yashaswikarnati :: PR: #5286
- Add RL rollout submission and consumption granularity controls by @lauradang :: PR: #5306
- Add --functional-test-name to trigger_internal_ci by @ko3n1g :: PR: #5449
- Rename CP batch helpers to describe balancing granularity by @deepakn94 :: PR: #5403
- build: point flash_mla at the nv_dev branch by @ko3n1g :: PR: #5448
- Add logprobs_mode (raw/processed) to inference config by @tdene :: PR: #5419
- ci: Set test_save_verify_integrity_manifest_directly as flaky by @chtruong814 :: PR: #5468
- Remove DBuffer mesh axis validation by @wujingyue :: PR: #5441
- Add hetero grid args and MoE process groups for MIMO example by @yashaswikarnati :: PR: #5375
- Support SWA and sink attention in dynamic inference by @cuichenx :: PR: #5249
- feat(inference): default use_coordinator to True in high-level APIs by @shanmugamr1992 :: PR: #5326
- Support HybridModel feature specs in ModelOpt by @Phlip79 :: PR: #5354
- Add experimental Megatron-FSDP fully_shard implementation by @wujingyue :: PR: #5387
- Add inference functions to support MCore-/MBridge- training refactor and remove legacy modelbuilder functions by @shanmugamr1992 :: PR: #5169
- ci: launch GB200 unit tests via launch_on_gb200 marker by @ko3n1g :: PR: #5477
- build: install flash_mla from source in the CI image by @ko3n1g :: PR: #5481
- [split 2/4] Scale DSA indexer loss in pipeline schedules by @HollowMan6 :: PR: #5244
- ci: check megatron.training imports in installation test by @ko3n1g :: PR: #5458
- Fix merges_file kwarg name in HuggingFaceTokenizer by @muyihao :: PR: #5406
- Clean up training.py module header (dedupe + reorganize imports/globals) by @ilml :: PR: #5469
- Automated community request assignment by @Phlip79 :: PR: #5147
- Thread process groups through training checkpoint paths by @yashaswikarnati :: PR: #5486
- Narrow oncall responsibilities by @Phlip79 :: PR: #5490
- Add MIMO forward step and per-token loss for hetero training by @yashaswikarnati :: PR: #5376
- Add Nemotron6-MoE VLM model provider for MIMO example by @yashaswikarnati :: PR: #5374
- ci: auto-retry test-data download in container-build job by @ko3n1g :: PR: #5498
- Force RL inference to CP=1 by @tdene :: PR: #5423
- Merge cu_seqlens across micro-batch for THD attention by @deepakn94 :: PR: #5454
- [split 4/4] Enable DSA CP and THD hooks by @HollowMan6 :: PR: #5246
- Fix fused MLA down projection with tensor parallelism by @sraman-rgb :: PR: #5383
- Fix NameError in is_flashinfer_min_version when check_equality=False by @adityasingh2

## retained text (truncated)



```
#### MoE

* **Quantile Balancing router:** adds auxiliary-loss-free load balancing using per-expert routing biases derived from quantile estimates ([\#5349](https://github.com/NVIDIA/Megatron-LM/pull/5349)).  
* **Fused shared-expert MLP:** enables shared experts through grouped GEMM, providing the path for fused GEMM, SwiGLU, and quantized execution ([\#5604](https://github.com/NVIDIA/Megatron-LM/pull/5604)).  
* **Packed-sequence MoE dispatch:** adds THD sequence-packing support to the DeepEP and HybridEP dispatcher backends ([\#5008](https://github.com/NVIDIA/Megatron-LM/pull/5008)).  
* **MoE routing analysis:** adds training/inference routing traces plus expert-concentration and one-layer-ahead predictability analysis tools ([\#5220](https://github.com/NVIDIA/Megatron-LM/pull/5220)).

#### Model Architecture and Support

* **HybridModel:** next-gen model class.  
  * `GPTModel` to `HybridModel` guide [published](https://docs.nvidia.com/megatron-core/developer-guide/nightly/user-guide/hybrid-model-migration.html).  
  * Added ability to directly load `GPTModel` checkpoints ([\#5792](https://github.com/NVIDIA/Megatron-LM/pull/5792)).  
  * Part 1 of the [design doc](https://github.com/NVIDIA/Megatron-LM/issues/4620) is complete, part 2 will be completed for 0.20.0.  
  * Deepseek v4 migrated to HybridModel on the `dev` branch ([\#5042](https://github.com/NVIDIA/Megatron-LM/pull/5042), [\#5485](https://github.com/NVIDIA/Megatron-LM/pull/5485), [\#4949](https://github.com/NVIDIA/Megatron-LM/pull/4949)).  
* **GPTModel:** [deprecation announcement](https://github.com/NVIDIA/Megatron-LM/issues/6071).  
* **DeepSeek Sparse Attention (DSA):** adds backend-neutral CP and THD support, pipeline-aware indexer-loss scaling, IndexShare, and fused cuDNN and TileLang implementations ([\#5244](https://github.com/NVIDIA/Megatron-LM/pull/5244), [\#5246](https://github.com/NVIDIA/Megatron-LM/pull/5246), [\#5099](https://github.com/NVIDIA/Megatron-LM/pull/5099), [\#5049](https://github.com/NVIDIA/Megatron-LM/pull/5049)).  
* **Multi-Latent Attention in HybridModel:** adds MLA layer support, Q/K norm spec resolution, and fused MLA projection handling to HybridModel ([\#4452](https://github.com/NVIDIA/Megatron-LM/pull/4452)).

#### Parallelism

* **FSDP:** MFSDP v2 is under development. See [the design doc](https://github.com/NVIDIA/Megatron-LM/blob/main/megatron/core/distributed/fsdp/src/docs/mfsdp_design.md) for links to the project tracker and implementation code.  
* **NCCL EP support:** adds NCCL EP token dispatcher backend ([\#5129](https://github.com/NVIDIA/Megatron-LM/pull/5129), [\#5735](https://github.com/NVIDIA/Megatron-LM/pull/5735)).  
* **Fuse per-sequence AlltoAll:** creates a unified AlltoAll ([\#4913](https://github.com/NVIDIA/Megatron-LM/pull/4913)).

#### Performance and Memory

* **CUDA Graph-compatible activation offloading:** enables fine-grained activation offloading during capture and replay, including deferred transfers and fused grouped-MLP offloading ([\#3697](https://github.com/NVIDIA/Megatron-LM/pull/3697), [\#5082](https://github.com/NVIDIA/Megatron-LM/pull/5082)).  
* **Streaming quantized checkpoint loading:** dequantizes FP8, MXFP8, blockwise-FP8, and NVFP4 tensors incrementally instead of allocating BF16 scratch space for the entire model ([\#4451](https://github.com/NVIDIA/Megatron-LM/pull/4451)).  
* **Partial CUDA Graph memory optimization:** moves graph surfaces into the global CUDA Graph memory pool and removes references and copies that retained excess buffers ([\#5451](https://github.com/NVIDIA/Megatron-LM/pull/5451)).

#### Precision and Optimizers

* **MXFP8 training paths:** take a look at our [Nemotron 3 Super MXFP8 example](https://github.com/NVIDIA/Megatron-LM/blob/core_r0.19.0/tests/functional_tests/test_cases/nemotron/nemotron3_super_release_gb200/model_config.yaml)  
* **Lion distributed-optimizer support:** routes Lion through `DistributedOptimizer` and adds single-moment checkpoint support, including Muon configurations that use Lion for scalar parameters ([\#5742](https://github.com/NVIDIA/Megatron-LM/pull/5742)).

#### Inference

* **Disaggregated-inference foundations:** adds heterogeneous inference-shard specifications, multi-destination refit, KV/Mamba-state resharding, local deterministic planning, and a NIXL copy backend ([\#5186](https://github.com/NVIDIA/Megatron-LM/pull/5186), [\#5187](https://github.com/NVIDIA/Megatron-LM/pull/5187), [\#5188](https://github.com/NVIDIA/Megatron-LM/pull/5188), [\#5872](https://github.com/NVIDIA/Megatron-LM/pull/5872)).  
* **Full-model CUDA Graphs for MTP inference:** enables full-model graph execution for Multi-Token Prediction inference ([\#4950](https://github.com/NVIDIA/Megatron-LM/pull/4950)).  
* **Load-aware prefix-cache routing:** routes requsts across inference workers using prefix-cache load information ([\#5607](https://github.com/NVIDIA/Megatron-LM/pull/5607)).

#### Datasets

* **Inter-document attention masking:** `GPTDataset` now emits document-boundary metadata, resets positions per document, and uses THD attention to prevent cross-document attention within packed samples, including TP and CP support ([\#5298](https://github.com/NVIDIA/Megatron-LM/pull/5298)).

#### Multimodal

* **MIMO heterogeneous training:** expands the current MIMO bullet with end-to-end non-colocated execution, stock training-loop integration, encoder prefetch, DDP overlap, and Nemotron MoE VLM/RADIO examples ([\#5602](https://github.com/NVIDIA/Megatron-LM/pull/5602), [\#5833](https://github.com/NVIDIA/Megatron-LM/pull/5833), [\#5979](https://github.com/NVIDIA/Megatron-LM/pull/5979)).

#### Known Issues

* **TE** **fused cross-entropy may** **degrade** **stability:** setting `cross_entropy_fusion_impl` to `te` causes convergence issues on some models. Use `native` instead, or upgrade Transformer Engine to 2.18, which carries the fix ([TE PR \#3193](https://github.com/NVIDIA/TransformerEngine/pull/3193)).  
* **SPCX grouped all-to-all payload corruption at high expert parallelism**: in the affected NCCL 2.30.5/SPCX stack, stale or uninitialized `ncclIbRequest.send.onlyWriteImm` state can misclassify sends in grouped variable-split BF16 all-to-all operations, causing payload loss or corruption. MCore's MoE `alltoall` dispatcher can expose this issue at high EP configurations, but the fault originates in the SPCX transport rather than the model or router. As a workaround, disable the external NCCL network plugin and use NCCL's internal IB transport (`NCCL_NET_PLUGIN=none NCCL_NET=IB`), or use the HybridEP dispatcher ([Megatron-Bridge issue \#5462](https://github.com/NVIDIA-NeMo/Megatron-Bridge/issues/5462)).

<details><summary>Changelog Details</summary>

- Add separate mtp_grad_scale_func for MTP loss scaling by @yfw :: PR: #3459
- [training migration] Migrate GPT builder by @maanug-nv :: PR: #4741
- Make Mamba conv params direct mixer params by @wujingyue :: PR: #4899
- Update oncall reviewer assignment by @Phlip79 :: PR: #5093
- Pass explicit process groups to hybrid logging by @yashaswikarnati :: PR: #4781
- Clean up top-level repository files by @Phlip79 :: PR: #5097
- [main] fix(moe): Fix several bugs for DSA rope and spec. by @yuzhongw-nvidia :: PR: #3026
- Move MIMO unit tests into models/mimo by @yashaswikarnati :: PR: #5063
- test: update DeepSeek FSDP2 GB200 memory golden by @wujingyue :: PR: #5094
- ci: make CI resilient to pip/uv network timeouts by @ko3n1g :: PR: #5118
- ci: treat docker container-removal conflict as flaky by @ko3n1g :: PR: #5120
- Fix GDN DTensor splitting for FSDP checkpointing by @conver334 :: PR: #4843
- Remove DeepEP hardware limit check by @janEbert :: PR: #4846
- Update transformer-engine dependency to revision 4220403 by @balasaajay :: PR: #5112
- Fix MoE aux_loss / z_loss gradient scaling with TP > 1 by @deepakn94 :: PR: #5047
- Update Claude copy workflow to enforce user restrictions and improve error messages by @balasaajay :: PR: #5117
- Add advisory process group guidance to Claude reviews by @yashaswikarnati :: PR: #5111
- build: cap pydantic<2.14 in transformer-engine dependency metadata by @cuichenx :: PR: #5125
- fix(test): skip scalar-less tensorboard event files in resume checks by @ko3n1g :: PR: #5121
- docs: fix contributor guide typo by @LeSingh1 :: PR: #4858
- ci(unit-tests): split slow unit-test buckets over 15min SLA by @ko3n1g :: PR: #5133
- Update MINOR version to 19 by @balasaajay :: PR: #5096
- Fix Muon QKV split for gated attention by @Moozy23232 :: PR: #4728
- Fix DSA indexer loss not averaged across micro-batches by @kaimo455 :: PR: #4070
- Roll input IDs for MTP labels by @yfw :: PR: #3457
- Adding blackwell tests by @shanmugamr1992 :: PR: #5113
- Refactor: Move paged stashing Triton kernels by @Phlip79 :: PR: #5003
- Relax atol for test_router_gating_linear router_dtype=torch.float32 by @adityasingh2400 :: PR: #4915
- Fix incorrect inference metadata tensor dtypes by @santhnm2 :: PR: #4855
- Disable TE cross entropy loss fusion by @mchrzanowski :: PR: #5115
- fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather by @ko3n1g :: PR: #4982
- Pass TP group to unfused cross entropy by @yashaswikarnati :: PR: #5128
- test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev by @ko3n1g :: PR: #5156
- ci(notify): mention mcore-oncall and philipp on critical CI events by @ko3n1g :: PR: #5152
- Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill by @mathemakitten :: PR: #3509
- ci: Disable a few gb200 test cases to support 2 branches. by @balasaajay :: PR: #5151
- build: Switch DSv3 on H100 to HybridEP by @balasaajay :: PR: #5164
- Add MTP acceptance rate metrics by @yfw :: PR: #3458
- Nemotron Ultra config for ModelOpt examples by @jenchen13 :: PR: #5159
- Make MTP / prefix cache stats persist for engine lifetime by @santhnm2 :: PR: #4101
- Restore Greptile configuration by @Phlip79 :: PR: #5166
- chore: bump `_code_freeze` workflow to `v1.4.2` by @ko3n1g :: PR: #5132
- ci: Remove docs build test in favor of release test by @chtruong814 :: PR: #5182
- Move TE cross entropy guard to training args by @yaoyu-33 :: PR: #5162
- Fix error in deepseek parser by @tdene :: PR: #5136
- Clamp `max_new_tokens` in MInf to mirror vllm by @tdene :: PR: #5181
- Fix logprob slicing for 0 generated token case by @santhnm2 :: PR: #5167
- [Perf] Fold frozen linear dgrad matmul by @cuichenx :: PR: #5092
- build: add managed = true to [tool.uv] by @kajalj22 :: PR: #5190
- Stabilize GB200 inference perf tests against cold-start noise by @shanmugamr1992 :: PR: #5171
- nvidia style guide audit for getting started folder by @megnvidia :: PR: #5168
- AI aided audit for Nvidia Style guidance by @megnvidia :: PR: #5141
- Avoid stat syscall in rerun result validation by @dimapihtar :: PR: #5107
- fix(combined-1f1b): release loss-node input storage after combined backward by @Wohox :: PR: #4909
- Enable selective recompute for `norm_out` in GDN layers  by @xuantengh :: PR: #4715
- fix(elastification): align with get_batch + utils refactors by @balasaajay :: PR: #5194
- docs: Update Latest News in README.md by @sbhavani :: PR: #3790
- Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. by @cspades :: PR: #4802
- ci: add smoke tests by @balasaajay :: PR: #5143
- Add mtp_detach_heads config to detach MTP head inputs by @yfw :: PR: #3456
- docs: fix install guide NGC container anchor by @Connor-XY :: PR: #5224
- Fuse per-sequence AlltoAll into a unified one in GDN forward by @xuantengh :: PR: #4913
- Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path by @yashaswikarnati :: PR: #5150
- Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool by @wujingyue :: PR: #4810
- Add named layout
```



document fetched 2026-08-26 05:55
