NVIDIA/Megatron-LM core_v0.18.0: NVIDIA Megatron Core 0.18.0
extract
<details><summary>Changelog Details</summary> - fix(ci): replace actions/setup-python with apt-get to avoid 429 rate limits by @ko3n1g :: PR: #4072 - ci: Fix package name for code-freeze workflow by @ko3n1g :: PR: #4077 - chore: bump `_code_freeze` workflow to `v0.86.0` by @ko3n1g :: PR: #4078 - Fix checkpoint inspector by @janEbert :: PR: #4079 - Update docs to conform to NVIDIA style guides by @megnvidia :: PR: #4068 - Miscellaneous inference fixes by @santhnm2 :: PR: #4030 - fix fine_grained_callables with fused rmsnorm residual by @CarlosGomes98 :: PR: #4026 - [Main][feat] Support overlapping A2A Combine backprop with wgrad GEMM by @Wohox :: PR: #3795 - Modify mfsdp default data-parallel-sharding-strategy by @wplf :: PR: #3691 - Fix fsdp_dtensor conversion for pretrained-only checkpoints by @DAISY-gh :: PR: #3912 - Guard NVshmem issues by @wdykas :: PR: #4093 - m-fsdp: wire use_precision_aware_optimizer from ddp_config to ParamAn… by @rapatel :: PR: #4024 - Megatron-FSDP: Add MXFP8 transpose helper buffer for Hybrid FSDP by @shjwudp :: PR: #3918 - feat(fsdp): use TE general_gemm for mixed-precision wgrad in FSDP path by @Victarry :: PR: #3822 - Megatron-FSDP: Fix insufficient double buffers during gradient reduce by @shjwudp :: PR: #4054 - Fix M-FSDP MXFP8 related BUGs by @shjwudp :: PR: #3991 - Megatron-FSDP: Make _pre_forward_param_unshard and _register_post_backward_hook formal by @shjwudp :: PR: #4029 - FIX: Use decoupled gradients for precision-aware M-FSDP grad norm by @XueSongTap :: PR: #3746 - Align chat completions endpoint with vLLM by @santhnm2 :: PR: #4063 - [Megatron-FSDP] Fix compatibility with frozen parameters and add unit tests by @shjwudp :: PR: #3287 - [M-FSDP] Refactor uneven dtensor to full tensor and add UT by @shjwudp :: PR: #3190 - Add agent instruction files by @Phlip79 :: PR: #4102 - Bump eopt version by @skyw :: PR: #4100 - Refactor emerging optimizer integration by @skyw :: PR: #4113 - Fix over provisioning of Mamba state memory when max_requests is set by @santhnm2 :: PR: #4114 - base strategy simplification by @dimapihtar :: PR: #4001 - add support for DCP and FSDP async save by @dimapihtar :: PR: #4027 - Add more emerging optimizers (#3907) by @skyw :: PR: #4119 - Fix FSDP checkpoint conversion and loading for Qwen3.5-VL by @DAISY-gh :: PR: #3936 - docs: update mcore optimizer docstrings to google style by @Akshat8510 :: PR: #2799 - Set tensor-parallel attributes irrespective of perform_initialization by @ilml :: PR: #4084 - docs: add developer-guide skill with CI/CD and failure navigation guidance by @ko3n1g :: PR: #4035 - chore: Move skills by @ko3n1g :: PR: #4136 - ci: Let Claude react to comment by @ko3n1g :: PR: #4135 - Nemotron3 Super GB200 release config by @maanug-nv :: PR: #4118 - Enable CUDA graph for ADAM optimizer by @vasunvidia :: PR: #3429 - Claude review should recommend testing by @Phlip79 :: PR: #4137 - cleanup: remove unused `scatter_gather_tensors_in_pipeline` argument by @Phlip79 :: PR: #4140 - fix: Remove fail-fast (-x) and guard distributed teardown against deadlock by @ko3n1g :: PR: #4139 - Claude: add respond-to-issue skill by @Phlip79 :: PR: #4141 - Fix muon getter backward compatability by @skyw :: PR: #4157 - Audit of user guide by @megnvidia :: PR: #4098 - Fix `RerunStateMachine` crash (`TypeError: 'NoneType' object is not subscriptable`) by not saving a checkpoint after a transient NaN / Inf by @yezhengmao1 :: PR: #3981 - Preserve type of decorated methods/classes by @nschank :: PR: #4062 - update muon test case to use new interface by @skyw :: PR: #4163 - [M-FSDP] Fix Tensor Parallel mode detection by @shjwudp :: PR: #3191 - fix: remove weights_only=False for multimodal example by @faradawn :: PR: #4104 - Cudagraphs: Fix sequence packing segfault more generally by @mathemakitten :: PR: #4162 - Make MTP work with materialize_only_last_token_logits by @santhnm2 :: PR: #4166 - Add unit test for Mamba EP inference (eager fallback with mixed CUDA graphs) by @santhnm2 :: PR: #4085 - update docs in respect to async changes by @dimapihtar :: PR: #4177 - update checkpointing docs in respect to async changes by @dimapihtar :: PR: #4208 - chore: improve build-and-test skill with trigger rules and dependency workflow by @ko3n1g :: PR: #4199 - Fix layerwise optimizer with `expt_dp_size=1` and contention with element-wise distributed optimizer by @skyw :: PR: #4138 - ci: add --cluster-a100/h100/gb200 args to trigger_internal_ci.py by @ko3n1g :: PR: #4195 - ci: Update golden values for nightly tests by @chtruong814 :: PR: #4215 - rename async_allgather to overlap_param_gather by @skyw :: PR: #4217 - Fix Slack sync for users with GitHub email privacy enabled by @Phlip79 :: PR: #4220 - Miscellaneous MTP inference fixes by @santhnm2 :: PR: #4191 - Move inference guards out of arguments.py by @mathemakitten :: PR: #4210 - Fix: enable fine-grained activation offloading for Mamba model. by @fanshiqing :: PR: #4173 - bump NVRx by @dimapihtar :: PR: #4178 - Update tokenizer args for Nemotron3 release config by @maanug-nv :: PR: #4239 - build: add dynamic git-versioning and drop rc0 pre-release tag by @ko3n1g :: PR: #4212 - Fix unnecessary permute padding for non-quantized MoE dispatch by @xiaoxi-wangfj :: PR: #4038 - Fix split state dict main by @kunlunl :: PR: #3676 - Add /split-pr Claude Code command for splitting PRs by CODEOWNERS by @Phlip79 :: PR: #4160 - Enable FP8 DPA for MXFP8 recipe by @vasunvidia :: PR: #4066 - Enable AG/RS overlap with explicit process group passing by @jeffnvidia :: PR: #3249 - Enable cpu_offloading with Full iteration CUDA graph by @vasunvidia :: PR: #3969 - Fix TransformerConfig validation for mixed dense/MoE upcycling by @rkteddy :: PR: #3647 - Remove cross-rank synchronization during checkpoint load & deprecate torch.distributed.checkpoint.state_dict_loader.load_state_dict by @asolergi-nv :: PR: #2864 - Fix incorrectly set decoupled_grad and DistOpt mechanics for MFSDP. by @cspades :: PR: #4133 - Refit Miscelaneous by @wdykas :: PR: #3973 - Add conditions_embeddings argument to TransformerBlock, TransformerLayer for DiT (diffusion transformer) by @huvunvidia :: PR: #4134 - Fix build_sequences_per_dataset output path arg usage by @DhineshPonnarasan :: PR: #4144 - ci: Flush pending CUDA work before the barrier in destroy_model_parallel by @chtruong814 :: PR: #4259 - Update oncall schedule by @Phlip79 :: PR: #4257 - docs(moe): Update MoE README by @sbhavani :: PR: #3664 - Revert "Add conditions_embeddings argument to TransformerBlock, Trans… by @ko3n1g :: PR: #4270 - reduce the number of shared expert streams by @yangbofun :: PR: #3752 - remove legacy Bert code by @dimapihtar :: PR: #4204 - [Main] Feat(moe): Gated delta net context parallel (CP) by @yuzhongw-nvidia :: PR: #2642 - remove t5 legacy code by @dimapihtar :: PR: #4203 - fix: handle list-typed process groups in ProcessGroupCollection.__repr__ by @cluster2600 :: PR: #3753 - Fix Context Parallelism documentation link by @liangxs :: PR: #4149 - [MLA] fix: Pad V when Q/V head dims differ for THD by @HollowMan6 :: PR: #3003 - fix(megatron-fsdp): build expt_device_mesh only for MoE models by @xuwchen :: PR: #3831 - Allow the evaluation batch size to differ from the training batch size by @michal2409 :: PR: #4014 - Add @NVIDIA/transformer review group to megatron/core/transformer/ by @Phlip79 :: PR: #4281 - Reset AG_pipeline bucket status after validation step. by @vasunvidia :: PR: #3155 - Enhance and fix NVTX for training by @yaox12 :: PR: #3642 - NVFP4 native weights for DDP by @WanZzzzzz :: PR: #4005 - Remove unnecessary arguments for layerwise distributed optimizer by @FDecaYed :: PR: #4272 - reuse grad buffer for layer-wise param allgather by @FDecaYed :: PR: #3751 - feat(ci): add strict review mode to Claude review workflow by @Victarry :: PR: #4197 - Fix stale approvals by @Phlip79 :: PR: #4280 - [MoE] Add a new score function to the router by @yaox12 :: PR: #3673 - [MoE] Improvement of shared expert overlap, support shared expert overlap for FlexDispatcher by @Victarry :: PR: #2207 - build: bump DeepEP to 34152ae by @ko3n1g :: PR: #4228 - ci: mark test_fused_indexer_loss_gradient_tp_consistency as flaky_in_dev by @ko3n1g :: PR: #4299 - Fix typo in PR4133. by @cspades :: PR: #4277 - ci: add retry loop to apt-get update to handle transient mirror sync failures by @ko3n1g :: PR: #4209 - fix: enforce correct pass thresholds for deterministic and approximate tests by @ko3n1g :: PR: #4238 - remove legacy biencoder and realm models by @dimapihtar :: PR: #4205 - ci: add configurable launcher support for functional tests (ft_launcher / torchrun) by @ko3n1g :: PR: #4298 - chore: document --target main for local Docker builds by @ko3n1g :: PR: #4307 - Extract args init to launch scripts by @maanug-nv :: PR: #4225 - [Main] Fix TE version check for retain_pinned_cpu_buffers in cpu offload by @BestJuly :: PR: #4267 - Fix documented shape by @janEbert :: PR: #3486 - ci: add sync-skills workflow, rename CLAUDE.md → AGENTS.md, move .claude/skills → skills/ by @ko3n1g :: PR: #4303 - Get `device` correctly when module returns a dict instead of individual tensor by @shifangx :: PR: #4265 - remove vision legacy code by @dimapihtar :: PR: #4202 - feat: long convergence resiliency for release tests by @ko3n1g :: PR: #4335 - ci(action): improve GitHub Actions output UX by @ko3n1g :: PR: #4337 - build: bump TransformerEngine to release_v2.14 by @ko3n1g :: PR: #4331 - feat: add create-issue skill by @ko3n1g :: PR: #4338 - M4 leftover for TE cuda graph by @shifangx :: PR: #3137 - fix: wait for async P2P send before deallocating output tensor by @ZhiyuLi-Nvidia :: PR: #4047 - ci(gb200): add 1-node mr-github functional test variants by @ko3n1g :: PR: #4334 - Fix potential coredump issue that occurs when saving a checkpoint by @ezioliao :: PR: #1871 - docs: bump versions1.json to 0.17.0 (latest) by @ko3n1g :: PR: #4360 - Port DeepSeek Sparse Attention to `MambaModel` by @janEbert :: PR: #3553 - Add tables and histogram for RL staleness by @tdene :: PR: #4097 - [docs] ci: use parent-relative json_url for version picker by @ko3n1g :: PR: #4367 - Fix bug with non-partial rollouts by @tdene :: PR: #3964 - Add QK layernorm support for dot-product attention in MambaModel by @Phlip79 :: PR: #4067 - Docs: improve docstrings and comments in example training loop by @DhineshPonnarasan :: PR: #4041 - feat(ckpt): add --async-ckpt-use-cpu-shm argument by @sbak5 :: PR: #4355 - cp: Fix UT timeout (#4310) by @chtruong814 :: PR: #4373 - Fix RL reward due to stop token by @tdene :: PR: #4096 - FA4 Inference by @wdykas :: PR: #4186 - Make param_index_map always use unpacked (full numel) offsets by @deepakn94 :: PR: #4328 - Add activation logging and tokens per expert logging by @Mellonta :: PR: #3842 - Fix RL to once again work with --skip-train by @tdene :: PR: #4249 - Fix Megatron initialization with extra_args_provider by @santhnm2 :: PR: #4327 - Rename MambaModel/MambaStack to HybridModel/HybridStack by @Phlip79 :: PR: #4099 - fix(ci): wrap uv install in retry block by @ko3n1g :: PR: #4387 - Call save_checkpoint_and_time() when saving checkpoint and compute elapsed duration for saving checkpoint before logging timer by @awsankur :: PR: #4263 - refactor(tests): move NCCL env vars from docker launcher to shell training script by @ko3n1g :: PR: #4390 - Remove packed_attention_mask unused parameter by @tdene :: PR: #3859 - Second batch of audit edits by @megnvidia :: PR: #4115 - revert: replace rampup batch size scheduler with custom step batch size schedules (#3779) by @ko3n1g :: PR: #4404 - Replace rampup batch size scheduler with custom step batch size schedules by @deepakn94 :: PR: #4411 - Megatron-FSDP: log mcore detection only after imports succeed by @wujingyue :: PR: #4400 - ci(gb200): re-enable tunable_overlap 1-node mr-github test by @ko3n1g :: PR: #4405 - Fix local docs building by @Phlip79 :: PR: #4416 - RL: Onload optimizer after logprobs computation by @tdene :: PR: #4235 - Add RL token throughput and packing metrics by @tdene :: PR: #3877 - ci: remove publish:merge_into_dev job by @ko3n1g :: PR: #4421 - docs: add data loading best practices for large-scale training by @sbhavani :: PR: #4236 - Fix: Auto enable manual registration and enhance the docummentation by @youngeunkwon0405 :: PR: #3295 - Fix nvtx_decorator to check _nvtx_enabled at call time by @minitu :: PR: #4184 - fix merges_file typo in megatron_hf_tokenizer by @chelseajohn :: PR: #4392 - Enable NullTokenizer for pretraining to reduce I/O access by @asolergi-nv :: PR: #4057 - docs: Add SECURITY.md by @chtruong814 :: PR: #4431 - Mamba inference opt by @wdykas :: PR: #4414 - DDP refactoring: Extract parameter layout computation into optimizer classmethod by @deepakn94 :: PR: #3812 - Update PR template with explicit request for issue by @Phlip79 :: PR: #4409 - Misc inference fixes by @sidsingh-nvidia :: PR: #4397 - Rename Mamba to Hybrid outside megatron/core by @Phlip79 :: PR: #4159 - Include mtp layers in token per expert logging by @Mellonta :: PR: #4412 - fix: NVRx async compatibility and defer resiliency import by @sbak5 :: PR: #4420 - fix(checkpoint_inspector): allow empty --param-to-param-group-map-json by @DAISY-gh :: PR: #4403 - Add the YARN support for hybrid_model by @guihong-nv :: PR: #4244 - [training migration] Add container class for config dataclasses by @maanug-nv :: PR: #4227 - Inference: Fix broken functional tests on gitlab by @sidsingh-nvidia :: PR: #4454 - SafeUnpickler class for safe pickle usage by @dimapihtar :: PR: #4319 - get rid of weights_only=False by @dimapihtar :: PR: #4434 - Inference | Per-block MoE routing storage for prefix caching by @lmcafee-nvidia :: PR: #4301 - Add troubleshooting tip for 'access forbidden' by @balasaajay :: PR: #4449 - Fix checkpoint loading with rerun state machine by @YangFei1990 :: PR: #4448 - Add misc CUDA graph sugar to CudaGraphManager by @tdene :: PR: #4425 - Inference: Add the embedding and output layer in the full_iteration_inference cuda graph scope for hybrid models by @sidsingh-nvidia :: PR: #4440 - Important bugfixes in local CG implementation that were leading to loss curve gaps for latent MoE models by @jiemingz :: PR: #4433 - fix: Replace polynomial rolling hash with SHA-256 for prefix caching by @lmcafee-nvidia :: PR: #4158 - feat(ckpt): expose validate_access_integrity knob on dist-ckpt load by @asolergi-nv :: PR: #4422 - Fix multivalidation by @RPrenger :: PR: #3388 - Add missing knob for reduce_scatter_with_fp32_accumulation by @WanZzzzzz :: PR: #4410 - Enable CUDA graphs for MTP inference by @santhnm2 :: PR: #4260 - checkpoint integrity verification by @dimapihtar :: PR: #4305 - Fix cache gating by @wdykas :: PR: #4455 - [Main] Fix FusedAdam.use_decoupled_grad mis-set for Megatron-FSDP. by @cspades :: PR: #4427 - add permute fusion into hybrid ep by @Autumn1998 :: PR: #4089 - Add ColocatedBridgeCommunicator for heterogeneous TP/DP MIMO training (NMFW-17) by @yashaswikarnati :: PR: #4368 - Fix incorrect bias display in extra_repr of Column/RowParallelLinear by @HelloWorldBeginner :: PR: #4330 - Fix assertion logic in combined_1f1b_schedule_for_interleaved_pipelining by @joapolarbear :: PR: #4276 - ci: Fix event name reference in CI workflow condition for merge group by @balasaajay :: PR: #4462 - Add nightly sync workflow from main to dev by @Phlip79 :: PR: #4165 - fix: handle list-format quant_cfg from ModelOpt PR #1094 by @ChenhanYu :: PR: #4187 - ci: also add Run MBridge tests label in nightly sync workflow by @Phlip79 :: PR: #4499 - [training migration] Add serialization features to config container by @maanug-nv :: PR: #4309 - Fix conflict with inference graphs by @tdene :: PR: #4504 - Add tools/prepare_cache.py for offline GPT dataset cache preparation by @asolergi-nv :: PR: #4080 - [build] fix: move mamba-ssm and causal-conv1d to optional [ssm] extra by @ko3n1g :: PR: #4517 - mamba: avoid redundant HBM reloads in causal_conv1d_update shift loop by @wdykas :: PR: #4460 - Standardize misc graph interface by @tdene :: PR: #4485 - Fix inference graph override in RL flow by @tdene :: PR: #4323 - Unify and refactor Megatron-FSDP documentation. by @cspades :: PR: #4418 - Revert "ci: add base_sha to codecov/codecov-action upload step (#4445)" by @chtruong814 :: PR: #4526 - Skills for running unit tests and working with slurm by @yashaswikarnati :: PR: #4502 - Reorganize order of operations in inference context and text generation controller by @tdene :: PR: #2929 - ci: Update CI workflow conditions to include merge group handling by @balasaajay :: PR: #4532 - ci: add base_sha to codecov/codecov-action upload step by @chtruong814 :: PR: #4540 - Fix release tests: remove --global-batch-size conflicting with --step-batch-size-schedule by @deepakn94 :: PR: #4545 - docs: use @file-path notation for file references in skills by @ko3n1g :: PR: #4542 - Support YAML quant recipe in PTQ and remove first/last layer modifier code by @jenchen13 :: PR: #4503 - Avoid nsys profile crash with CUDA graphs by @tdene :: PR: #4541 - fix(ci): add retry with backoff to approve-test-queue bot by @ko3n1g :: PR: #4559 - New allgathervdispatcher for inference and simplify old dispatcher. by @sidsingh-nvidia :: PR: #4258 - Fixes for modelopt examples and SFTTokenizer for transformers v5 by @jenchen13 :: PR: #4450 - Adding code for Flextron by @sheliang-nv :: PR: #4429 - Fix partial cudagraphs + HybridEP not properly triggering DDP hook by @jiemingz :: PR: #4500 - Ignore pytorch link anchors by @maanug-nv :: PR: #4582 - MoE dispatcher fixes: size NVLS dispatcher buffers from actual tensor sizes by @mathemakitten :: PR: #4576 - Finalize all builders in preprocess_data, not just the last key by @sayalinvidia :: PR: #4573 - refactor(skills): add when_to_use frontmatter, split ci-test-system, enforce skill workflow by @ko3n1g :: PR: #4574 - Make last_token_logits graphable by @tdene :: PR: #4552 - fix(ci): correct off-by-one in total_steps_evaluated formula by @ko3n1g :: PR: #4591 - Add fault injection support via nvidia_resiliency_ext. by @hexinw-nvidia :: PR: #4370 - Guard vocab reduce_scatter on TP > 1 by @mathemakitten :: PR: #4565 - Move inference context bookkeeping to CPU with ContextGPUView by @lmcafee-nvidia :: PR: #4306 - Enable InJob restart on failures. by @hexinw-nvidia :: PR: #4594 - Enable shared expert overlap with allgatherv in inference by @sidsingh-nvidia :: PR: #4570 - Add vLLM grouped gemm backend for MoE inference by @santhnm2 :: PR: #4566 - Move KD teacher loading to after Float16Module by @AAnoosheh :: PR: #4394 - ci: update gpt3_7b_tp4_pp1_memory_speed gb200 golden values by @ko3n1g :: PR: #4601 - Fix inference unit test by @maanug-nv :: PR: #4589 - Checkpoint conversion between GPT_model and Hybrid_model by @guihong-nv :: PR: #4482 - ci: add cadence input for test filtering in CI workflows by @balasaajay :: PR: #4561 - Fix `mtp_use_repeated_layer` behavior for GPT models by @rkarimimahab :: PR: #3965 - Handle SSM sharded tensor merge OOM with CPU fallback by @returnL :: PR: #4442 - FlashInfer sampling by @tdene :: PR: #2456 - Fix main2dev workflow by @Phlip79 :: PR: #4610 - Add logic to enable chunked MLP during training by @pengdurice :: PR: #3656 - Inference bug-fixes: Re-enable EP syncs for the legacy A2A dispatcher and re-simplify ep_sync accidentally reverted by #4306 by @sidsingh-nvidia :: PR: #4587 - Remove invalid `timeout` argument for dist.barrier by @zhaoyinglia :: PR: #4512 - Fix buffers in refit by @wdykas :: PR: #4580 - Named validation sets by @RPrenger :: PR: #4578 - Fix Hang in tests by @wdykas :: PR: #4575 - Single commit for main2dev nightly by @Phlip79 :: PR: #4614 - convert tokenizer args to config by @dimapihtar :: PR: #4406 - Siddharth/fix ep sync by @wdykas :: PR: #4607 - mmiranda working on another set of broken links by @megnvidia :: PR: #4534 - Fix gradient corruption with layerwise param all-gather overlap by @deepakn94 :: PR: #4609 - test: mark TestFu
retained text (truncated)
<details><summary>Changelog Details</summary> - fix(ci): replace actions/setup-python with apt-get to avoid 429 rate limits by @ko3n1g :: PR: #4072 - ci: Fix package name for code-freeze workflow by @ko3n1g :: PR: #4077 - chore: bump `_code_freeze` workflow to `v0.86.0` by @ko3n1g :: PR: #4078 - Fix checkpoint inspector by @janEbert :: PR: #4079 - Update docs to conform to NVIDIA style guides by @megnvidia :: PR: #4068 - Miscellaneous inference fixes by @santhnm2 :: PR: #4030 - fix fine_grained_callables with fused rmsnorm residual by @CarlosGomes98 :: PR: #4026 - [Main][feat] Support overlapping A2A Combine backprop with wgrad GEMM by @Wohox :: PR: #3795 - Modify mfsdp default data-parallel-sharding-strategy by @wplf :: PR: #3691 - Fix fsdp_dtensor conversion for pretrained-only checkpoints by @DAISY-gh :: PR: #3912 - Guard NVshmem issues by @wdykas :: PR: #4093 - m-fsdp: wire use_precision_aware_optimizer from ddp_config to ParamAn… by @rapatel :: PR: #4024 - Megatron-FSDP: Add MXFP8 transpose helper buffer for Hybrid FSDP by @shjwudp :: PR: #3918 - feat(fsdp): use TE general_gemm for mixed-precision wgrad in FSDP path by @Victarry :: PR: #3822 - Megatron-FSDP: Fix insufficient double buffers during gradient reduce by @shjwudp :: PR: #4054 - Fix M-FSDP MXFP8 related BUGs by @shjwudp :: PR: #3991 - Megatron-FSDP: Make _pre_forward_param_unshard and _register_post_backward_hook formal by @shjwudp :: PR: #4029 - FIX: Use decoupled gradients for precision-aware M-FSDP grad norm by @XueSongTap :: PR: #3746 - Align chat completions endpoint with vLLM by @santhnm2 :: PR: #4063 - [Megatron-FSDP] Fix compatibility with frozen parameters and add unit tests by @shjwudp :: PR: #3287 - [M-FSDP] Refactor uneven dtensor to full tensor and add UT by @shjwudp :: PR: #3190 - Add agent instruction files by @Phlip79 :: PR: #4102 - Bump eopt version by @skyw :: PR: #4100 - Refactor emerging optimizer integration by @skyw :: PR: #4113 - Fix over provisioning of Mamba state memory when max_requests is set by @santhnm2 :: PR: #4114 - base strategy simplification by @dimapihtar :: PR: #4001 - add support for DCP and FSDP async save by @dimapihtar :: PR: #4027 - Add more emerging optimizers (#3907) by @skyw :: PR: #4119 - Fix FSDP checkpoint conversion and loading for Qwen3.5-VL by @DAISY-gh :: PR: #3936 - docs: update mcore optimizer docstrings to google style by @Akshat8510 :: PR: #2799 - Set tensor-parallel attributes irrespective of perform_initialization by @ilml :: PR: #4084 - docs: add developer-guide skill with CI/CD and failure navigation guidance by @ko3n1g :: PR: #4035 - chore: Move skills by @ko3n1g :: PR: #4136 - ci: Let Claude react to comment by @ko3n1g :: PR: #4135 - Nemotron3 Super GB200 release config by @maanug-nv :: PR: #4118 - Enable CUDA graph for ADAM optimizer by @vasunvidia :: PR: #3429 - Claude review should recommend testing by @Phlip79 :: PR: #4137 - cleanup: remove unused `scatter_gather_tensors_in_pipeline` argument by @Phlip79 :: PR: #4140 - fix: Remove fail-fast (-x) and guard distributed teardown against deadlock by @ko3n1g :: PR: #4139 - Claude: add respond-to-issue skill by @Phlip79 :: PR: #4141 - Fix muon getter backward compatability by @skyw :: PR: #4157 - Audit of user guide by @megnvidia :: PR: #4098 - Fix `RerunStateMachine` crash (`TypeError: 'NoneType' object is not subscriptable`) by not saving a checkpoint after a transient NaN / Inf by @yezhengmao1 :: PR: #3981 - Preserve type of decorated methods/classes by @nschank :: PR: #4062 - update muon test case to use new interface by @skyw :: PR: #4163 - [M-FSDP] Fix Tensor Parallel mode detection by @shjwudp :: PR: #3191 - fix: remove weights_only=False for multimodal example by @faradawn :: PR: #4104 - Cudagraphs: Fix sequence packing segfault more generally by @mathemakitten :: PR: #4162 - Make MTP work with materialize_only_last_token_logits by @santhnm2 :: PR: #4166 - Add unit test for Mamba EP inference (eager fallback with mixed CUDA graphs) by @santhnm2 :: PR: #4085 - update docs in respect to async changes by @dimapihtar :: PR: #4177 - update checkpointing docs in respect to async changes by @dimapihtar :: PR: #4208 - chore: improve build-and-test skill with trigger rules and dependency workflow by @ko3n1g :: PR: #4199 - Fix layerwise optimizer with `expt_dp_size=1` and contention with element-wise distributed optimizer by @skyw :: PR: #4138 - ci: add --cluster-a100/h100/gb200 args to trigger_internal_ci.py by @ko3n1g :: PR: #4195 - ci: Update golden values for nightly tests by @chtruong814 :: PR: #4215 - rename async_allgather to overlap_param_gather by @skyw :: PR: #4217 - Fix Slack sync for users with GitHub email privacy enabled by @Phlip79 :: PR: #4220 - Miscellaneous MTP inference fixes by @santhnm2 :: PR: #4191 - Move inference guards out of arguments.py by @mathemakitten :: PR: #4210 - Fix: enable fine-grained activation offloading for Mamba model. by @fanshiqing :: PR: #4173 - bump NVRx by @dimapihtar :: PR: #4178 - Update tokenizer args for Nemotron3 release config by @maanug-nv :: PR: #4239 - build: add dynamic git-versioning and drop rc0 pre-release tag by @ko3n1g :: PR: #4212 - Fix unnecessary permute padding for non-quantized MoE dispatch by @xiaoxi-wangfj :: PR: #4038 - Fix split state dict main by @kunlunl :: PR: #3676 - Add /split-pr Claude Code command for splitting PRs by CODEOWNERS by @Phlip79 :: PR: #4160 - Enable FP8 DPA for MXFP8 recipe by @vasunvidia :: PR: #4066 - Enable AG/RS overlap with explicit process group passing by @jeffnvidia :: PR: #3249 - Enable cpu_offloading with Full iteration CUDA graph by @vasunvidia :: PR: #3969 - Fix TransformerConfig validation for mixed dense/MoE upcycling by @rkteddy :: PR: #3647 - Remove cross-rank synchronization during checkpoint load & deprecate torch.distributed.checkpoint.state_dict_loader.load_state_dict by @asolergi-nv :: PR: #2864 - Fix incorrectly set decoupled_grad and DistOpt mechanics for MFSDP. by @cspades :: PR: #4133 - Refit Miscelaneous by @wdykas :: PR: #3973 - Add conditions_embeddings argument to TransformerBlock, TransformerLayer for DiT (diffusion transformer) by @huvunvidia :: PR: #4134 - Fix build_sequences_per_dataset output path arg usage by @DhineshPonnarasan :: PR: #4144 - ci: Flush pending CUDA work before the barrier in destroy_model_parallel by @chtruong814 :: PR: #4259 - Update oncall schedule by @Phlip79 :: PR: #4257 - docs(moe): Update MoE README by @sbhavani :: PR: #3664 - Revert "Add conditions_embeddings argument to TransformerBlock, Trans… by @ko3n1g :: PR: #4270 - reduce the number of shared expert streams by @yangbofun :: PR: #3752 - remove legacy Bert code by @dimapihtar :: PR: #4204 - [Main] Feat(moe): Gated delta net context parallel (CP) by @yuzhongw-nvidia :: PR: #2642 - remove t5 legacy code by @dimapihtar :: PR: #4203 - fix: handle list-typed process groups in ProcessGroupCollection.__repr__ by @cluster2600 :: PR: #3753 - Fix Context Parallelism documentation link by @liangxs :: PR: #4149 - [MLA] fix: Pad V when Q/V head dims differ for THD by @HollowMan6 :: PR: #3003 - fix(megatron-fsdp): build expt_device_mesh only for MoE models by @xuwchen :: PR: #3831 - Allow the evaluation batch size to differ from the training batch size by @michal2409 :: PR: #4014 - Add @NVIDIA/transformer review group to megatron/core/transformer/ by @Phlip79 :: PR: #4281 - Reset AG_pipeline bucket status after validation step. by @vasunvidia :: PR: #3155 - Enhance and fix NVTX for training by @yaox12 :: PR: #3642 - NVFP4 native weights for DDP by @WanZzzzzz :: PR: #4005 - Remove unnecessary arguments for layerwise distributed optimizer by @FDecaYed :: PR: #4272 - reuse grad buffer for layer-wise param allgather by @FDecaYed :: PR: #3751 - feat(ci): add strict review mode to Claude review workflow by @Victarry :: PR: #4197 - Fix stale approvals by @Phlip79 :: PR: #4280 - [MoE] Add a new score function to the router by @yaox12 :: PR: #3673 - [MoE] Improvement of shared expert overlap, support shared expert overlap for FlexDispatcher by @Victarry :: PR: #2207 - build: bump DeepEP to 34152ae by @ko3n1g :: PR: #4228 - ci: mark test_fused_indexer_loss_gradient_tp_consistency as flaky_in_dev by @ko3n1g :: PR: #4299 - Fix typo in PR4133. by @cspades :: PR: #4277 - ci: add retry loop to apt-get update to handle transient mirror sync failures by @ko3n1g :: PR: #4209 - fix: enforce correct pass thresholds for deterministic and approximate tests by @ko3n1g :: PR: #4238 - remove legacy biencoder and realm models by @dimapihtar :: PR: #4205 - ci: add configurable launcher support for functional tests (ft_launcher / torchrun) by @ko3n1g :: PR: #4298 - chore: document --target main for local Docker builds by @ko3n1g :: PR: #4307 - Extract args init to launch scripts by @maanug-nv :: PR: #4225 - [Main] Fix TE version check for retain_pinned_cpu_buffers in cpu offload by @BestJuly :: PR: #4267 - Fix documented shape by @janEbert :: PR: #3486 - ci: add sync-skills workflow, rename CLAUDE.md → AGENTS.md, move .claude/skills → skills/ by @ko3n1g :: PR: #4303 - Get `device` correctly when module returns a dict instead of individual tensor by @shifangx :: PR: #4265 - remove vision legacy code by @dimapihtar :: PR: #4202 - feat: long convergence resiliency for release tests by @ko3n1g :: PR: #4335 - ci(action): improve GitHub Actions output UX by @ko3n1g :: PR: #4337 - build: bump TransformerEngine to release_v2.14 by @ko3n1g :: PR: #4331 - feat: add create-issue skill by @ko3n1g :: PR: #4338 - M4 leftover for TE cuda graph by @shifangx :: PR: #3137 - fix: wait for async P2P send before deallocating output tensor by @ZhiyuLi-Nvidia :: PR: #4047 - ci(gb200): add 1-node mr-github functional test variants by @ko3n1g :: PR: #4334 - Fix potential coredump issue that occurs when saving a checkpoint by @ezioliao :: PR: #1871 - docs: bump versions1.json to 0.17.0 (latest) by @ko3n1g :: PR: #4360 - Port DeepSeek Sparse Attention to `MambaModel` by @janEbert :: PR: #3553 - Add tables and histogram for RL staleness by @tdene :: PR: #4097 - [docs] ci: use parent-relative json_url for version picker by @ko3n1g :: PR: #4367 - Fix bug with non-partial rollouts by @tdene :: PR: #3964 - Add QK layernorm support for dot-product attention in MambaModel by @Phlip79 :: PR: #4067 - Docs: improve docstrings and comments in example training loop by @DhineshPonnarasan :: PR: #4041 - feat(ckpt): add --async-ckpt-use-cpu-shm argument by @sbak5 :: PR: #4355 - cp: Fix UT timeout (#4310) by @chtruong814 :: PR: #4373 - Fix RL reward due to stop token by @tdene :: PR: #4096 - FA4 Inference by @wdykas :: PR: #4186 - Make param_index_map always use unpacked (full numel) offsets by @deepakn94 :: PR: #4328 - Add activation logging and tokens per expert logging by @Mellonta :: PR: #3842 - Fix RL to once again work with --skip-train by @tdene :: PR: #4249 - Fix Megatron initialization with extra_args_provider by @santhnm2 :: PR: #4327 - Rename MambaModel/MambaStack to HybridModel/HybridStack by @Phlip79 :: PR: #4099 - fix(ci): wrap uv install in retry block by @ko3n1g :: PR: #4387 - Call save_checkpoint_and_time() when saving checkpoint and compute elapsed duration for saving checkpoint before logging timer by @awsankur :: PR: #4263 - refactor(tests): move NCCL env vars from docker launcher to shell training script by @ko3n1g :: PR: #4390 - Remove packed_attention_mask unused parameter by @tdene :: PR: #3859 - Second batch of audit edits by @megnvidia :: PR: #4115 - revert: replace rampup batch size scheduler with custom step batch size schedules (#3779) by @ko3n1g :: PR: #4404 - Replace rampup batch size scheduler with custom step batch size schedules by @deepakn94 :: PR: #4411 - Megatron-FSDP: log mcore detection only after imports succeed by @wujingyue :: PR: #4400 - ci(gb200): re-enable tunable_overlap 1-node mr-github test by @ko3n1g :: PR: #4405 - Fix local docs building by @Phlip79 :: PR: #4416 - RL: Onload optimizer after logprobs computation by @tdene :: PR: #
document fetched 2026-06-29 10:23