loading
loading
Tencent Opens Hy-Nomas, an Interactive Video World Model Running on One RTX 4090 Tencent's Hunyuan team released Hy-Nomas, an open source interactive video world model with text and camera control that runs on a single RTX 4090. - Tencent Hunyuan released Hy-Nomas, an open-source interactive video world model under Apache 2.0. - Supports free-form text prompts and camera trajectory control across autoregressive video segments. - A lifelong KV cache preserves world state and object consistency across long multi-turn generations. - Runs on a single RTX 4090 via INT8 DiT weights, sliding-window history, and CPU offload. - Ships cinematic and dual-arm robotics LoRA adapters, plus full fine-tuning and captioning pipelines. - Weights, demo, and training code live on Hugging Face and the web demo. Tencent’s Hunyuan team has released Hy-Nomas, an Apache 2.0-licensed interactive video world model. It generates scenes one segment at a time, accepting new free-form text instructions or camera trajectories during generation. The release includes training code, inference scripts, a web demo, LoRA fine-tuning tools, and Hugging Face weights, including an INT8 checkpoint configured for a single NVIDIA GeForce RTX 4090. In Hy-Nomas, “world model” refers to a video generator that predicts each new segment from prior generated footage and the latest control input. Users can redirect the scene during a session while the model carries visual context forward, creating a development loop closer to an interactive environment than a fixed prompt-to-video clip. How Hy-Nomas remembers a scene Hy-Nomas uses a diffusion transformer, or DiT, to generate video in autoregressive segments. Diffusion handles iterative denoising, while the transformer processes the sequence and its controls. Each segment receives either a text prompt or a camera-control signal. The model’s “lifelong KV cache” stores attention keys and values from earlier segments. Giving later segments access to that history helps preserve objects, lighting, and geometry as generation continues, although the release does not yet provide benchmarks for long-horizon consistency. - Fast sampling: The reference interactive setup uses eight denoising steps. The distilled checkpoint also supports four-step and six-step inference. - Compressed video: The model operates on 32-channel DCAE-V latents, a compact representation of the source frames. - Video decoding: A fine-tuned PixelUMM decoder converts those latents back into frames. - Prompt processing: A separate Nomas-PE model turns raw instructions into structured generation prompts during interactive sessions. - Domain adapters: The repository includes a cinematic LoRA and a robotics LoRA trained on the RealSource-World dual-arm dataset. LoRA adapters modify a model through small trainable matrices, reducing the cost of domain adaptation. Three checkpoints target different jobs How the RTX 4090 configuration saves memory The dedicated low-memory configuration combines quantization, bounded history, and staged CPU offloading to fit inference on one RTX 4090. Its main settings are: - INT8 storage: DiT weights and the KV cache use INT8, while linear and attention calculations remain in BF16. - Sliding history: A fixed context window bounds KV-cache growth. This reduces memory use by limiting how much earlier history remains directly available. - CPU offloading: The text encoder, DiT, and VAE move between system memory and GPU memory as each stage runs. - Optimized kernels: The configuration uses FlexAttention for the DiT and CUTLASS FP8 for video decoding. - No sequence parallelism: The single-GPU path disables the sequence splitting used by the multi-GPU configuration. Default multi-GPU inference enables sequence parallelism, which distributes sequence processing across devices. The supplied supervised fine-tuning configuration uses eight-way sequence parallelism. On 128 GPUs, that produces 16 data-parallel replicas, each spanning eight GPUs. The reference LoRA recipe sets both rank and alpha to 256 and trains for 2,000 steps with the Muon optimizer. Fine-tuning keeps the fast inference path Hy-Nomas separates adaptation from deployment. Developers train a LoRA against the non-distilled base checkpoint, then merge the resulting adapter into the distilled checkpoint used for inference. This workflow preserves access to the distilled model’s four-step, six-step, and eight-step generation modes. - Caption each video segment and attach any normalized camera trajectories. - Encode the frames into DCAE-V latents. - Package the processed samples as Arrow or Parquet data for supervised fine-tuning. - Train the LoRA against the base checkpoint. - Merge the adapter into the distilled checkpoint for deployment. From packed data to merged weights python scripts/data_processing/build_sft_packing_plan.py \ --input /path/to/train.arrow \ --output /path/to/train_packing.json \ --max-seq-len 360000 \ --num-groups 16 bash scripts/train/train_from_config.sh configs/experiment/sft.yaml \ dit_weight="$Hy_Nomas_Model_Path/DiT/base_ar/...safetensors" \ dataset.data_file=/path/to/train.arrow python -m scripts.train.merge_lora \ --base "$Hy_Nomas_Model_Path/DiT/distilled/...safetensors" \ --adapter ./outputs/segment_lora/adapter \ --output ./outputs/merged_model.safetensors Where developers can apply it - Robotics research: The dual-arm adapter can support synthetic observation generation and controlled scenario exploration. Generated video still requires validation before use as physically accurate simulation or policy-training data. - Interactive prototypes: Text and camera controls can drive game-like scenes without requiring a fixed clip to be regenerated from the beginning. - Cinematic planning: The cinematic adapter supports iterative previs and storyboard development. - Long-horizon video research: The exposed KV-cache implementation gives researchers a concrete system for studying memory, drift, and temporal consistency. The benchmarks have not arrived At publication, the GitHub repository had a single commit and listed the technical report as forthcoming. No public benchmark results establish prompt adherence, camera-control accuracy, generation latency, physical consistency, or long-horizon stability against systems such as Genie 3, Wan, or Veo. The low-memory path also leaves practical deployment questions unanswered. CPU offloading requires additional system RAM and introduces CPU-GPU transfer overhead, but the release does not specify the required RAM or measured throughput on an RTX 4090. The quality impact of INT8 weights, an INT8 KV cache, and a sliding history window also remains unquantified. Hy-Nomas gives developers an inspectable stack for interactive video generation, including model weights, data preparation, training, LoRA adaptation, and local inference. The Apache 2.0 license also permits modification and commercial use under its terms. Developers can test the interactive demo or download the code from the GitHub repository.
document fetched 2026-10-24 15:22