Run AI Models for Edge Application Development with Just a Few Commands

retained text

AI is expanding beyond the data center as developers build a new generation of agents to act at the edge: monitoring driver attention and cabin conditions, helping robots understand their surroundings and plan tasks, and coordinating fleets of machines on production lines. These physical AI applications rely on models that perceive, reason, and act within tight memory, power, and latency budgets. But getting a model ready to run can involve substantial setup.
NVIDIA DGX Spark, now available in a 64GB configuration, gives embedded developers a platform to prototype, fine-tune, and test models and agents. NVIDIA TensorRT Edge-LLM provides a lightweight C++ inference runtime designed for embedded workloads, bringing the same inference framework to development on Spark and deployment on platforms such as NVIDIA Jetson Thor, NVIDIA DRIVE AGX Thor, and NVIDIA IGX Thor.
You can get started in just a few commands with the trtedge command-line interface (CLI). It downloads your selected model and builds TensorRT engines optimized for inference on Spark.
In this tutorial, you’ll run NVIDIA Nemotron-3.5 Lightning, a text-only model, on Spark using TensorRT Edge-LLM. You’ll test prompts in your terminal and serve the model through an API for application prototyping. You’ll also enable DSpark speculative decoding to accelerate generation.
Prerequisites: Set up your development environment
To get started, you’ll need a DGX Spark with a supported NVIDIA software stack, NVIDIA CUDA 13, TensorRT 10.16.1.11 or newer within the 10.x series, and Python 3.10 to 3.12. The examples below were tested with TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11.
Open a terminal on your DGX Spark, create and activate a virtual environment, and install TensorRT Edge-LLM:
python3 -m venv --system-site-packages .venv-edgellm
source .venv-edgellm/bin/activate
python -m pip install --upgrade pip
python -m pip install "tensorrt-edgellm[server]==0.11.1"
Note: The virtual environment keeps newly installed packages separate while allowing access to all system Python packages, including TensorRT.
Check the TensorRT version and confirm that the TensorRT Edge-LLM runtime loads:
python -c "import tensorrt as trt; v = tuple(map(int, trt.__version__.split('.'))); assert (10, 16, 1, 11) <= v < (11,), f'TensorRT >=10.16.1.11 and <11 required; found {trt.__version__}'; from tensorrt_edgellm import runtime; runtime.load()"
Note: If compatible system TensorRT bindings are unavailable, follow the NVIDIA TensorRT pip installation instructions to install a compatible TensorRT 10.x version in the activated environment, then rerun the verification command.
With the virtual environment still active, start a chat with Nemotron-3.5 Lightning:
trtedge chat nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
On your first launch, wait for the model download (approximately 21GB) and engine build to finish. Files are stored in ~/.cache/tensorrt-edgellm by default, so allow space for both the model and engines. Later launches reuse matching cached engines.
Once the chat prompt appears, enter:
Convert 25°C to Fahrenheit. Answer in one sentence.
Example response:
25°C is equal to 77°F.
Continue the conversation with follow-up questions, or type /exit to end the session.
Serve your model
Serve your model through a local OpenAI-compatible API on Spark for application development and testing.
With your virtual environment active, start the server:
trtedge serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --port 8000
The server builds or reuses an engine for this configuration. Once startup completes, open a second terminal on your DGX Spark and check that it is ready:
curl -fsS http://127.0.0.1:8000/health/ready 
A response containing "status":"ready" confirms that the server is ready. Send your first prompt to the chat completions endpoint:
curl -fsS -N http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
  "model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
  "messages": [
    {
      "role": "user",
      "content": "Convert 25°C to Fahrenheit. Answer in one sentence."
    }
  ],
  "stream": true,
  "temperature": 0,
  "max_tokens": 128
}'
The response streams as JSON events. Keep the server running for more requests, or press Ctrl+C in the server terminal to stop it.
Optional: Enable speculative decoding
You can speed up generation with DSpark speculative decoding, which uses a draft model to propose several tokens for the main model to verify in a single pass. To enable it with three draft tokens, press Ctrl+C in the server terminal, then restart the server with DSpark enabled:
trtedge serve nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 \
  --host 127.0.0.1 \
  --port 8000 \
  --speculative-config '{
    "method": "dspark",   
    "model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark",
    "num_speculative_tokens": 3
  }'
Once the server is ready, send another prompt to the chat completions endpoint to try DSpark-enabled generation.
You can experiment with num_speculative_tokens and other settings to tune generation speed for your workload. See the speculative decoding documentation to learn more.
Performance on DGX Spark 64GB
TensorRT Edge-LLM delivers low response latency and fast token generation with NVIDIA Nemotron-3.5 Lightning on DGX Spark 64GB.
The table below reports time to first token (TTFT) and generation speed with and without DSpark speculative decoding.
Generation speed is the average token generation rate per request, excluding time to first token. DSpark uses three draft tokens. Peak total system memory use during inference with DSpark was 27.9GB, including the OS.
Measured on DGX Spark 64GB running DGX OS 7.6.0 (Ubuntu 24.04.4 LTS), using NVFP4 with thinking enabled, batch size one, one concurrent streaming request, and a 128-token output limit. Average input lengths were 1,153 tokens for SpeedBench 1K, 2,247 for SpeedBench 2K, and 139 for HumanEval.
Next steps
Try another model from the supported models list and continue developing your application on Spark. Options include vision-language models that interpret images and video, as well as speech recognition and text-to-speech models.
Explore the TensorRT Edge-LLM documentation for model-specific workflows and requirements on your target embedded platform.
And visit the Nemotron developer page for resources to get started. Explore open Nemotron models and datasets on Hugging Face and Blueprints on build.nvidia.com.

document fetched 2026-10-31 02:54 · canonical https://developer.nvidia.com/blog/run-ai-models-for-edge-application-development-with-just-a-few-commands