[WIP] I had Claude build a story-to-video pipeline: paste a short story → local LLM storyboards it → ComfyUI renders Krea 2 images or MiniMax H3 clips with auto-wired character refs. What should I ask it to add or fix?
retained text
Full disclosure up front: Claude wrote all of this, including this post. I described what I wanted, answered a few questions about my setup (local weights, local LLM), and it designed and built the whole thing. I’m the one testing it and deciding what to ask for next, which is where I’d like your help. **The idea** You paste a short story. A local LLM (Ollama, whatever model you want) reads it and breaks it into an ordered list of scenes plus a cast list where every character gets a locked physical description. Then it writes a model-specific prompt for each scene. From there you pick a route: * **Krea 2 route:** each scene becomes one still image. The whole story ends up as a sequence of frames, like a picture book. * **H3 route:** each scene becomes a \~10s MiniMax H3 clip with native audio. Before any clips render, it generates a clean character sheet per character with Krea 2, then feeds those into H3’s reference-to-video mode as `<Picture 1>`, `<Picture 2>` etc. The LLM is told exactly which character maps to which tag when it writes the clip prompt, so the refs and the prompt agree. Everything runs one after another on a single click. Character sheets first, then scenes in order. **How Claude built it** It talked me out of doing the orchestration inside ComfyUI. A story produces an unknown number of scenes and ComfyUI’s static graph doesn’t want to loop over N dynamic items. So ComfyUI is just the render farm: a Python script talks to it over the HTTP API, queues one job at a time, waits, downloads, moves on. The workflows are my own API-format exports of the official Krea 2 and H3 templates. I rename a handful of nodes (PROMPT, SEED, OUTPUT, REF\_1..3) and the script patches them by title. Claude specifically refused to hand-write the node JSON because the H3 nodes changed as recently as August and it couldn’t verify them, which I appreciated more than if it had just guessed. There’s a separate R2V export per reference count (1/2/3 refs) because you can’t leave a LoadImage empty. The LLM step is two passes. First pass is a “storyboard director” prompt that returns JSON (characters + scenes). Second pass writes the actual prompts, with MiniMax’s official R2V prompting guide and Krea’s prompting doc loaded into the system prompt. Output goes to a `plan.json` I can edit before burning GPU time. Render is resumable. It also wrote a mock test for the ref-wiring logic and caught its own bug (refs were being ordered by cast list instead of scene importance, which would have mismatched the Picture tags). **What I want to ask it to do next — help me prioritize** This is the actual reason for the post. I can keep feeding Claude requests, but I’m not experienced enough with multi-shot video to know what’s worth asking for. Some things I’ve thought of: 1. Multiple angles per character sheet (front/side/back) instead of one frontal shot, if R2V really benefits from it. Does it? 2. A continuity pass: have the LLM re-read the whole plan after writing it and check that locations, time of day and costume stay consistent between adjacent scenes. 3. First/last-frame chaining between clips so scene N+1 starts where scene N ends, instead of each clip being independent. 4. Automatic stitching of the clips into one video with crossfades, plus optional narration from a local TTS reading the story excerpt. 5. A quick review loop: render a cheap low-res preview of each scene first, let me reject/regenerate, then do the full render. 6. Pull it back into pure ComfyUI if there’s a loop-node approach that’s actually stable now. Which of these matter, which are a waste, and what am I not thinking of? Also curious what prompting tricks people have found for H3 R2V that I should add to the guide file the LLM reads. Still rough and not released. Not going to post it until I’ve run a few full stories through it. Happy to share the plan.json format or the prompts if anyone wants to see them.
document fetched 2026-10-31 00:32 · canonical https://reddit.com/r/StableDiffusion/comments/1xj2sd6/wip_i_had_claude_build_a_storytovideo_pipeline