Looking for some help optimizing for a 300-page manual and transcripts
retained text
Hello, I am just getting started and I got great answers to my [previous post asking what was possible](https://www.reddit.com/r/LocalLLM/comments/1xndiy1/how_powerful_of_a_local_llm_is_needed_to_build_a/). I still have lots to learn but, with the help of Claude, I was able to get something running and it is working pretty well. My goal as been to build a fully local AI study assistant to analyze a \~300-page technical training manual alongside instructor transcripts and generate chapter-based study guides and Quizlet flashcards with accurate page citations. **The problem is:** I'm running into is this error: `request (34009 tokens) exceeds the available context size (32768 tokens).` And this was after doing lots of tweaking and getting it not fail at 16,000 tokens. There are so many settings and configurations that I might be able to change to improve that, but it is overwhelming and I have no idea what to do. I would really appreciate any ideas, suggestions that I can do to get this working for what I am trying to achieve. Or let me know that it isn't possible. Here's my setup (sorry, AI slop to make it clearer): **Hardware:** * **Windows 11 PC:** i7-14700K, 64GB RAM, RTX 4070 SUPER (12GB VRAM) * **Unraid server:** Core Ultra 9 285K, 32GB RAM, RTX 5070 Ti (16GB VRAM) **Software / Models:** * llama.cpp (CUDA, router mode) * Open WebUI 0.12.3 * **Qwen3.6-35B-A3B UD-Q4\_K\_XL:** Main model, context increased from 32K to 64K * **Gemma 4 12B Q4\_K\_M:** Secondary model * Both models run on the Windows PC. **RAG configuration:** * Open WebUI Knowledge Base with the PDF and Markdown transcripts * Embeddings: bge-m3 (1024 dimensions), hosted on Unraid * Reranker: bge-reranker-v2-m3 on Unraid (API working; Open WebUI integration unverified) * Focused Retrieval, chunk size 1000, overlap 100, Top K = 3 * All inference and document processing must remain local. **More details on the issue:** Basic RAG questions work well. However, when I ask the model to analyze an entire chapter, cross-reference relevant trainer discussions, and create 15–25 Quizlet flashcards, I encounter context errors as I said above: `request (34009 tokens) exceeds the available context size (32768 tokens)` Increasing Qwen's configured context to 64K eliminated the immediate context error, but generation then stopped halfway through creating flashcards without an obvious error. I haven't confirmed whether an output-token limit caused this. I've tried reducing retrieval results, using smaller prompts, and starting fresh chats. These help, but I'd prefer a more automated and comprehensive workflow. Thank you!
document fetched 2026-11-14 00:47 · canonical https://reddit.com/r/LocalLLaMA/comments/1xuwowk/looking_for_some_help_optimizing_for_a_300page