← sources

on prem LLM stack for data that cant leave the building

source: reddit (reddit:LocalLLaMA)published 2026-09-30 07:18ingested 2026-10-01 02:44

retained text

Running the model locally is not a problem thats easy part but the leaks are the third party integrations along with it, like you designed everything perfect and then just added a cloud api along with it maybe a hosted judge for evals or a tracing saas or embedding point. one http call and the on prem things over

parts we already keep local are

1. runtime: llama.cpp/ vllm /ollama

2. models: qwen or llama family depending on rig

3. vector db: pgvector or qdrant 

some that leak but remain unnoticed: 

1. ingestion: pdfs and scans for some projects need a parse and the ocr step before chunking them and its where we often reach for a cloud parser and break the rule, although we can keep it local with liteparse sort of inbound parsers or other open source options on huggingface

2. Eval: plenty of local setups still need prompts and outputs to a hosted judge or a tracing dashboard to see quality which is the same leak but seems different. instead a  local score set or a local judge model and keeping it self hosted if possible handles the tracing part

I am curious to know about others end to end stack who keep it 100% local, eager to learn more

document fetched 2026-10-01 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1wtyflt/on_prem_llm_stack_for_data_that_cant_leave_the