# on prem LLM stack for data that cant leave the building — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/a324d068d9165181

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-09-30 07:18 ingested 2026-10-01 02:44

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1wtyflt/on_prem_llm_stack_for_data_that_cant_leave_the/)

## retained text



```
Running the model locally is not a problem thats easy part but the leaks are the third party integrations along with it, like you designed everything perfect and then just added a cloud api along with it maybe a hosted judge for evals or a tracing saas or embedding point. one http call and the on prem things over

parts we already keep local are

1. runtime: llama.cpp/ vllm /ollama

2. models: qwen or llama family depending on rig

3. vector db: pgvector or qdrant 

some that leak but remain unnoticed: 

1. ingestion: pdfs and scans for some projects need a parse and the ocr step before chunking them and its where we often reach for a cloud parser and break the rule, although we can keep it local with liteparse sort of inbound parsers or other open source options on huggingface

2. Eval: plenty of local setups still need prompts and outputs to a hosted judge or a tracing dashboard to see quality which is the same leak but seems different. instead a  local score set or a local judge model and keeping it self hosted if possible handles the tracing part

I am curious to know about others end to end stack who keep it 100% local, eager to learn more
```



document fetched 2026-10-01 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1wtyflt/on_prem_llm_stack_for_data_that_cant_leave_the
