What's the actual self-hosted RAG stack when docs can't leave your cloud?
retained text
We ended up pulling our managed APIs out entirely, because our documents aren't allowed to leave our own infrastructure for compliance reasons. And our problem has nothing to do with cost, because the per-page pricing was perfectly reasonable. And that 'docs not able leave our cloud' constraint rules out Textract, OpenAI embeddings and Cohere in one go, since every one of them ships your data off to someone else. That forced us into self-hosting the whole pipeline, OCR, embeddings and reranking included, and after a couple of weeks of testing this is the shortlist I've built up: docling holds up well on clean digital PDFs, but it gets noticeably shaky once you hand it scans or handwriting PaddleOCR-VL deals well with messy multilingual layouts and stays small enough to run on a fairly modest GPU TEI is dependable for embeddings, though it only serves one model per server Infinity handles embeddings and reranking together from a single process SIE runs OCR, embeddings and reranking behind one endpoint with model-swapping by ID, and it runs inside your own cloud. What you give up by self-hosting is all the turnkey convenience you used to get for free, so there are no parsed tables handed to you, no confidence scores, and the entire pipeline becomes yours to own and maintain. We've made peace with that because the residency requirement simply isn't negotiable for us, but I'll admit that if cost were the only thing driving this decision it would be a far harder one to justify.
document fetched 2026-10-28 00:30 · canonical https://reddit.com/r/selfhosted/comments/1xgirpf/whats_the_actual_selfhosted_rag_stack_when_docs