What platform do banks and insurers use to run LLMs in production compliantly?
retained text
I’m on a three-person ML team at a small bank in Germany we run Llama 3.3 70B on vLLM across two A100s in our own server room for an internal policy assistant It works well, but internal audit just asked which prompt and document version produced a wrong answer last month, and we couldn’t tell them. We’re choosing between two setups one is fully DIY, with LiteLLM for keys and logging and self-hosted Langfuse for traces other is Orq ai, which runs on-prem with Helm (even air-gapped), takes our vLLM endpoint as a private model, keeps a version history for every prompt and adds audit logs on the enterprise plan.The catch is that enterprise pricing isn’t public Did your auditors accept logs from tools you host yourselves, or did they want a vendor they could point at? That’s really what decides this for us
document fetched 2026-11-20 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1y0al3r/what_platform_do_banks_and_insurers_use_to_run