Is self-hosting embeddings actually cheaper or are we kidding ourselves?
retained text
There's this line that gets repeated constantly, that self-hosting your embeddings is automatically cheaper, and I wanted to actually pressure-test it against our own workload before I believed a word of it. The thing people conveniently skip over is that a GPU costs you exactly the same whether it's pinned at full load or sitting there doing nothing, so the moment your card isn't busy the per-token APIs quietly start winning. We index in bursts rather than running constantly, which meant that for a good while we were effectively paying premium rates for a very expensive paperweight. Here's what actually shifted the math once I dug into it properly: OpenAI's small embedding model is cheap enough per token that it's hard to beat when your volume is spiky and unpredictable Self-hosting only starts paying off once that card is busy for most of the day, and not a moment before The real win for us was never open versus closed, it was getting more than one model onto the same GPU so it stops sitting idle, and Infinity does embeddings and reranking that way Superlinked's SIE pushed that further for us by running embeddings, reranking and extraction on a single card with hot-swapping, though it's pre-1.0 so caveat emptor (I follow the project) So the honest conclusion is that the whole thing hinges on your utilisation, which is a deeply boring answer, but it happens to be the true one. If your traffic is bursty like ours, the API is almost certainly cheaper than your gut is telling you it is.
document fetched 2026-10-22 00:32 · canonical https://reddit.com/r/selfhosted/comments/1xbe5ns/is_selfhosting_embeddings_actually_cheaper_or_are