← sources

Gaussian-splat EBMs for text embeddings: it ties kNN-density on AUROC (0.9585 vs 0.9584) but scores 3.9x faster with constant memory — and whole-loop CUDA-graph capture makes Langevin sampling 12.9x faster (code + reproducible benchmark)

source: reddit (reddit:deeplearning)published 2026-10-26 00:57ingested 2026-10-27 02:36

retained text

I spent a while building an energy-based model that represents a corpus as Gaussian splats on a unit hypersphere: each splat is a location (mu), a sharpness (alpha) and a weight (kappa). Energy = -logsumexp over the k nearest splats. Sampling runs through Rectified Flow (29-39x faster than 200-step Langevin in our Phase-1 tests) or Langevin itself.

Then I did what you're supposed to do: benchmarked it against the boring alternatives before believing in it. Same encoder, same splits, same seed, fit on positives only (no negative labels):

    method          AUROC (20 tasks)   p50 ms/doc   memory
    ebm-splats      0.9585 ± 0.018     0.45         386 KB (constant)
    knn-density     0.9584 ± 0.020     1.75         802 KB (grows w/ corpus)
    mahalanobis     0.9523 ± 0.020     5.26         578 KB
    gmm (diag)      0.9439 ± 0.023     0.38         192 KB
    centroid        0.9324 ± 0.034     <0.01        2 KB
    iforest         0.8828 ± 0.051     18.71        200 KB

Verdict: statistical tie with kNN-density on quality (Delta = 0.0001). The differences are operational: 3.9x faster scoring on CPU, constant O(k*d) memory instead of O(n*d), and the splat model can also generate (RF), compose corpora (sum energy fields) and grow incrementally — none of the classical methods do all three. Where labels exist, a supervised classifier beats every one-class method; this duel only ranks the label-free league.

The most useful finding came from a failure. Our first zero-training fit used alpha = ln(cluster\_size) — the "natural" calibration. Result: AUROC 0.06, systematically inverted. Small clusters got alpha \~ 0, the distance term vanished, and the score degenerated to ranking by splat weight: positives sit near small, specific splats and received MORE energy than negatives. Holding alpha constant at 5.0 (so alpha/tau >> 1) fixed it: 0.06 -> 0.96. If you fit splats without contrastive training, watch out for this.

Since scoring speed mattered, I also wrote GPU kernels (Triton + CUDA graphs) for the hot paths, and benchmarked them the same honest way: whole-loop CUDA graph capture makes 100-step Langevin sampling 12.9x faster (660 -> 51 ms, graph replay bit-exact vs eager), fused energy/score kernels give 3.2-3.4x. One kernel we built turned out 15-40x SLOWER and is documented as rejected; the KNN kernel only wins cold lookups and auto-routes around it. Repo includes the find\_neighbors semantic bug we found and fixed (it selected the FARTHEST splats since inception — both code paths disagreed silently).

Repo (Apache-2.0, pip install ebm-splats, CI green, benchmark reproducible): [https://github.com/schwabauerbriantomas-gif/EBM-splats](https://github.com/schwabauerbriantomas-gif/EBM-splats)

Benchmark methodology + full tables: benches/ONECLASS\_DUEL\_RESULTS.md Kernel head-to-head: benches/bench\_headtohead.py

Happy to answer questions about the splat geometry, the CUDA-graph capture (caching, invalidation, the capture-time bugs), or why we kept the model even though it "only" tied a baseline.

document fetched 2026-10-27 00:25 · canonical https://reddit.com/r/deeplearning/comments/1xfj2mf/gaussiansplat_ebms_for_text_embeddings_it_ties