Embedding model can be used to search text inside image
retained text
Just something I've been testing over few week since Embedding Gemma 2 released which is about 330MB at Q8. Not sure if this would count as 'resource' flair...
Image shows cosine similarity changes as I refine the text query (aka text embedding vector) over time. Shows how well it work & quirks it has at same time.
I'll leave tl;dr version for now, 3:30 am here.
If anyone is interested I'll do few more tests and make example query image like above - still have 7 more sets totaling 22448 screenshots with mix of eng / kor / jpn already indexed, but I'm out of test ideas.
===
TAKE WITH GRAIN OF SALT because this was tested only on few sets that isn't really scientific.
TL;DR for EmbeddingGemma2 at least:
* For it's size & embedding dim (330MB Q8 / 768 embedding dim) this is very much usable
* This **understands** a word's meaning across language a bit but has much lower similarity (again, prob due to size)
|query|cos sim.|delta to next candidate|
|:-|:-|:-|
|image with sentence 삼각건이 머리에 얹어졌다|0.71156|\+0.00810 (38.jpg)|
|image with sentence triangular cloth hat was place on top of head|0.62414|\+0.00015 (38.jpg)|
* Requires careful wordings (e.g. blond vs blonde) - maybe SLM's limitation or embedding dim's limitation
* (at least eng & kor) Text ***seemingly*** ***DOMINATE*** cosine similarity more than visual traits
* sometimes adding visual trait HARM (refer chair above) the similarity
* not sure if it's due to text being more informationally dense (aka more 'surprising')
* or just model being small and not recognizing such visual traits, idk
* regardless, visual traits are less useful in this test set since like 90% of 3003 imgs would have blonde girl in it.
|query|cos sim.|delta to next candidate|
|:-|:-|:-|
|image with dialog '삼각건이 머리에 얹어졌다'|0.75960|\+0.01774|
|image with blonde girl|0.68208|\+0.00250|
|image with blonde girl making >< expression|0.70416|\+0.00340|
* (at least kor) Word's existence dominate over whole sentence
* e.g. for above image's example "삼각건"(cleaning hat thing) dominates more than other words around it
* fyi only 2 image out of 3003 contains world "삼각건", which you can see via delta gap drops
|query|cos|delta|
|:-|:-|:-|
|image with word 삼각건|0.69251|\+0.01211 / +0.00766 / +0.00008|
|image with word 삼각건 & 머리|0.72881|\+0.01331 / +0.02728 / +0.00203|
|image with sentence 삼각건이 머리에 얹어졌다|0.71156|\+0.00810 / +0.01660 / +0.00041|
===
Test setup:
* Visual novel screenshots of a single route
* 3003 screenshots from hanasaki workspring
* 3078 scs from irotoridori no hikari
* EmbeddingGemma2 via llama cpp
* model = KV = Q8\_0
* nb = b = ctx = 4096 (embedding model restriction, all need to be same)
* RX 7900GRE via vulkan (f u rocm, so annoying to build)
===
Sample output for anyone curious about processing time
Index mode:
$ py embed_img_▽.py "./하나사키워크스프링 히카리"
Batch: 8
Resize width: 1280
Resize threshold: 1536
[VectorDB] open w/ 0 rows
removing 0 missing records from DB
Checking new/modified images in dir
Got 3003 new/modified imgs
[EmbedServer] open for http://127.0.0.1:8081
100%|██████████| 376/376 [08:30<00:00, 1.36s/batch]
[EmbedServer] closed
[VectorDB] closed
Indexed 3003 new/modified files in 510.10 (5.887/s)
Query mode:
$ py embed_img_▽.py "./하나사키워크스프링 히카리"
-q "image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"
[VectorDB] open w/ 3003 rows
[EmbedServer] open for http://127.0.0.1:8081
[EmbedServer] closed
[VectorDB] closed
Query:
"image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"
Top-5:
0.76813 +0.01052 Artemis_2026-09-08-00-57-38.jpg
0.75761 +0.00773 Artemis_2026-09-08-00-57-26.jpg
0.74988 +0.00341 Artemis_2026-09-07-02-36-38.jpg
0.74647 +0.00067 Artemis_2026-09-07-02-36-52.jpg
0.74580 -------- Artemis_2026-09-08-01-58-14.jpg
'.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-38.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-26.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-38.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-52.jpg'
'.../하나사키워크스프링 히카리/Artemis_2026-09-08-01-58-14.jpg'
Data points: [moved to gist due to formatting bugs](https://gist.github.com/jupiterbjy/0adf1c59c6185e1ac966c49a0ce3e870)document fetched 2026-10-19 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1x9acj4/embedding_model_can_be_used_to_search_text_inside