← sources

Embedding model can be used to search text inside image

source: reddit (reddit:LocalLLaMA)published 2026-10-18 18:22ingested 2026-10-19 02:43

retained text

Just something I've been testing over few week since Embedding Gemma 2 released which is about 330MB at Q8. Not sure if this would count as 'resource' flair...

Image shows cosine similarity changes as I refine the text query (aka text embedding vector) over time. Shows how well it work & quirks it has at same time.

I'll leave tl;dr version for now, 3:30 am here.

If anyone is interested I'll do few more tests and make example query image like above - still have 7 more sets totaling 22448 screenshots with mix of eng / kor / jpn already indexed, but I'm out of test ideas.

===

TAKE WITH GRAIN OF SALT because this was tested only on few sets that isn't really scientific.

TL;DR for EmbeddingGemma2 at least:

* For it's size & embedding dim (330MB Q8 / 768 embedding dim) this is very much usable



* This **understands** a word's meaning across language a bit but has much lower similarity (again, prob due to size)

|query|cos sim.|delta to next candidate|
|:-|:-|:-|
|image with sentence 삼각건이 머리에 얹어졌다|0.71156|\+0.00810 (38.jpg)|
|image with sentence triangular cloth hat was place on top of head|0.62414|\+0.00015 (38.jpg)|

  


* Requires careful wordings (e.g. blond vs blonde) - maybe SLM's limitation or embedding dim's limitation
* (at least eng & kor) Text ***seemingly*** ***DOMINATE*** cosine similarity more than visual traits
   * sometimes adding visual trait HARM (refer chair above) the similarity
   * not sure if it's due to text being more informationally dense (aka more 'surprising')
   * or just model being small and not recognizing such visual traits, idk
   * regardless, visual traits are less useful in this test set since like 90% of 3003 imgs would have blonde girl in it.

|query|cos sim.|delta to next candidate|
|:-|:-|:-|
|image with dialog '삼각건이 머리에 얹어졌다'|0.75960|\+0.01774|
|image with blonde girl|0.68208|\+0.00250|
|image with blonde girl making >< expression|0.70416|\+0.00340|

* (at least kor)  Word's existence dominate over whole sentence
   * e.g. for above image's example "삼각건"(cleaning hat thing) dominates more than other words around it
   * fyi only 2 image out of 3003 contains world "삼각건", which you can see via delta gap drops

|query|cos|delta|
|:-|:-|:-|
|image with word 삼각건|0.69251|\+0.01211 / +0.00766 / +0.00008|
|image with word 삼각건 & 머리|0.72881|\+0.01331 / +0.02728 / +0.00203|
|image with sentence 삼각건이 머리에 얹어졌다|0.71156|\+0.00810 / +0.01660 / +0.00041|

===

Test setup:

* Visual novel screenshots of a single route
   * 3003 screenshots from hanasaki workspring
   * 3078 scs from irotoridori no hikari
* EmbeddingGemma2 via llama cpp
   * model = KV = Q8\_0
   * nb = b = ctx = 4096 (embedding model restriction, all need to be same)
   * RX 7900GRE via vulkan (f u rocm, so annoying to build)

===

Sample output for anyone curious about processing time

Index mode:

    $ py embed_img_▽.py "./하나사키워크스프링 히카리"
    Batch: 8
    Resize width: 1280
    Resize threshold: 1536
    
    [VectorDB] open w/ 0 rows
    removing 0 missing records from DB
    Checking new/modified images in dir
    Got 3003 new/modified imgs
    [EmbedServer] open for http://127.0.0.1:8081
    100%|██████████| 376/376 [08:30<00:00,  1.36s/batch]
    [EmbedServer] closed
    [VectorDB] closed
    
    Indexed 3003 new/modified files in 510.10 (5.887/s)

Query  mode:

    $ py embed_img_▽.py "./하나사키워크스프링 히카리"
    -q "image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"
    
    [VectorDB] open w/ 3003 rows
    [EmbedServer] open for http://127.0.0.1:8081
    [EmbedServer] closed
    [VectorDB] closed
    
    Query:
    "image with dialog '삼각건이 머리에 얹어졌다' with blonde girl making >< face"
    
    Top-5:
      0.76813  +0.01052  Artemis_2026-09-08-00-57-38.jpg
      0.75761  +0.00773  Artemis_2026-09-08-00-57-26.jpg
      0.74988  +0.00341  Artemis_2026-09-07-02-36-38.jpg
      0.74647  +0.00067  Artemis_2026-09-07-02-36-52.jpg
      0.74580  --------  Artemis_2026-09-08-01-58-14.jpg
    
    '.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-38.jpg'
    '.../하나사키워크스프링 히카리/Artemis_2026-09-08-00-57-26.jpg'
    '.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-38.jpg'
    '.../하나사키워크스프링 히카리/Artemis_2026-09-07-02-36-52.jpg'
    '.../하나사키워크스프링 히카리/Artemis_2026-09-08-01-58-14.jpg'

Data points: [moved to gist due to formatting bugs](https://gist.github.com/jupiterbjy/0adf1c59c6185e1ac966c49a0ce3e870)

document fetched 2026-10-19 00:37 · canonical https://reddit.com/r/LocalLLaMA/comments/1x9acj4/embedding_model_can_be_used_to_search_text_inside