← sources

My Reading Library: Evaluating LLMs on Android Tasks

source: reddit (reddit:LocalLLaMA)published 2026-09-26 11:36ingested 2026-09-27 02:40

retained text

Can LLM agents actually get through a day in the life of a normal user?

That question got me reading papers on Android agents and mobile benchmarks over the past few months.

A few patterns kept showing up:

- Most benchmarks run on emulators, making real-device metrics difficult to measure.
- Important deployment metrics like battery, thermals, and temperature are often missing.
- Everyday tasks are scattered across benchmarks, languages, and apps, rather than forming a consistent, globally relevant task set.
- This makes it harder to evaluate whether an agent can actually work reliably on a real phone, for real users.


For now, I’ve put together a library of papers on benchmarking mobile/Android agents for you all to read!

Link: https://www.alphaxiv.org/shared/folder/01a070c6-29a0-77a9-a5b4-b670d5eee169

document fetched 2026-09-27 00:39 · canonical https://reddit.com/r/LocalLLaMA/comments/1wqobag/my_reading_library_evaluating_llms_on_android