← sources

I gave an LLM agent (Claude Code) spatial memory so it can run my home robot

source: reddit (reddit:robotics)published 2026-11-01 11:27ingested 2026-11-02 02:41

retained text

An LLM agent understands "find my scissors", but it doesn't know my home: what is where, what it saw last time, how to get back there. EvergreenSLAM is a 2D lidar SLAM I wrote to be that memory (I posted it [on r/ROS](https://www.reddit.com/r/ROS/comments/1x88796/using_slam_as_an_agents_spatial_memory_i_wrote_an) two weeks ago: https://github.com/ThomasZB/EvergreenSLAM). The agent sees it through a CLI called `egs`: each place is a folder anchored to the lidar map, like `places/living-room/sofa/` with a `notes.md` and maybe a photo, and the agent writes the notes itself. Nobody operates the SLAM: there is no mapping or localization mode to switch, and places stay valid when the map is re-optimized or the robot restarts.

Here Claude Code is the agent and an XLeRobot is the body. The video is a 25-minute take from an empty map plus a short one the next morning, cut to 3:18, speed and clock on every frame (also on YouTube: https://youtu.be/FXoj2yCrbuo).

Day 1:

* "Follow me" to the fridge, "your rest spot is beside it", then I nudge it into place by voice (we mixed up my left and its left once). The only place I teach it.
* "Explore on your own." No room list, no route. In about 6 minutes it saves six places with notes and photos. One line in the living room note: "a glass door to what looks like a balcony (阳台, inferred)". It never went there.
* Nav2 creeps when docking by the fridge, so it finishes by hand and writes itself a parking rule in `places/rest-spot/SKILL.md`.
* "Do you know where the scissors are?" Nothing in memory, so it checks four known places, writes each miss down, asks me where scissors usually go ("on a table"), then finds them in a corner exploration had missed. That's search, not memory.
* "Take a photo of the balcony." `egs find 阳台` finds nothing (never saved), so it greps its notes, hits that inferred line, drives over and sends the photo on Telegram about 100 s after I asked, then saves `places/balcony` with a note of what's inside.

The fair objection to day 1: it's one session, so all of that was still in the model's context. That's what day 2 is for.

* Robot powered off overnight. It booted where it had parked, and egs put it back on the old map at the rest spot with no relocalize command. One restart, robot not moved; not a long-run test.
* New Claude Code session, no chat history, a 4-line prompt: read the three skills, use only your memory in egs. It does load the project's CLAUDE.md, which lists "find scissors" as a goal, but nothing says where they are. That was only in egs.
* I typed "do you remember where the scissors are? take a photo for me". It found `places/side-table/scissors`, "seen 16 h ago", said it would go check rather than trust old memory, drove 12 s, pointed the head where memory put them (vision server was off), and sent the photo 44 s after it read the question. From my enter key it was \~90 s, because Claude Code held my message until its listen call ended.
* "Go back to the rest spot": it parked by yesterday's rule (fridge at 0.49 m, rule says 0.43-0.45, backed up).

Being honest:

* Day 1's scissors search was the worst of three that day. In the other two, exploration had passed that table and it took about two minutes, no hints. Here, about ten, and I answered four of its questions, one with a hint.
* On day 1 the GPU box (vision + speech) stopped responding mid-take and nobody noticed. The agent did ("My vision model is timing out right now, so I'll look at a full-resolution photo myself"). I blamed it for the slow search, but the failed calls returned in \~1 s each; the time went to driving, reading photos and four empty places.
* No teleoperation (gamepad as e-stop, unused). Subtitles translate my Chinese; labelled speed-ups hide the cloud LLM's pauses. No arms yet: this is about memory.

Stack: Orange Pi 5 Ultra, ROS 2 Humble + Nav2 (MPPI), EvergreenSLAM, and an optional LAN GPU box. The SLAM itself only needs a 2D lidar (mine cost 215 CNY, \~30 USD) and an RK3588 board, which is all the small mecanum car in the README had; it doesn't use the XLeRobot's arms or cameras. The robot side (RobbyBot) is private for now; it's vibe-coded and I can't vouch for it.

Day 2 is roughly the behaviour I want ("seen 16 h ago, I'll go check"). What I'm unsure about is weeks: notes say seen / told by me / inferred and when, but I don't know if that scales past a month. How do you do it?

document fetched 2026-11-02 00:33 · canonical https://reddit.com/r/robotics/comments/1xl0qe7/i_gave_an_llm_agent_claude_code_spatial_memory_so