Using OpenAI Realtime API with a Robot
retained text
I hooked up the OpenAI Realtime API (gpt-realtime-2.1 over WebRTC) to a small robot arm I built, so I can just talk to it. In the video it plays tic-tac-toe with me, puts a cube in a bowl, picks its favourite toy car and hands a cube back to me. How it works: * The voice model gets only a handful of tools: look, reach, grab, drop, gripper, park and stop. Reach and grab aren't hand-coded moves. They're policies I trained in simulation, so the model decides *what* to do and the trained skills decide *how*. * Fresh images from both cameras (top view and wrist) go into the conversation with every user turn, so it always sees the current board. It reads the tic-tac-toe grid itself from the images. * To pick a target, a vision model puts a point on the object in the top camera image, and the reach or grab policy goes to that point.
document fetched 2026-10-24 10:30 · canonical https://reddit.com/r/OpenAI/comments/1xddpuh/using_openai_realtime_api_with_a_robot