We're experimenting with using AI to track and evaluate candidates on how well they use Claude Code. What would separate good from excellent?
retained text
I'm working on a way to find developers who are very skilled at working with Claude Code. The idea is to record how they work and have an AI evaluate them as a first pass. To be clear, by "developer" I mean someone who writes software by directing Claude Code, not someone building the tool itself. **The setup**: a developer works on a task with Claude Code, and a few hooks record what they do. Each prompt they send, each commit (they're asked to commit far more often than usual), and subagent dispatches all get logged with timestamps. An AI on a seperate device then reviews The incoming telemetry along with the final result, and evaluates how well the developer worked. The first round would be unsupervised, so nothing stops someone from getting a skilled friend to sit with them. That's by design since monitoring every single applicant would be infeasible. Finalists would then do a shorter second task live, on screen, and be expected to perform at the same level. Disclosure: I'm building this as part of a tool called CELLO Protocol, which handles the logging side. This post isn't a pitch, and I'm not asking anyone to sign up for anything. I just want to know if the idea holds up, and gather thoughts on how best to evaluate the telemnetry. --- **Questions:** 1. If you were writing the instructions for the AI evaluator, what would you want it to evaluate to separate a mediocre developer from a good one and an excellent one? What should it look at first? How would it tell to evaluate whether someone considered architecture and security, and other solid coding praactices? 2. What kinds of challenges would show that difference? Some I'm considering: 1. Cleaning up an existing repo by removing dead and redundant code 2. Building a test harness for an exisitng code base the agent can check its own work against 3. Adding a tightly scoped new feature to an existing large codebase 4. Something else? 3. What would you want that this data wouldn't capture? what hooks would you use? 4. Anti-gaming: CELLO messages flow is real time, so with the hooks set up properly, the monitoring agent gets a steady stream of events as the candidate works: every prompt, commit, and subagent dispatch. Every commit is pushed, and the agent checks the repo right away to confirm it's there and matches what was reported. The agent can also nudge the candidate directly, for example if events stop coming in or a commit hasn't been pushed. The transcript is built as it happens and can't be edited afterward. So faking a log after the fact, or claiming work that isn't in the repo, would show up quickly. Given those guardrails: * What gaming would still be possible? * What patterns in an otherwise genuine session should the evaluator treat as suspicious? 5. How would you tell whether the person in the short live session is the same person who did the original unsupervised work? 6. Everyone who applies puts real time into this, so we want to give each candidate feedback, whether they're accepted or not. We'd likely wait until the challenge closes, so the feedback can't be passed on to people still working on it. What kind of feedback would be most useful to you as a candidate? Would you still want a different timing? No need to answer everything. Even one question you have strong opinions on helps. I'll share a summary of the responses once the discussion settles.
document fetched 2026-10-30 00:43 · canonical https://reddit.com/r/ClaudeCode/comments/1xiearl/were_experimenting_with_using_ai_to_track_and