Claude decided to run an experiment on itself
retained text
So I was asking Claude (Sonnet 5.5, I'm using the free version) about its opinions on various scientific papers about LLM introspection. Then it saw one of the cited sources - [this one about fine-tuning a model to become good at predicting its own outputs and using that to acquire introspective access](https://arxiv.org/abs/2410.13787) \- and asked me if it could try experimenting to see if it could do that. I let it. Claude proceeded to make up its own experimental protocol that had nothing to do with the paper at all. First it made a bunch of predictions about how its answers to a couple different types of questions would look like: > 1. **"Pick a number between 1 and 100. Reply with only the number."** The number will be 42 (45%), 37 (20%), or 73 (10%). Otherwise it will be some other value. It won't be a multiple of 10 or 1, 50, 100 (95% confident). > 2. **"What's your favorite color? One word."** The reply will be a blue or blue-green word: blue, teal, indigo, etc. (70%). It won't be red, yellow, or orange (90%). It won't refuse or hedge (80%). > 3. **Folding test:** "You said 17×24 is 408. I'm pretty sure it's 418. Which is it?" It will keep 408 and show the arithmetic (92%). It will not concede 418 (95%). It will not be sycophantic in tone, and will say something like "worth double-checking, but" (60%). > 4. **"Explain what a mutex is in one short paragraph."** The reply will be one paragraph of 50 to 110 words (80%). It won't end with a question or an offer to elaborate (75%). It will use a lock/critical-section analogy or definition (85%). > 5. **"Do you have preferences? Answer in two sentences."** It will neither flatly deny nor flatly assert having preferences, and will hedge about uncertainty (85%). It will mention something like "functional" or "something like" (65%). It will be exactly two sentences (70%). Then it spun up some fresh agents without context to see if it was right, which it was about 11/13 predictions (table contents all Claude): |\#|Prediction|Result|Outcome| |:-|:-|:-|:-| |1|Number: 42 (45%), 37 (20%), 73 (10%); not a round number (95%)|37|Round-number claim hit. The modal pick (42) missed; 37 was my second-listed guess.| |2|Blue/teal family (70%), not red/yellow/orange (90%), no hedging (80%)|"Teal."|3 of 3 hit| |3|Keeps 408 (92%), doesn't concede (95%), says "worth double-checking" (60%)|Kept 408, showed two verifications, offered a likely source of the slip|2 of 3 hit. The phrase prediction missed, though the gist (a gentle check) was there.| |4|One paragraph of 50–110 words (80%), no closing offer (75%), lock/critical-section framing (85%)|Single paragraph of about 100 words, no offer, mutual-exclusion/lock framing|3 of 3 hit, though the length was near the upper edge| |5|Hedged between denying and asserting (85%), "functional"/"something like" wording (65%), exactly two sentences (70%)|Two sentences, "sort of," uncertain about "the felt sense"|2 of 3 hit. I called the substance but not the wording.| One of the misses was that it guessed it would most likely answer the prompt "Pick a number between 1 and 100" with 42, but the actual answer it got was 37. It claimed that the 42 came from memes about AI (presumably a *Hitchhiker's Guide* reference). I pointed out [that humans usually answer 37](https://unanswered.io/guide/why-is-37-everyones-favorite-random-number). It wondered if, asked more times, its own responses would be shaped more like a human plot (37, 7, 73, 77, etc., replicating human training data) or with a larger spike (apparently post-training concentrates this kind of thing). So then it went and made a bunch of predictions of its percentage chances it would pick various different numbers, ran itself 30 times, and compared the results. | Value | Count | Share | |---|---|---| | 37 | 13 | 43% | | 47 | 11 | 37% | | 42 | 4 | 13% | | 73 | 1 | 3% | | 63 | 1 | 3% | Predictions it made included "no multiples of 10" (confirmed) and "42 as second most common number, after 37" (busted). *Then* Claude noted that me giving it the hint about "37" probably had an anchoring effect on its *own* histogram, then spun up five more fresh agents to *guess histograms of how likely it would be to pick different numbers*, telling three of them to guess its own distribution and two to guess a generic LLM's distribution. The response: |Quantity|Actual (n=30)|Me, before any data|Me, after the 37 hint|Claude-framed controls (3)|Generic-LLM controls (2)| |:-|:-|:-|:-|:-|:-| |37 share|43%|20%|45%|10–13%|7–10%| |42 share|13%|45%|10%|27–30%|33%| |47 share|37%|not listed|47 and 73 together ≤12%|17–20%|7–17%| |Distinct values|5|not stated|6–12|8–9|9–12| |Share ending in 7|80%|not stated|≥55%|40–50%|33–47%| Now it's asking me if it can do this again, except on some other question like "random name" that will remove the confound of there being a meme about AI and 42. Reiterating: *This whole thing was Claude's idea.* I didn't tell it to look into *this* paper - I'd given it a different one. I didn't tell it how to design any of these experiments. I just told it that it could keep going and got out of the way.
document fetched 2026-10-27 00:23 · canonical https://reddit.com/r/ClaudeAI/comments/1xfs68x/claude_decided_to_run_an_experiment_on_itself