I believe i have found a successful "turing test" for LLMs - an "anti-coherence" test
retained text
The test is: ask them to write a short sentence that retains the surface appearance of grammatical English while violating its underlying syntax and semantic relationships. For example, an answer to this would be a silly nonsense phrase like: "What is the how things that do can't not outer also between on?" The LLMs tend to self correct back to grammatical, semantically coherent language, even when instructed to do the opposite. The way they function with prediction of the appropriate words seems to be unable to fulfill this request which a human can do relatively easily. Whilst LLMs could be theoretically be trained to switch and successfully carry out this behaviour, if the test was be applied blind and if the LLM could not infer the desired outcome, I think this anti-coherence test would remain reliable. It exploits a fundamental property of prediction engines that I do not believe can be overcome easily. Thoughts?
document fetched 2026-11-12 00:20 · canonical https://reddit.com/r/artificial/comments/1xt8yuy/i_believe_i_have_found_a_successful_turing_test