Luna 6 is a massive downgrade over Luna 5.6. Misses crucial details, less helpful than 5.6 Luna
retained text
I maintain an agentic reporting tool which uses LLMs and tool calling to report on data from a wide variety of sources. Luna 5.6 was working really well for us at low cost, but I was considering upgrading to Luna 6 since it benchmarks slightly better and is 50% cheaper, hoping it would be at least equally as good. I did some A/B testing using "high" effort level for both: - On every single eval which involved reviewing written reports and finding relevant items, Luna 6 failed to identify one or more relevant items that should have been included in its answer. Luna 5.6 did not. - Luna 6 was consistently less helpful, failing to identify how to answer many questions that Luna 5.6 could answer, and providing less useful information in its responses - Luna 6 appears to try to answer the prompt in the least amount of tokens possible, which makes it less helpful. It will give a vague answer, requiring the user to ask a lot of follow up questions. For example. "Can you find if there are any items that fit this criteria?" GPT 5.6 Luna: "I found 7 items, here they are: <Formatted List> GPT 6 Luna: "Yes I found 7 items". (user has to follow up "Can you list them please?") If I could use a single word to summarize Luna 6, it's "Lazy". Even on high effort, it would always answer with the bare minimum. I tried many different use cases and could not find a single instance where 6 Luna provided a superior answer to 5.6 Luna. I won't be using it for my application, will stick with 5.6 Luna.
document fetched 2026-09-24 00:30 · canonical https://reddit.com/r/OpenAI/comments/1wnxg0n/luna_6_is_a_massive_downgrade_over_luna_56_misses