# I handed Claude Code a database of 11 million news articles, it found some weird patterns — retained record

> Retained reddit record from reddit:ClaudeCode.

Canonical: https://highsignal.app/data/records/851d9bda2f8c1628

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:ClaudeCode)  published 2026-10-20 17:12 ingested 2026-10-21 02:40

[original source ↗](https://www.reddit.com/r/ClaudeCode/comments/1xaxzqv/i_handed_claude_code_a_database_of_11_million/)

## retained text



```
Side project, built nights and weekends (just Claude and me): a pipeline that reads \~100k news articles a day from 40k+ sources in 40 languages and clusters coverage of the same event into one item. After 164 days it had 11,079,667 articles, so I gave Claude Code a database export and a list of questions and let it work.

**What it found**:

* **12:1 ratio** of articles written per thing that actually happened
* **4 in 10** events never get a second day of coverage
* One world leader shows up in **\~84 new events every single day** on average, more than the next five most-covered people combined
* Only **0.08% of events score as "exceptional"** (as per an LLM judge), the mundane middle is where most of the news lives
* Biggest single event: 1,540 different outlets on one story (try to guess which one, you're probably wrong)

**How Claude did it**: First it listed a series of hypotheses / guesses of potential patterns based on previous discussions we'd had about the project over the months. Then it started writing a **long** list of queries against the export, sanity-checked its own numbers against the live API, then built the charts and the page that presents them.

My part was reviewing every number, throwing out two findings that didn't survive a second look + a few other ones that were too project-specific (rather than more general observations about the news), and redrafting the copy.

The pipeline itself is my code, built with Claude and other tools over months; the [**/world**](https://clstr.news/world?utm_source=reddit&utm_medium=post&utm_campaign=claudecode-oct) and the [**/observatory**](https://clstr.news/observatory?utm_source=reddit&utm_medium=post&utm_campaign=claudecode-oct) pages are mostly Claude's.

**Where it struggled**: While Claude did find a bunch of actual patterns in the data, a good chunk of them were just... not interesting. Or at least not interesting for anyone but me - they were too specific to the project and how it works (e.g. figuring out how long it took for an average cluster to become part of an ongoing situation), but not so much for the general public that's more interested in the news itself.

All the charts and the method: [**clstr.news/observatory**](https://clstr.news/observatory?utm_source=reddit&utm_medium=post&utm_campaign=claudecode-oct)

There's also a live map it helped build, which countries the world's news is about right now: [**clstr.news/world**](https://clstr.news/world?utm_source=reddit&utm_medium=post&utm_campaign=claudecode-oct)

For anyone building, or if you just want your agents to be up to date, free API and an MCP server with all the data produced by the pipeline: [clstr.news/developers](https://clstr.news/developers?utm_source=reddit&utm_medium=post&utm_campaign=claudecode-oct) (or just add [**https://mcp.clstr.news/**](https://mcp.clstr.news/) in Claude)



# Important edit

Just want to clarify some more details that the post didn't make explicit (more details in the Methodology section).

Claude **did not read the text for 11 million articles**. There is a pipeline which handles the clustering process of comparing source articles and joining together the ones related to a single event, which is comprised of several steps including text embedding comparisons, some additional heuristics and a final LLM judge pass to determine whether N articles are indeed referring to the same event or not.

What Claude did was process all the aggregated data derived from that pipeline - biggest clusters, clusters by country, clusters by time of the day, clusters mentioning a certain country or person, etc.
```



document fetched 2026-10-21 00:39 · canonical https://reddit.com/r/ClaudeCode/comments/1xaxzqv/i_handed_claude_code_a_database_of_11_million
