I'm using Claude to extract, segment and label every article in the Library of Congress's Newspaper Collection (350k / 25 million pages so far)
retained text
I built [snewpapers.com](http://snewpapers.com/) over about a year with Claude Code. You can find a little bit [more about my project here](https://snewpapers.com/snewpapers-difference). Using Claude or whatever AI tool you like you can connect to the MCP, and they'll become your research assistant. Have a chat and it'll analyze the corpus for you and do either aggregations or full-text extractions, returning the individual component so you don't have to sift through 5,000 pages of raw archive scans to find what you want. If you want to keep track of how much of the Library of Congress' newspapers we have extracted you can [track our extraction progress here](https://snewpapers.com/ca-progress) Claude has helped with nearly every aspect: * On the image processing front there is document layout analysis, custom OCR, bespoke vision models (it helped set up free human labeling tools as well), and heuristics. We came up with a system of golden fixtures and regression testing to slowly build a robust layout analysis and probabilistic human reading order pre-processing layer, with automation channels for green-light / red-light (human review) * We've tested dozens of LLMs to find the right logic-level vs. price point for various post-processing steps, building lots of prompts (and iterating many times on them / doing token optimizations) during our segmentation efforts (extracting the individual components) * We built the front-end together little by little. There's not much there though, the main focus is the MCP / agentic tools via API rather than traditional search * We've put together private APIs (for the UI), and public APIs (for and in addition to the MCP layer) for the agentic tools to use * We've optimized postgres for pipeline management and opensearch for the research agents * I've sent Claude off on thousands of exploratory data analysis tasks * Claude (and AI agents), have helped with all the research projects for [the blog](https://snewpapers.com/blog) which showcase the cool stuff you can do with the MCP The list goes on and on, as this has been more than 4,000 hours so far. On any given day, I have 4-6 Claudes going on stuff like this, making small improvements and helping manage the workload. Our MCP is live in [claude.ai](http://claude.ai/), just search for SNEWPapers. It is free to try, but you do need to sign up for the free trial. Here's an example of [what you can do with the MCP](https://snewpapers.com/blog/snewpapers-research-features)
document fetched 2026-11-18 00:44 · canonical https://reddit.com/r/ClaudeCode/comments/1xyo77b/im_using_claude_to_extract_segment_and_label