← sources

How I crawl 1M+ jobs from 36k career pages and rank them against a resume

source: reddit (reddit:developersIndia)published 2026-11-14 17:00ingested 2026-11-15 02:34

retained text

I'm a platform engineer by day, and I built this job search tool on my own as a side project. This is the engineering side.

**Problem:** Job boards are full of stale and reposted listings. I kept applying to roles that had closed weeks earlier. So I read postings straight from company careers pages, notice when one disappears, and rank what's left against my resume.

**Crawl:**

* Mostly ATS APIs, with headless Playwright for pages that need JS. Boards are split into batches that run every couple of hours, in parallel shards, each behind its own VPN exit.
* Each shard writes plain JSONL files. A later step merges them, fetches descriptions for new jobs, and diffs the result against Postgres: new jobs get added, jobs missing from their board get closed.
* Boards that error out or list far fewer jobs than last time go through a retry ladder (other VPN exits) before I trust them.
* Every job has a unique fingerprint for dedup, and the same company + title + location collapses to one result at read time.
* After the diff, a small LLM extracts structured fields and each job gets an embedding.

**Ranking:**

* The score is deterministic: title, skills, years and seniority, weighted. It's cheap and explainable, and the same input gives the same score.
* Jobs and resumes are also embedded (768-dim, pgvector with an HNSW index). Similarity adds up to +10 points, but only reorders jobs that already passed the base score. It never decides what's shown.
* Each user's ranked list is built once and stored, then updated as new jobs arrive, so reads are an indexed lookup. Profile edits trigger a debounced, rate-limited rebuild.

**Keeping bots out:** an open job index invites scrapers, so I use daily row quotas, per-minute search limits, a per-search result cap and a Turnstile check. Anonymous users are counted by IP plus browser fingerprint, since IP alone wasn't enough.

**Hard parts:**

* Closing jobs safely. If a board suddenly drops a lot of its listings, it's usually the board that broke, not the jobs. A big drop is held for review instead of mass-closing everything. Before I added that, thousands of dead jobs stayed active because boards were held forever, so it needed a rule for what counts as a working listing.
* Reliable server-side LaTeX/Typst resume rendering.

**Lesson:** I changed the job fingerprint rule after I already had data, which meant merging several databases into one. I'd lock the identity rule down first.

**Still wrong:** it's in beta. Search has edge cases with multi-word queries, and some company names and logos are off. I'd like to hear how you'd have designed the crawl, closure detection or ranking.

Coming from platform work, building the whole thing end to end taught me a lot: running crawlers at scale, Postgres tuning, vector search and rate limiting.

document fetched 2026-11-15 00:23 · canonical https://reddit.com/r/developersIndia/comments/1xw7c7c/how_i_crawl_1m_jobs_from_36k_career_pages_and