Crawl pages for images: A Sample System Design Solution

The system design question

Design a web scraping service. A client submits a list of URLs and gets back a job ID. The service crawls those pages and the pages they link to, collects image URLs, and lets the client check job progress and fetch results per URL.

Agree on the scope (5 minutes)

Say up front what you're covering.

We'll design	We'll skip
Submitting a job and getting a job ID	Downloading the images themselves
Crawling the given pages and the pages they link to	Running JavaScript to render pages
Collecting image URLs per page	Billing and accounts
Checking progress and fetching results	Searching across old results

Then agree on what "good" means:

Quick to submit: the client gets a job ID back right away, even for thousands of URLs.
Finishes: every page ends up done or failed with a reason. A job never hangs forever.
No repeat work: a page reached from ten different links in one job gets crawled once.
Polite: we never hammer one website, even if a job points 5,000 URLs at it.

Rough numbers (5 minutes)

Ask how deep to follow links. Here we'll assume one level: the pages you gave us, plus the pages they link to. The rest are assumptions too: 10,000 jobs a day, 100 URLs per job on average, about 30 new links per page after removing repeats, 100 KB per page, and 20 images per page.

~3,000
pages crawled per job: ~350
pages fetched per second, on average: ~3,000
fetches in flight at once at peak: ~60 GB
of image URLs saved per day

Question	Rough math	What it means
How many pages a day?	10,000 jobs × (100 + 100 × 30) pages	About 30 million pages a day, or 350 a second, maybe 1,500 at peaks
How many fetches at once?	1,500 a second × about 2 seconds per fetch	Around 3,000 open requests, spread over a few dozen workers
How much do we keep?	30 million pages × 20 images × 100 bytes	About 60 GB a day. We throw the page HTML away once it's parsed

The overall design (10 minutes)

A job turns into one task per page. Workers fetch pages, save image URLs, and add new links as more tasks until the job runs out of pages.

The client sends a list of URLs. The job service saves a job, saves one row per URL with status 'pending', puts one task per URL on a queue, and returns the job ID.
A crawl worker takes a task, checks it's allowed to hit that website right now, and fetches the page with a short timeout.
It pulls out the image URLs and the links. It saves the image URLs for that page and marks the page done.
If the page is still within the depth limit, it adds each link it hasn't seen in this job as a new task.
The client asks for progress, which is a count of pages by status, and pages through results per URL.

What the apps need to do

Action	Who does it	What's sent
Submit a job	Client	A list of URLs, and optionally how deep to follow links. Returns a job ID
Get progress	Client	Job ID. Returns pages pending, done and failed, and whether the job's finished
Get results	Client	Job ID, plus a URL or a page cursor. Returns each page's image URLs or its error
Cancel a job	Client	Job ID. Pending pages are skipped

What we store and where

Data	Where it goes	Why
Jobs	Relational database	Who owns it, settings, status and counters per status
Pages per job	Wide-column store, keyed by job ID then URL	Status, depth, error, and the image URLs. Also answers 'have we seen this URL in this job?'
Fetch tasks	A message queue	Hands work out to workers and retries tasks whose worker died
Per-website limits	In-memory store	When each website was last hit, and how many requests are open to it

Step 1: Tracking a job until it's done (10 minutes)

The tricky bit is knowing when a job is finished. You don't know up front how many pages it'll have, because new links show up as you crawl. So track it per page. Every page row is pending, done or failed. The job is finished when nothing's pending and no task for it is still in the queue.

Keep counters on the job row: pending, done, failed. When a worker adds 30 new pages, it bumps pending by 30. When it finishes a page, it moves one from pending to done. Progress is then one cheap read, not a count over thousands of rows.

Counters drift if a worker crashes halfway through. So treat them as a fast estimate, and have a small job that recounts from the page rows when a job looks finished, or every few minutes for big jobs. The page rows are the truth.

A page can't stay pending forever. Give each task a lease. If the worker doesn't finish within a couple of minutes, the queue hands it to another worker. After three tries, mark it failed with the last error and move on.

Step 2: Not crawling the same page twice (10 minutes)

Pages link to each other all the time, so the same URL shows up from lots of places. Before adding a link as a task, clean it up: lowercase the host, drop the '#section' part, drop tracking bits like 'utm_source', and sort the query string. Otherwise 'example.com/a' and 'example.com/a#top' look like two pages.

Then claim it. Write the page row with 'only insert if this job and URL don't exist yet'. If the insert works, add the task. If it fails, someone already has it. That one check keeps two workers from both deciding a link is new. Because the row is keyed by job ID then URL, all of one job's rows sit together, which is good sharding for this.

Workers can still process the same task twice, like when a lease runs out just as the first worker finishes. Make the work idempotent: saving image URLs for a page overwrites, and moving a page to done only works if it's still pending. Doing a page twice then costs a fetch but never double counts.

Seen across jobs?

Ask whether two jobs hitting the same page should share the fetch. It's a nice saving, but results get stale and you'd need to decide how old is too old. Start with 'seen' per job, and add a short-lived cache of recent fetches by URL if lots of jobs overlap.

Step 3: Slow, broken and busy websites (10 minutes)

Most of the pain is other people's sites. Set a connect timeout of a few seconds and a total timeout of maybe 15. Cap the page size at a few megabytes and stop reading past it. Only parse HTML, and skip anything else by its content type.

Don't hit one site too hard. Keep a rate limit per website, like one request a second and two at once, in a shared in-memory store. If a worker picks up a task for a site that's at its limit, put the task back with a short delay rather than waiting around. Read the site's robots.txt once and cache it, and skip pages it says not to crawl.

Sort errors into kinds. A timeout or a 503 is worth retrying later, with a growing delay. A 404 or 'not HTML' isn't, so fail it right away. A site that keeps timing out gets marked as down for a few minutes, so its other pages fail fast instead of each one waiting 15 seconds.

Some sites redirect in loops or link to endless pages, like calendars with a 'next month' link. The depth limit stops most of that. Also cap pages per job and per website within a job, and stop following redirects after five.

Choices you should be able to defend

We chose	Instead of	Because	We'd reconsider if
One queue task per page	One worker crawls a whole job	Big jobs spread across many workers, and one bad site only stalls its own pages	Jobs were tiny, so the queue's overhead wasn't worth it
'Seen' tracked per job	One global list of crawled pages	Every job gets fresh results, and jobs don't affect each other	Lots of jobs crawled the same pages and fetch costs mattered
Counters for progress, recount to check	Counting page rows every time	Progress checks stay cheap on huge jobs	Progress had to be exactly right at every moment
Put tasks back when a site is busy	Workers wait for the site	Workers keep busy with other sites	Almost every job pointed at the same one site

What could go wrong (5 minutes)

What happens	What we do about it
A worker dies mid-page	Its lease runs out and the queue gives the task to another worker
A website is down	Retry a few times with growing gaps, then mark its pages failed with the reason
A job links into a huge site	The depth limit and the per-job page cap stop it from crawling forever
The results store is slow	Workers slow down and tasks wait in the queue. Nothing's lost, jobs just take longer
A client submits 100,000 URLs to one site	The per-site rate limit holds, and their job runs slowly instead of hurting that site

Follow-up questions (5 minutes)

How would you tell the client when a job finishes?

Let them pass a callback URL when they submit. When the recount says nothing's pending, send one request to it, with retries.

What about pages that build their content with JavaScript?

Send those to a separate pool of workers running headless browsers. They're much slower and costlier, so only use them when the client asks.

How do you stop one big customer hogging everything?

Give each customer a cap on pages in flight, and pull tasks fairly across customers instead of strictly first come, first served.

How the 60 minutes add up

Minutes	What you're doing
0–5	Agree on what you're building
5–10	Rough numbers
10–20	Draw the overall design
20–30	Tracking a job until it's done
30–40	Not crawling the same page twice
40–50	Slow, broken and busy websites
50–55	What could go wrong
55–60	Follow-up questions

System design: crawl pages for images