Atlassian System Design Interview: Web Scraping Service for Image URLs

The interviewer opens

Hi, and thanks for joining. Welcome to your interview for the software engineer role at Atlassian. We'll design a system together, with you sketching on the whiteboard on the right of your screen as you talk, and you can pause at any time with the Pause button at the top right if you need a break.

The problem is a web scraping service that collects image URLs, and it's in the Problem tab. Please read through it and ask me anything you'd like clarified before you start drawing.

The problem

Design a web scraping service: a client submits a list of URLs and gets back a job ID. The service crawls those pages and the pages they link to, collects image URLs, and lets the client check job progress and fetch results per URL.

Must do

• Submit a list of seed URLs and immediately receive a job ID
• Crawl each seed page and the pages it links to, collecting image URLs
• Report job progress while the job runs and when it finishes
• Fetch results per crawled URL: image URLs found, or why it failed
• Never crawl the same page twice within a job

Must hold

• A slow or broken site must not stall or crash the rest of a job
• Jobs must eventually finish and report completion reliably