Design a Web Crawler
Design a Web Crawler — A Staff playbook focused on URL frontier governance, politeness as infrastructure constraint, dedup at scale, and crawl freshness vs coverage tradeoffs — not just Scrapy tutorials.
This playbook is part of the full calibration library.
Staff interviews are decided on nuance — tradeoff framing, ownership boundaries, and failure anticipation. This playbook covers the depth that separates Senior from Staff+.
What's inside this playbook
Core sections
- •1. The Staff Lens
- •2. Problem Framing & Intent
- •3. Fault Lines (Part 1)
- •4. Fault Lines (Part 2)
- •5. Evaluation Rubric
- •6. Interview Flow & Pivots
- •7. Active Drills
- •8. Deep Dive Scenarios
- •9. Level Expectations Summary
- •10. Staff Insiders: Controversial Opinions
Practice & Reference
- •How to Use This Playbook
- •What This Interview Actually Tests
- •The L5 vs L6 Contrast (Memorize This)
- •The Three Intents (Pick One and Commit)
- •The Fault Lines (Preview)
- •Quick Reference: What Interviewers Probe
- •Jump to Practice
- •System Architecture Overview
- •Interview Walkthrough: How to Present This in 45 Minutes