A large share of web traffic is automated, and only a small part of that is hostile in the denial-of-service sense. Confusing the categories leads to two expensive mistakes: treating a scraper like an attack, which blocks partners and search engines, and treating an attack like a scraper, which leaves the site down while someone tunes a bot policy.
How They Differ
- Intent: a flood wants your capacity; a scraper wants your content and would prefer you stay online to serve it.
- Volume: floods are loud by design. Good scrapers are deliberately quiet, often slower than a human browsing session.
- Shape: an attack concentrates on one or two paths. A scraper walks the site systematically, following links like a crawler.
- Tolerance: a scraper usually respects a 429 and backs off, because being blocked ends its job. A flood does not care.
The overlap
Aggressive scraping becomes a denial-of-service condition without anyone intending it — a poorly written crawler hitting an expensive endpoint in parallel can be indistinguishable from an attack, from the server's point of view.
Identifying Automation
Declared identity is the easy case: legitimate crawlers announce themselves and can be verified by checking that the address really belongs to the operator, not by trusting the user agent string. Undeclared automation is harder, and the useful signals are behavioural — request timing that is too regular, navigation that ignores the page structure a human would follow, and connection characteristics that do not match the browser being claimed.
Different Responses
- Verified search crawlers: exempt from challenges, rate limited generously. You want them.
- Known partners and monitoring: allow explicitly, so a protection change never silently breaks an integration.
- Unwanted but polite automation: rate limit and return 429. Most of it will comply.
- Abusive automation: challenge it. A JavaScript check ends the majority of scraping runs without affecting a single real visitor.
- Actual floods: filter before the application, because none of the above matters at volume.
Measure Separately
If bot traffic and attack traffic land in the same counter, neither can be understood. Keep them apart in your dashboards: verified crawlers, unverified automation, challenged requests, and filtered floods are four different populations with four different responses. That separation is usually what turns a confusing traffic graph into a decision.