Almost every site wants the same thing from bot management: let the search engines and the ad crawlers through untouched, and challenge or block the rest. Blocking Googlebot costs you rankings. Blocking AdsBot-Google breaks the landing-page checks behind your ad spend. So allow-lists exist, and they are usually built on the User-Agent header.
That is the problem. The User-Agent is not an identity. It is a free-text field the client fills in, and anyone can set it to anything.
The one-line version
curl -A "Googlebot/2.1" https://yoursite.example - if that request skips your protection, your allow-list is an open door with a sign on it.
Why this is worse than a normal bypass
A bot allow-list is not a small exemption. It is usually the most privileged path through the stack: allowed bots skip the browser challenge, skip rate limits, and often skip the WAF too, because operators do not want a crawler tripping a rule and disappearing from the index.
So an attacker who can put a string in a header inherits the highest-trust path you have. Under a flood, every request carrying the magic User-Agent walks past exactly the defences you deployed for that flood.
The fix: verify the claim against the network
A User-Agent is a claim. The source IP address is evidence. Search-engine and advertising crawlers run from their vendor's own networks, and those networks are published and stable. So the check is simple: only honour a good-bot User-Agent when the request actually comes from that vendor's network.
- Google, including Googlebot, AdsBot-Google, AdsBot-Google-Mobile and Mediapartners-Google: AS15169.
- Microsoft, including Bingbot: AS8075.
- Yandex: AS13238.
- Baidu: AS55967 and AS38365.
- Apple, including Applebot: AS714.
On our edge this is a single lookup. The ASN of the source address is already resolved for the request log, so verifying a bot costs nothing extra. A request whose User-Agent says Googlebot and whose ASN is Google is treated as Googlebot. A request whose User-Agent says Googlebot from anywhere else is not merely unverified, it is dropped, because a client lying about its identity is not a client you want to serve.
Three outcomes, not two
Verified good bot: allowed, always. Unrecognised client: handled normally, which may mean a challenge. Claims to be a good bot but comes from the wrong network: dropped. That third case is the one most allow-lists miss entirely.
Reverse DNS works too, with a caveat
The method Google itself documents is a reverse DNS lookup on the source IP, followed by a forward lookup on the resulting hostname to confirm it maps back. It is accurate and it is vendor-maintained. The caveat is cost: it puts a DNS round trip in the request path, which is precisely what you cannot afford when you are being flooded. An ASN check is a local lookup against a database you already hold, so it stays cheap under load. Reverse DNS is a fine choice for a low-traffic origin; it is a poor choice for an edge under attack.
Categories deserve different answers
Once verification is in place, bot policy stops being one switch and becomes a short list of decisions, each with a different business consequence.
- Search and advertising crawlers: always allow. Blocking these costs rankings or breaks ad delivery, so they should not be behind a toggle at all.
- AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and CCBot: a genuine choice. Some sites want the visibility, others do not want the content used. Either answer is defensible; it should be explicit.
- Commercial SEO and scraper crawlers such as AhrefsBot, SemrushBot and MJ12bot: not malicious, but they crawl hard and cost origin bandwidth. Many sites block them and lose nothing.
- Scanner and exploit tooling: drop without a body. It is not a visitor and it does not need an explanation.
Design rules worth keeping
Two rules have saved us repeatedly. First, a verified good bot must be exempt from your own broad rules, not just from the challenge. If a customer blocks a User-Agent substring that happens to appear inside a crawler string, the verified crawler should still get through. Losing your search traffic to a careless substring rule is a painful way to learn that.
Second, make it impossible to block the crawlers that carry revenue. On our panel there is deliberately no switch to block Google's search or ads crawlers. A setting that exists will eventually be toggled by someone who does not know what it does, usually at the worst moment.
How to check your own setup in a minute
- Send a request with a Googlebot User-Agent from an address that is not Google's, and see what happens. If it is served, your allow-list trusts the header.
- Confirm the opposite direction too: make sure real crawler traffic is genuinely reaching you, in your request log, not just theoretically allowed.
- Check what your allow-list actually exempts. If it skips the WAF as well as the challenge, a spoofed User-Agent is also an application-layer bypass.
Bot management is one of the few places where the right answer is not a smarter classifier but a stricter proof. Ask for evidence instead of a claim, and the whole category of User-Agent spoofing stops being a concern.