Web Bot Auth: Verify AI Agents With Signed Requests

Quick Takeaways (read this first)

• “AI crawlability” is now an identity problem. Robots.txt and user-agent strings can’t reliably separate real agents from spoofed bots.
• Web Bot Auth + HTTP Message Signatures lets reputable agents prove who they are with cryptographic signatures you can verify at the edge.
• Don’t go “signatures-only” yet. Not all legitimate requests are signed, so you need a fallback (IP range + reverse DNS / rDNS verification) to avoid accidental blocking.
• Signed identity unlocks cleaner measurement. You can log “verified agent” vs “claimed agent” and stop mixing spoofed noise into AEO/GEO reporting.
• Start at your CDN/WAF. You’ll reduce origin changes, avoid misconfig, and enforce policy consistently across the site.

What changed: the gatekeeper isn’t robots.txt anymore—it’s verifiable identity

The most operationally important shift for AEO/GEO teams right now is quiet: AI agents are increasingly the “user” fetching your pages, and the real risk isn’t that you forgot to allow a bot in robots.txt. The risk is that your edge stack can’t tell the difference between:

• a legitimate AI agent (Gemini/Google-hosted fetchers, ChatGPT’s cloud browser, etc.),
• a scraper spoofing a user-agent string, and
• a botnet hammering your origin while pretending to be “helpful AI.”

That identity ambiguity creates a classic internal conflict: marketing wants AI agents to access content (visibility, citations, mentions), while security sees “unknown automation” and blocks it. The result is often invisible-to-you failures: your content is technically public, but the agents that matter can’t reliably fetch it.

Web Bot Auth is the first practical bridge between those teams: it gives you a way to verify certain agent requests cryptographically, then apply policy (allow, rate-limit, block, or serve a reduced response) based on verified identity instead of guesswork.

Web Bot Auth in plain English: signed requests you can verify

Web Bot Auth is an emerging approach (currently in IETF draft form) that uses RFC 9421 HTTP Message Signatures so an agent can sign parts of an HTTP request. Your CDN/WAF/origin then verifies the signature against a published key directory.

Google has published an experimental guide showing how certain Google-hosted agents can send signed requests using headers like Signature-Agent, Signature-Input, and Signature. You verify the signature using the agent’s published keys (for example, via an agent directory such as agent.bot.goog). See: Google’s Web Bot Auth (experimental) developer documentation.

Importantly, Google also notes that not all requests are signed yet, which means you still need IP/rDNS verification as a fallback to avoid blocking legitimate traffic.

This isn’t “just a Google thing.” OpenAI documentation indicates ChatGPT’s cloud browser can also sign outbound requests using Web Bot Auth, which means you can verify it in a way user-agent strings never enabled. Reference: OpenAI’s Cloud Browser allowlisting guidance.

If you want the standards context, the registry + “Signature Agent card” concept is described in the IETF draft: IETF Internet-Draft: Web Bot Auth registry.

Why this matters for AEO/GEO specifically (not just security)

AEO/GEO is increasingly judged by whether reputable systems can fetch and trust your content at the moment they need it. Signed identity turns that into something you can manage and measure:

• Safe allowlisting: you can allow verified agents without opening the door to everything that claims to be an agent.
• Cleaner measurement: you can separate “verified agent fetches” from spoofed traffic in logs and analytics pipelines.
• Policy-based access: you can allow read-only paths, tune rate limits, or block suspicious automation without breaking legitimate discovery.

The non-obvious insight: signed requests are an observability upgrade, not just an allowlist tool

Most teams hear “bot verification” and think: cool, we’ll block bad bots. The more valuable win for AEO/GEO teams is the ability to answer questions you couldn’t answer reliably before:

• Did a real agent fetch the URL that stopped getting cited?
• Did our WAF start challenging those requests last week?
• Are we measuring actual agent discovery—or just a pile of spoofed user-agents?

Search news coverage has started to frame Web Bot Auth as Google’s experimental method to validate authentic bots. If you’re tracking industry chatter, the Search Engine Land topic hub is a useful pointer: Search Engine Land’s coverage hub (user agents / Web Bot Auth).

How verification works (conceptual flow)

Here’s the practical model you should align on internally—especially between SEO/AEO, analytics, and security:

1. Agent sends request with identity headers (e.g., Signature-Agent) and a cryptographic signature over selected request components.
2. Edge verifies signature using a published key directory for that agent identity.
3. Edge applies policy based on verification result (verified vs unverified) and endpoint sensitivity (public content vs account endpoints).
4. Edge logs verification outcome so AEO/GEO reporting can use “verified agent fetches” as a signal.

Where to implement: CDN/WAF first (then origin)

We recommend starting at your CDN/WAF for three reasons:

• Consistency: one policy applies across all origins and paths.
• Lower risk: fewer app changes, fewer accidental cache or auth regressions.
• Better performance: verification and rate limiting happen before your origin is touched.

If you’re on Cloudflare, review their reference implementation notes: Cloudflare Developers: Web Bot Auth bot verification.

Step-by-step: build an “AI agent access policy” at the edge

The goal isn’t “let all AI in” or “block all bots.” The goal is tiered access: verified agents get reliable access to public content, while unknown automation gets constrained.

Step 1) Inventory what you actually want agents to access

Before you touch WAF rules, define your intended surface area:

• Allow: public HTML pages, key documentation, pricing, help center, category pages.
• Allow carefully: JSON endpoints that power pages (if agents need them), sitemap endpoints.
• Never allow (or heavily rate-limit): login, checkout, cart, account, password reset, internal search, and any endpoint that triggers side effects.

Tip: if you’re already doing “agentic” work (forms, flows, task completion), align this step with your agent-ready UX audits. Our related guide on making sites usable for agents can help you define what “safe” looks like: Agentic Engine Optimization: Make Sites Work for AI.

Step 2) Implement verification: accept signed requests, but keep fallbacks

Your verification logic should result in a small set of outcomes you can use everywhere:

• VERIFIED_SIGNED: signature present and valid; agent identity matches a trusted directory.
• VERIFIED_FALLBACK: signature absent, but request matches known IP range + rDNS pattern for that agent (temporary reality while signing coverage expands).
• UNVERIFIED: signature invalid, missing with no fallback match, or identity mismatch.

This “three-bucket” model is the difference between a clean rollout and a chaotic one. It prevents you from blocking legitimate but unsigned requests during the transition period.

Step 3) Apply a tiered policy (example rules you can copy)

Use verified identity to decide how you respond. A practical baseline policy looks like this:

1. Public content (GET/HEAD):
   • Allow VERIFIED_SIGNED and VERIFIED_FALLBACK.
   • Rate-limit UNVERIFIED automation (e.g., 60 requests/min per IP) and block obvious abuse patterns.
2. High-risk endpoints (POST, auth, account):
   • Block or challenge all automation by default (including verified agents) unless you intentionally support agentic actions.
   • If you do support agentic actions, require additional controls (OAuth scopes, user consent, replay protection, strict rate limits).
3. Expensive endpoints (search, large exports):
   • Allow verified agents but apply lower burst limits and caching where possible.

Non-obvious tip: if your WAF currently uses “bot score” heuristics, keep them—but only as a secondary signal. Signed verification should become your primary allowlisting mechanism for reputable agents.

Step 4) Add logging fields that make AEO/GEO measurable

This is where most implementations miss the AEO/GEO payoff. Add explicit log fields so you can build dashboards and alerts.

At minimum, log:

• signature_agent: value of Signature-Agent (or equivalent)
• signature_input_present: boolean
• signature_present: boolean
• signature_verification_result: pass/fail/error
• verified_agent_uri: canonical agent identifier (post-verification)
• verification_mode: signed vs fallback
• waf_action: allow/rate-limit/challenge/block

Then you can answer: “How many verified agent fetches did our top cited URLs receive this week?”—instead of guessing based on user-agent strings.

Three real-world examples (good vs bad) you can use immediately

Example 1: The “spoofed Gemini” incident that breaks AEO measurement

Bad pattern: Your dashboard counts anything with a User-Agent containing “Google” or “Gemini” as “AI traffic.” A scraper spoofs that string and hits 200k URLs/day. Your reports show “AI discovery is up,” but citations don’t move—and your origin costs spike.

Better pattern with Web Bot Auth: You log verified signatures separately. You discover that only 2–5% of “Gemini-labeled” requests were actually verified. Your AEO/GEO reporting becomes credible overnight because “verified agent fetches” becomes the metric.

Action: Add a dashboard split: Verified agent fetches vs Claimed agent fetches vs Unknown automation. If you’re already building revenue attribution for AI channels in GA4, this pairs well with your measurement model—see GA4’s AI Assistant Channel: Attribute AEO/GEO Revenue.

Example 2: The “security team blocks all bots” rollout that kills citations

Bad pattern: A new WAF managed rule challenges “automated traffic” with JavaScript or CAPTCHA. Humans pass; agents don’t. Your pages remain accessible in a browser, but AI agents fail silently.

Better pattern: You create a WAF exception: allow verified signed requests (and verified fallback requests) to GET/HEAD public content paths. You keep the challenge for unverified automation.

Action: Run an edge audit specifically for this failure mode. If this sounds familiar, the pattern is common enough that we wrote a dedicated playbook: Your robots.txt Allows AI—But Your CDN Blocks Crawlers.

Example 3: Debugging a visibility drop with “verified-agent fetch logs”

Scenario: Your brand stops appearing in AI answers for a cluster of queries. Content hasn’t changed. Rankings look stable. The team suspects “model updates.”

What verified logs reveal: Verified agent fetches for the exact cited URLs dropped from ~1,200/day to ~30/day the day your CDN enabled a stricter bot rule. That’s not a model issue—it’s access.

Action: Build an alert: if verified-agent fetches to your “citation-critical URLs” fall by >50% day-over-day, notify SEO + security in Slack.

Common mistakes (and how to avoid them)

Mistake 1: Treating signatures as mandatory today

Because not all legitimate traffic is signed yet, “block if not signed” will create false negatives. Use the signed + fallback model until signing coverage is truly broad.

Mistake 2: Allowlisting by user-agent string

User-agent allowlists are trivial to spoof. If you do keep them temporarily, only use them as a hint—not as proof.

Mistake 3: Verifying identity but not changing logging

If you don’t log verification outcomes, you’ve built a security feature but missed the AEO/GEO win: measurement you can trust.

Mistake 4: Forgetting caching and “agent-safe” responses

Verified agents often fetch at scale. Make sure your edge caching strategy is compatible with their access, and avoid personalized content on public URLs that would fragment cache keys.

Implementation checklist (one-week rollout plan)

Day 1–2: Align policy + paths

• List public paths to allow (docs, blog, product, category pages).
• List sensitive paths to block/challenge (auth, cart, account, internal search).
• Define the three outcomes: VERIFIED_SIGNED, VERIFIED_FALLBACK, UNVERIFIED.

Day 3–4: Configure verification at the edge

• Enable Web Bot Auth verification where supported (CDN/WAF).
• Implement fallback verification (IP ranges + rDNS) for known agents you care about.
• Dry-run in “log-only” mode if your platform supports it.

Day 5: Add observability

• Ship the new log fields (signature presence, verification result, verified agent URI).
• Create a baseline report: verified agent fetches per day, top URLs, top 4xx/5xx URLs.

Day 6–7: Enforce + validate

• Turn on allow rules for verified agents on GET/HEAD public content.
• Turn on rate limits for unverified automation.
• Validate with controlled tests: fetch a set of URLs and confirm expected WAF actions.

FAQ (optimized for quick answers)

Is Web Bot Auth replacing robots.txt?

No. Robots.txt is still a crawl directive. Web Bot Auth is identity verification. You typically use both: robots.txt to express intent, and Web Bot Auth to enforce safe access and measure real agent traffic.

Can I block all AI agents unless they’re signed?

Not safely today. Because signing is not universal yet, you should maintain a fallback verification method (like IP/rDNS) for legitimate but unsigned requests—at least during the transition period.

What’s the minimum logging I need for AEO/GEO measurement?

Log signature header presence, verification pass/fail, verified agent identifier, and WAF action. Without those, you can’t separate verified agent fetches from spoofed traffic.

Where should I implement this if I have multiple apps and origins?

At the CDN/WAF layer. It’s the only place you can enforce a single policy across everything and avoid inconsistent app-level implementations.

What to do next (action steps you can assign today)

1. Create an AI Agent Access Policy doc (1 page): allowed paths, blocked paths, rate limits, and escalation owner.
2. Enable Web Bot Auth verification at your edge (or plan the quickest path to do so) and keep fallback verification until signing coverage is reliable.
3. Add verification-aware logging and build a “Verified Agent Fetches” dashboard.
4. Set an alert for sudden drops in verified agent fetches to citation-critical URLs.
5. Use the data to resolve the marketing vs security conflict: verified allowlisting for discovery, strict controls for unknown automation.

Make this measurable (and easier) with aeotool.ai

If you want to operationalize this beyond one-off WAF rules, we recommend treating “verified agent access” as a first-class AEO/GEO metric—right next to citations and revenue attribution. We’ve found teams move faster when the edge policy, logs, and visibility reporting live in one workflow.

Try the AEO tool dashboard by signing up at https://aeotool.ai/register. And if you want a quick on-page check while you browse, install our Chrome extension: AEO Analyzer Chrome extension.