Quick Takeaways (read this first)

• PSI/Lighthouse now scores “Agentic Browsing”, which engineering teams treat like a real defect list—regardless of what Google Search says about rankings.
• Broken llms.txt can be worse than no llms.txt. If you publish it, publish it correctly—or remove it until you can.
• Think “agent navigation contract,” not “AI Overviews hack.” llms.txt is operational documentation for agents, QA, and support flows.
• Agent readiness is mostly deterministic engineering work: stable HTML, accessible navigation, predictable forms, and bot-friendly CDN/WAF behavior.
• Run a 7-day Agent Readiness Sprint to create a measurable backlog, ship quick wins, and set monitoring.

Why this matters now (even if rankings don’t change tomorrow)

Google is aggressively accelerating adoption of AI-powered search experiences. For example, Google has confirmed it’s testing homepage buttons that push users into AI search features (driving users to AI-powered search features). When user behavior shifts toward AI-assisted discovery and action, the sites that “work for agents” become the easiest for AI products to recommend, summarize, and transact with.

Now layer in the tooling reality: Lighthouse and PSI are widely used as a gating mechanism for releases. When PSI starts showing an “Agentic Browsing” category, teams will treat failures as:

• a QA problem (“our site isn’t machine-navigable”),
• a compliance risk (“are we blocking legitimate bots?”), or
• a conversion risk (“agents can’t complete checkout/support flows”).

This is why we recommend treating agent readiness as a technical checklist you can ship without rewriting your content strategy. You’re not betting on a ranking factor—you’re getting ahead of a measurable score that will influence internal prioritization.

The “two-speed” reality: Search visibility vs. agentic usability

Speed 1: Traditional search visibility (SEO/AEO basics still win)

For Google Search, you still need the fundamentals: crawlability, indexation, internal linking, helpful content, and structured data where appropriate. Google’s own Search Central updates are still where core guidance lands (see Google Search Central “What’s new” documentation updates for ongoing changes, including clarifications around AI-related reporting and structured data).

Net: your editorial and on-page strategy remains important.

Speed 2: Agentic usability (GEO + “agent commerce readiness”)

Agentic usability is about whether an automated system can reliably:

• render your key pages,
• find navigation, pricing, docs, policies, or inventory,
• complete forms/checkout flows,
• and do it without being blocked by bot protection.

That’s why this Lighthouse suite quietly changes the AEO/GEO technical checklist. It creates a shared language with engineering: “PSI says we fail agentic browsing.”

If you want a broader conceptual foundation, our guide on Agentic Engine Optimization: Make Sites Work for AI pairs well with this post—but here we’ll stay focused on what PSI is measuring and how to win quickly.

What Lighthouse’s “Agentic Browsing” audits are really doing

Think of this category as a set of heuristics that approximate: “Could an agent browse this site like a user would?” It’s not a promise that Google Search will rank you higher. It’s a quality signal surfaced in a place that influences roadmaps.

Important nuance: Lighthouse audits can be imperfect

There are already edge cases and false failures being discussed publicly. For example, a Lighthouse issue reports PSI/Chrome 146 failing llms.txt checks even for spec-compliant files (Agentic Browsing audit fails llms.txt for spec-compliant files). This matters because you can do “the right thing” and still get a red flag—so you need a validation workflow (we’ll cover it below).

Google’s messaging is fragmented (and that’s the opportunity)

One of the most useful summaries right now is that Google’s llms.txt guidance depends on which product you ask—Search vs. Lighthouse (SEJ’s breakdown of the split guidance). That split is exactly why teams get stuck debating “is llms.txt real?”

Our take: stop treating it as a ranking lever. Treat it as agent documentation and navigation—and make sure it doesn’t fail audits.

The contrarian rule: “No llms.txt” can be safer than a broken one

If your org is rushing to publish llms.txt because “AI is coming,” you’re likely to ship one of these failure modes:

• Empty file (or placeholder boilerplate)
• Malformed Markdown links that fail validation
• Wrong content-type or unexpected redirects
• Blocked by CDN/WAF for “unknown bot” requests

In Lighthouse/PSI, that can become a visible “defect.” Engineering teams often treat PSI defects as release blockers. So yes: publishing nothing can be lower-risk than publishing something broken.

Actionable decision rule

1. If you can’t commit to maintaining llms.txt quarterly, don’t publish it yet.
2. If you publish it, make it pass audits and be genuinely useful to agents.
3. If PSI flags it but you believe it’s correct, check for known Lighthouse quirks (see the GitHub issue above) before you churn engineering time.

A “good” llms.txt is not marketing copy—it’s an agent navigation contract

The best mental model we’ve found: llms.txt is a lightweight agent runbook. It’s where you tell automated systems (and your own team) “here are the canonical entry points and rules of the road.”

What to include (to satisfy audits and be useful)

• A clear title/H1 at the top (yes, even in a text file—make it obvious what it is).
• Non-boilerplate description of what the agent can do on the site.
• At least one valid Markdown link to key entry points (pricing, docs, support, policies).
• Optional: “safe paths” and “avoid paths” (e.g., staging, infinite faceted URLs, internal search results).

Example 1: SaaS company llms.txt (good vs. bad)

Bad (what triggers defects)

llms.txt
TODO: add later

Good (minimal but useful)

# ExampleCo Agent Guide

ExampleCo helps teams monitor API uptime and incident response.

## Key pages
- [Pricing](https://example.com/pricing)
- [Docs](https://example.com/docs)
- [Status](https://status.example.com)
- [Contact Sales](https://example.com/contact)

## Notes
- Prefer canonical URLs.
- Avoid crawling /search and /tags/* due to infinite combinations.

Tip: keep links absolute, not relative, to reduce agent confusion and validation quirks.

Example 2: E-commerce llms.txt that supports “agentic commerce”

# ShopCo Agent Guide

ShopCo sells refurbished laptops with a 12-month warranty.

## Start here
- [Current deals](https://shopco.com/deals)
- [Laptop category](https://shopco.com/laptops)
- [Shipping & returns](https://shopco.com/shipping-returns)
- [Warranty policy](https://shopco.com/warranty)
- [Order support](https://shopco.com/support/orders)

## Checkout notes
- Checkout requires a verified email.
- Coupon codes apply on the payment step.

That last “checkout notes” block is not SEO—it’s operational clarity. It reduces agent failure when an assistant tries to help a user purchase.

Example 3: B2B services site llms.txt for lead gen

# FirmCo Agent Guide

FirmCo provides SOC 2 compliance consulting for SaaS companies.

## High-intent pages
- [Services](https://firmco.com/services)
- [Case studies](https://firmco.com/case-studies)
- [Book a consult](https://firmco.com/book)

## Form behavior
- The booking form uses required fields: name, work email, company size.
- If form fails, use https://firmco.com/contact as fallback.

Why this matters: agents fail when forms are unpredictable. A fallback URL is a simple reliability hack.

The 7-Day “Agent Readiness Sprint” (a measurable AEO/GEO deliverable)

This sprint is designed to produce artifacts engineering respects: PSI screenshots, reproducible failures, and retest results. Marketing can co-own it without pretending to be a crawler engineer.

Day 1: Baseline PSI on money pages (and capture evidence)

1. Pick 10–20 pages that drive revenue or leads (pricing, product, checkout, top landing pages).
2. Run PageSpeed Insights on each.
3. Export or screenshot the Agentic Browsing section results.
4. Create a simple spreadsheet: URL → pass/fail items → owner → target fix date.

Tip: prioritize templates. Fixing one template can fix hundreds of URLs.

Day 2: Verify agent fetch behavior in logs (don’t guess)

Before changing anything, validate whether automated agents can actually fetch your site.

• Check CDN/WAF logs for requests to /llms.txt, key pages, and robots/sitemaps.
• Look for 403/429 spikes that indicate bot protection is tripping.
• Confirm content is served consistently (no geo-based blocks, no interstitials).

This step aligns with what OpenAI documents about web access: eligibility can depend on crawler behavior and infrastructure allowlisting. Their help center explicitly notes constraints and access behavior for ChatGPT’s web browsing (Searching the web with ChatGPT), which is a useful reminder that “robots.txt allows it” doesn’t mean your edge stack does.

If you suspect edge blocking, our post Your robots.txt Allows AI—But Your CDN Blocks Crawlers has a concrete allowlist audit workflow you can reuse.

Day 3: llms.txt—implement correctly, or remove it

Make a clean call:

• If you already have llms.txt, validate it (syntax, content-type, no weird redirects), then retest PSI.
• If you don’t have llms.txt, decide whether you can maintain it. If yes, ship a minimal “agent guide” (examples above). If no, don’t ship a placeholder.

Implementation checklist

• Serve at https://yourdomain.com/llms.txt
• HTTP 200 (avoid chained redirects)
• Content-Type: text/plain; charset=utf-8
• Contains meaningful content + at least one valid Markdown link
• Not blocked by WAF rules

Troubleshooting tip: If PSI flags your file but it looks correct, search the Lighthouse issue tracker—there are known cases where audits fail spec-compliant files (again, see this Lighthouse issue). In those cases, you may want to document the false failure and move on rather than endlessly tweaking a correct file.

Day 4: Accessibility tree sanity checks (agents rely on it)

Many agent systems effectively “read” your UI through semantic structure. If your navigation is div soup, an agent might not reliably find key actions.

Do these checks on 3–5 critical flows (pricing → signup, product → checkout, help → contact):

1. Open Chrome DevTools → Elements and Accessibility pane.
2. Verify your primary nav is a real <nav> with accessible links.
3. Ensure buttons are <button> (not clickable <div>).
4. Make sure form fields have labels (or aria-label).
5. Confirm focus order makes sense with keyboard-only navigation.

Fast win: Fixing missing labels and button semantics often takes minutes and improves both human accessibility and agent reliability.

Day 5: JS-rendering fallback for critical content (determinism over cleverness)

If key information only appears after client-side rendering (pricing tables, inventory, availability, even basic H1s), agents can fail depending on execution limits.

Pick one money page template and ensure:

• Core content (H1, price, primary CTA, key specs) is present in initial HTML.
• Lazy-loaded sections have accessible placeholders (not blank divs).
• Canonical URLs and meta robots are consistent server-side.

Good vs. bad example:

• Bad: Pricing page renders “Loading…” in HTML; prices appear only after an API call.
• Good: Server renders prices; JS enhances with toggles (monthly/annual) but doesn’t hide the data.

Day 6: WAF/CDN allowlisting for legitimate agent traffic

Even if you “allow” bots in robots.txt, your edge layer may block them. We see this constantly in audits: 403s from Cloudflare/Akamai/Fastly bot rules, rate limits, or geo restrictions.

Action steps:

1. Identify which agents matter to your business (Google, OpenAI, Perplexity, etc.).
2. Confirm their documented user agents and IP ranges where available.
3. Create a controlled allowlist or “challenge-free” rule for read-only paths (docs, pricing, policies, product pages).
4. Keep checkout/auth endpoints protected, but ensure browsing isn’t blocked.

Tip: Start with read-only pages. You can be agent-friendly without turning your site into an open door for abuse.

Day 7: Re-test PSI, document deltas, and set a monitoring cadence

Re-run PSI on your original URL list and record:

• Which Agentic Browsing checks flipped from fail → pass
• Which checks remain (and whether they’re real issues or tooling quirks)
• Which templates were fixed (for scale impact)

Then set a cadence:

• Monthly: PSI spot checks on top templates
• Quarterly: llms.txt review + link freshness
• Ongoing: WAF/CDN log alerts for 403/429 spikes to known bots

Common mistakes we see (and how to avoid them)

Mistake #1: Treating llms.txt like a meta keywords tag

If your llms.txt is stuffed with marketing phrases or vague claims, it doesn’t help an agent complete tasks. Keep it operational: entry points, constraints, and canonical resources.

Mistake #2: Shipping llms.txt once and never maintaining it

Agents will follow dead links. PSI may flag issues later. Tie llms.txt ownership to a team (growth engineering, devrel, or technical SEO) and review it quarterly.

Mistake #3: Ignoring “false fails” and burning sprint time

When PSI/Lighthouse flags something that appears correct, validate against known issues and reproduce locally. The Lighthouse issue tracker is your friend (start with this llms.txt audit thread).

Mistake #4: Focusing on bots while your UI is inaccessible

Agents depend on the same semantics that improve accessibility. If your site isn’t keyboard-friendly, it’s often not agent-friendly either.

Mistake #5: Letting security tools silently block “good” traffic

This is the most expensive failure mode because everything looks fine in SEO tools—yet agents can’t fetch pages. Use logs and controlled allowlisting.

FAQ: Agentic Browsing audits, llms.txt, and what to prioritize

Does llms.txt help Google rankings or AI Overviews?

Not in any direct, confirmed way. Google Search has indicated it’s not required for AI features, while Lighthouse/PSI may still score it. The practical play: treat llms.txt as agent documentation and keep it audit-safe. For more on the split messaging, see this SEJ analysis.

Should we publish llms.txt right now?

Only if you can publish a non-empty, valid, maintained file. Otherwise, it can create PSI failures that engineering will feel obligated to “fix,” costing you time without upside.

What pages should we test first in PSI?

Start with money pages and templates: pricing, product, category, signup/lead forms, checkout help, and your top organic landing pages.

How do we prove this work matters to leadership?

Use PSI evidence + conversion-path reliability. A simple KPI is “% of top templates passing Agentic Browsing checks” plus edge metrics like reduced 403/429 blocks to known bots. If you’re also tracking AI citations, pair this sprint with our measurement loop in GSC’s Generative AI Report: 30‑Min Citation Audit Loop.

What if ChatGPT/other agents can’t access our pages?

Check your CDN/WAF first. OpenAI notes that web access depends on eligibility and infrastructure behavior (OpenAI’s browsing documentation). In practice, your bot rules may be blocking requests even when robots.txt allows them.

What to Do Next (copy/paste action plan)

1. Today: Run PSI on 10–20 money pages and export the Agentic Browsing results.
2. Tomorrow: Pull CDN/WAF logs for 403/429s to key paths (/, /pricing, /docs, /llms.txt).
3. This week: Decide on llms.txt (implement correctly or remove placeholder), then re-test PSI.
4. This week: Fix 3 accessibility semantics issues (nav, buttons, labels) on your top template.
5. This week: Ensure server-rendered HTML contains critical content (H1, price, CTA).
6. Ongoing: Add a monthly “agent readiness” check to your technical SEO QA.