Searchestradocs

Methodology

Agent readiness methodology

Site Audit has a separate Agent readiness section. It reads three site files that AI agents and AI fetchers use and the forms on audited pages, and renders a sample of pages in a browser. It does not give a score and it is not part of the Site Audit health score. Only findings verified in a browser can create Action Center actions.

What is checked

  • /llms.txt: a Markdown summary of the site for language models.
  • The ARD manifest (Agentic Resource Discovery) at /.well-known/ard.json, the older /.well-known/ai-catalog.json and any address the site declares for it.
  • /robots.txt rules for the four clients that fetch a page when a user asks for it: ChatGPT-User, Claude-User, Perplexity-User and MistralAI-User.

The files are read from the root of the audited site (the host of the first audited page). Training and search crawlers such as GPTBot are covered in Bot Analytics, not here.

Status values

StatusMeaning
ValidThe file exists and meets the rules below.
Problem foundThe file exists but breaks a rule, or the site declares an address that cannot be loaded.
Not declaredAn optional file returns 4xx and nothing points to it. This is not an error.
Could not be measured5xx, 429, a network error, or an address that resolves to a private network.
Your policyA robots.txt restriction you marked as intentional. It is shown, not counted as a finding.

Suggestions (for example a missing summary line in llms.txt) are listed but do not change the status.

llms.txt

Same rules as Lighthouse 13.6: the file has a top level heading (# Name), at least one Markdown link ([name](url)) and at least 50 characters. A missing > summary line under the heading (llmstxt.org format) is a suggestion only.

Google says it does not use llms.txt in Search or its AI features. We do not claim any visibility effect from it.

ARD manifest

Discovery order: robots.txt Agentmap: lines, <link rel="ard"> on the home page, an HTTP Link header with rel="ard", then /.well-known/ard.json, followed by the same signals for the older ai-catalog name.

Schema errors, suggestions and the older format are reported separately:

  • Errors: invalid JSON, no entries array, and per entry a missing or malformed identifier (format urn:air:<publisher>:[namespace:]<name>), displayName or type, or not exactly one of url and data. Other root fields such as specVersion, host and @context, and namespaced extension fields in entries, are allowed by ARD v0.91 and are not reported.
  • Suggestions: no or more than five representativeQueries, an old or non-standard type, the retired collections field.
  • Older format: only ai-catalog.json was found.

An address the site declares but that cannot be loaded is a problem. A manifest that is neither declared nor present is not.

robots.txt for user-initiated fetchers

Three cases are kept apart:

  1. Your policy: you marked the restriction as intentional in Site Audit. No finding.
  2. Rule only: robots.txt restricts the client. A rule alone does not prove that requests are blocked; OpenAI says robots.txt may not apply to user-initiated fetches.
  3. Verified technical block: only with your server logs in Bot Analytics, when the client's requests actually receive 401 or 403. Site Audit never reports this case from robots.txt alone.

Forms

Four checks read the forms on every audited page in raw HTML. Their findings are review candidates: a script can add a label or hide a field. On pages in the browser sample the candidates are replaced by the browser result (see Browser check).

  • agent-form-unlabeled: fields with no accessible name (no label, aria-label, aria-labelledby, title or placeholder). Hidden fields are skipped.
  • agent-form-honeypot: a text field inside the form that is hidden (hidden, aria-hidden, an inline style or a trap class name) and not reachable with the keyboard (tabindex="-1" or autocomplete="off"). An agent may fill it as a real field, and bot protection may then drop the request. Loss is not certain.
  • agent-form-captcha: a verification challenge (reCAPTCHA, hCaptcha, Turnstile, Friendly Captcha). Information only; it is a security mechanism, not an error.
  • agent-webmcp-declarative: forms that declare a WebMCP tool with toolname, and whether tooldescription is filled.

Browser check

In every audit up to 10 pages are rendered in a real browser (headless Chrome) and Chrome's accessibility tree is read. The sample takes the home page first, then pages with raw HTML form candidates, then other pages with forms. Nothing is clicked or submitted. A page that cannot be rendered keeps its raw HTML candidates and is not counted as checked. Pages that resolve to the same address are checked once, and the same audit always gives the same sample. The Agent readiness card shows the coverage of each audit: pages audited, pages checked in a browser, pages measured and pages that could not be measured, with the reason (timed out, browser service unavailable, page did not render, request rejected).

  • agent-form-unlabeled and agent-form-honeypot on a checked page use the rendered page. A name added by a script closes the candidate; a field with a placeholder counts as named, as in the raw HTML check. A trap counts only if its form has a visible field and it is not a search box, so a collapsed site search or a closed login window is not reported.
  • agent-button-unnamed: visible buttons and links with no accessible name in Chrome's accessibility tree, usually icon only controls.
  • agent-consent-overlay: a cookie consent layer on first load (known consent platforms or a fixed layer that mentions cookies) and how much of the viewport it covers. Information only.
  • client-side-rendering-issues: the main content appears only after JavaScript. The page fetcher had to switch to a browser because the raw response was nearly empty, the raw HTML has less than 30 percent of the rendered text (rendered page at least 150 words), or the H1 exists only after rendering.

Test on 60 public sites (17 November 2026, checked by hand): 22 of 25 raw HTML candidates for unnamed fields were native selects or region pickers hidden behind a custom control, and 1 of 4 measured trap candidates was a search box. With the rules above, all 6 browser verified form findings were correct. Known gap: a form that is entirely hidden on first load, such as a multi step contact form, is not checked for traps.

Observed agent access

When a log source is connected, Bot Analytics shows the last 30 days of requests from the four user-initiated fetchers and the responses they received. This is observed data, not a rule result.

  • Rates count only requests verified against the operator's official address list, with the GET method and a known response status. Requests for the discovery files are counted separately.
  • A percentage is shown only when the customer confirms that the site's logs come from a single source and a client has at least 30 verified requests. 30 is a display threshold, not a statistical confidence claim. Otherwise only counts are shown.
  • Empty states are separate: no log source connected, partial observation coverage, no eligible requests observed, and not measurable (no known response status). None of them means "no traffic" or "all successful".
  • A 403 response is reported as "HTTP 403 observed". A cause such as a firewall is stated only with edge evidence.
  • Requests before the status fix of 17 November 2026 are excluded, because a missing status was then recorded as 200.

Evidence levels

Every agent readiness finding has an evidence level:

  • Review candidate: seen only in raw HTML.
  • Verified in a browser: seen in the rendered page or its accessibility tree.
  • Verified by an expert: reproduced by hand or in a task test.

The three site file rules read the file itself, so their result is the file as served. Raw HTML form findings are review candidates and create no actions. Only rows verified in a browser reach the Action Center, counted by verified pages; the cookie layer is information and never does. Agent readiness has its own place in the Action Center: up to three actions per audit run, separate from the other Site Audit actions, so technical findings on many pages do not push them out. The panel shows the evidence level of every row.

Network limits

Only the audited site and its registrable domain are fetched. Every request and every redirect step is checked after DNS resolution; private networks (RFC 1918), loopback, link-local, IPv6 ULA and cloud metadata addresses are refused. Addresses the site declares on another host are reported without being fetched. At most three redirects are followed.

Free report

The free AI visibility report includes only the three site file checks in its first version; form findings stay out of it for now: the free report reads the home page as raw HTML and has no browser step, and at least one is required for verified form findings. It states which addresses were checked and which declared addresses were not fetched. Bot logs, task results and scores are not part of it.

Limits

  • This is not a task test: it does not show whether an AI agent can complete a task on your site.
  • No score is given and the Site Audit health score is the same with or without these checks.

Rule version

Rule set version 1.2 (November 2026): site files agent-llms-txt, agent-ard-manifest, agent-robots-live-fetch (1.0), forms agent-form-unlabeled, agent-form-honeypot, agent-form-captcha, agent-webmcp-declarative (1.1), and the browser check with agent-button-unnamed, agent-consent-overlay, client-side-rendering-issues and browser verification of the two form rules (1.2). This page is updated together with the code when a rule changes.