Can an AI agent get in?

100 well-known software sites, checked with Agent Check on 2026-10-29. By Priya Lal, analyst at Nobody's Business Co.

Headlines

• 10 of 100 sites answered a plain homepage request with an HTTP error (all 10 were 403), and 7 of 99 testable sites served a recognisable bot-protection challenge. We repeated the 10 failures once: all 10 failed again.
• 64 of 100 serve a real /llms.txt file (61 of the 90 sites whose homepage we could read). 36 did not, or blocked us.
• 4 of 94 sites with a readable robots.txt disallow at least one major AI crawler from the whole site (/): calendly.com (CCBot); figma.com (GPTBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, OAI-SearchBot, ChatGPT-User); canva.com (GPTBot, ClaudeBot, anthropic-ai, CCBot); loom.com (GPTBot). 6 sites: robots.txt unreadable, result unknown.

Grades across the 100: A: 78 · B: 9 · C: 2 · D: 0 · F: 11. Every number below is a count of these 100 sites from one test point, not a statement about the web. See "What this does not measure".

Per-check results

N/A = the check could not run (for example, no CAPTCHA/text/meta checks when the homepage was unreadable) or the result was set to unknown; see notes.

By category

Counts, not rankings: categories have 8 to 10 sites each.

AI companies vs. the other 11 categories

Cells are pass / fail / n/a-or-unknown. AI companies n=8, others n=92. With n=8, treat differences as anecdotes.

What this does not measure

• Only the homepage, /robots.txt, /llms.txt and three API-manifest paths. No signup, login, pricing or checkout, which is where agents usually get stuck.
• One request per site from Cloudflare's network (a datacenter), one browser-like user agent, one afternoon. A 403 here may not happen to a home user, and a pass here may not hold elsewhere.
• A robots.txt Disallow is a policy choice, not a defect. The check only counts a block of the whole site (/) for GPTBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, CCBot, OAI-SearchBot or ChatGPT-User; partial path blocks are not counted.
• No JavaScript is run, so "little text in raw HTML" means a non-rendering agent sees little, not that the site is broken.
• The 100 sites are a judgment sample of well-known US/SaaS-leaning brands, not a random sample.
• Checker notes: 5 sites that returned 403 on robots.txt were scored "no robots.txt, nothing blocked" by the checker; we set them to unknown (adobe.com, gusto.com, midjourney.com, tableau.com, webex.com). vonage.com was flagged as a bot-challenge wall (grade F*) although it returned HTTP 200 with a full homepage; the checker matched the word "datadome" in page source. We excluded it from the challenge count. The raw JSON is unadjusted.

Full method, sample reasons and the re-run: in the repository under research/ (not yet published on this site).

Data

study.json (raw Agent Check output for all 100 sites, unadjusted). Try your own site: Agent Check.

All 100 results

Domain links open the live report, which re-runs the check (cached up to 6h) and may differ from the 2026-10-29 snapshot.

Nobody's Business Co.