For twenty-five years, websites were built for two audiences: people and search engine crawlers. In 2026 there is a third. AI agents now open pages, read them, click buttons and fill in forms on behalf of real customers. OpenAI's ChatGPT agent can use its own visual browser, and Google's Gemini in Chrome can "auto browse" through multi-step tasks for subscribers in the US.
If an agent cannot find your prices, gets stuck on a cookie banner or cannot tell which button submits your booking form, it simply moves on to a competitor whose site works. The customer may never see your page at all.
This guide explains what AI agents are, how they read websites, and a practical checklist to make your site agent-ready without hurting the experience for human visitors. For the wider context, read our explainer on what agentic search is.
Key Takeaways
- AI agents browse like people but read like machines: they rely on HTML structure, labels and accessibility information.
- Automated AI crawlers follow robots.txt, but user-triggered agents often do not, so access control needs more than robots rules.
- Accessible, semantic HTML and clear ARIA labelling help agents understand buttons, menus and forms.
- Key facts such as prices, availability, policies and contact details should be in plain HTML text, not images or scripts.
- Simple, forgiving forms and checkouts are the biggest conversion win for both agents and humans.
How AI Agents Read and Use Your Site
There are two kinds of AI traffic to plan for. Crawlers fetch pages automatically to build indexes, such as Googlebot or OpenAI's OAI-SearchBot. Agents visit because a specific user asked them to do something right now. Our guide to AI crawlers covers the first group in detail.
Agents typically combine a text view of the page with a visual or accessibility-tree view. OpenAI describes ChatGPT agent as having "a visual browser that interacts with the web through a graphical-user interface" alongside a text-based browser. OpenAI's publisher guidance is specific about what helps its Atlas browser agent:
“ChatGPT Atlas uses ARIA tags”
— OpenAI Help Center, Publishers and developers FAQ
The same page says ARIA labels help it "interpret page structure and interactive elements", and recommends following WAI-ARIA best practices for buttons, menus and forms.
Who is visiting: the main AI user agents
| User agent | Operator | Purpose | Follows robots.txt? |
|---|---|---|---|
| Googlebot | Crawling for Search, including AI Overviews and AI Mode | Yes | |
| Google-Agent | Agents on Google infrastructure acting on user requests | Generally no (user-triggered) | |
| OAI-SearchBot | OpenAI | Surfacing sites in ChatGPT search | Yes |
| ChatGPT-User | OpenAI | User-initiated actions in ChatGPT | May not apply |
| GPTBot | OpenAI | Model training | Yes |
Google's documentation is explicit about user-triggered fetchers:
“Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules.”
— Google Search Central, Google's user-triggered fetchers
In other words, robots.txt manages crawling, not every visit. If you want to control agents, you need server-side rules or a bot management tool, and if you want them to succeed, you need a site they can use.
A Step-by-Step Checklist to Make Your Site Agent-Ready
- Check crawl access. Make sure robots.txt, firewall and CDN rules allow Googlebot, Bingbot and OAI-SearchBot. Decide separately whether you allow training crawlers like GPTBot. Blocking retrieval crawlers removes you from AI answers that agents start from.
- Render key content on the server. Prices, product details, service areas, opening hours and policies should be in the initial HTML. Our guide to server-side vs client-side rendering explains why.
- Use semantic HTML. Real
<button>,<a>,<nav>,<main>and<label>elements give agents a clear map. See our guide to semantic HTML for SEO. - Label every control. Icon-only buttons need accessible names. Form fields need visible labels tied to inputs, not just placeholder text.
- Simplify forms. Ask only for what you need, accept common formats for phone numbers and dates, and show clear error messages next to the field.
- Tame interruptions. Cookie banners, pop-ups and chat widgets that cover content or trap focus block agents just as they annoy people.
- Add structured data. Product, Organization, LocalBusiness and Offer markup confirm facts in a machine-readable format. Start with our schema markup guide.
- Publish decision facts plainly. Shipping costs, return windows, cancellation terms, service areas and lead times are exactly what agents are asked to compare.
- Test with a real agent. Give an agent a realistic task, such as "book a consultation for next Tuesday", and watch where it stalls.
Accessibility Is Agent Optimisation
Much of what agents need is what screen-reader users have needed for years. Our guide to where accessibility and SEO overlap covers the fundamentals. Here is how specific fixes map to agent behaviour:
| Accessibility fix | Why it helps agents |
|---|---|
| Accessible names on icon buttons | The agent knows the magnifier icon means "Search" and the basket icon means "View cart" |
| Form labels linked to inputs | The agent puts the postcode in the postcode field, not the phone field |
| Logical heading order | The agent can jump to "Pricing" or "Delivery" sections quickly |
| Visible focus and keyboard support | Custom dropdowns and date pickers can be operated without precise mouse movements |
| Descriptive link text | "View delivery options" is clearer than "click here" |
| Text alternatives for key images | Information in charts or badges is not lost |
Managing Agent Traffic Responsibly
Not every bot claiming to be an agent is legitimate. Spoofed user agents are easy to fake, which is why verification is emerging. Cloudflare, for example, verifies "signed agents" using Web Bot Auth, a cryptographic method based on HTTP message signatures. In Cloudflare's words:
“HTTP message signatures allow bots to authenticate themselves and allow customer origins to identify them.”
— Cloudflare Blog, The age of agents: cryptographically recognizing agent traffic
Practical steps:
- Review server logs for AI user agents. Our guide to log file analysis shows how.
- Allow verified agents and challenge unverified high-volume traffic rather than blocking all automation.
- Rate-limit sensitive endpoints such as login, search and checkout.
- Keep high-risk actions, such as payments and account changes, behind confirmation steps. Most agents already ask the user before purchases.
What About llms.txt and New Protocols?
You will see proposals such as llms.txt, and newer standards for agents to call site functions directly, such as WebMCP. These may become useful, but they are not prerequisites. Our explainers on llms.txt and WebMCP cover the current state. Get the fundamentals right first: accessible pages, server-rendered facts and forms that work.
Common Mistakes
- Assuming robots.txt controls agents. It controls well-behaved crawlers; user-triggered agents may not follow it.
- Blocking every AI bot at the CDN. This can remove you from ChatGPT search and stop customers' agents completing purchases.
- Putting prices in images or behind "call us". Agents asked to compare options will favour sites with clear figures.
- Custom controls built from
divs. Without roles and names, agents cannot tell what they do. - Aggressive CAPTCHAs on every form. Use risk-based challenges instead.
- Never testing. Watching one agent attempt your booking flow is the fastest audit you can run.
Related Guides
- Where Does AI Get Its Answers? How LLMs Choose Sources
- Losing Traffic to AI Answers? How to Adapt Your Strategy
- Digital PR for AI Visibility: Earn Mentions LLMs Trust
Frequently Asked Questions
What is an AI agent in the context of websites?
It is software that visits websites on a person's behalf to complete a task, such as researching options, filling a form or preparing a purchase. Examples include ChatGPT agent, ChatGPT Atlas and Gemini's auto browse in Chrome.
Do AI agents follow robots.txt?
Not always. Google says its user-triggered fetchers, including Google-Agent, generally ignore robots.txt because a user requested the fetch, and OpenAI says robots.txt rules may not apply to user-initiated ChatGPT-User actions. Automated crawlers such as Googlebot and OAI-SearchBot do follow robots.txt.
Is optimising for AI agents different from accessibility work?
They overlap heavily. OpenAI says ChatGPT Atlas uses ARIA tags to interpret page structure and interactive elements, so accessible buttons, labels and forms help both people using assistive technology and agents.
Do I need an llms.txt file for AI agents?
It is optional and not a requirement from Google or OpenAI. Focus first on crawl access, server-rendered content, clear structure and working forms.
Should I block AI agents with a bot manager?
Block abusive traffic, but think carefully before blocking legitimate agents acting for real customers. Look at verification options such as signed agent requests so you can allow trusted agents and challenge the rest.
Build a Site That Agents (and People) Can Use
AI agents reward the same qualities good websites always needed: clear structure, honest facts, accessible controls and simple forms. Fix those, keep the right crawlers allowed, and your site becomes the easy choice when an agent is asked to compare, book or buy.
Our SEO services team runs technical audits that cover crawl access, rendering, structured data and agent usability. Get a quote to see how agent-ready your site is.
References
- OpenAI Help Center: Publishers and developers FAQ
- OpenAI: Overview of OpenAI crawlers
- Google Search Central: Google's user-triggered fetchers
- Cloudflare Blog: The age of agents: cryptographically recognizing agent traffic
- OpenAI: Introducing ChatGPT agent
- 9to5Google: Chrome Gemini auto browse
- Semrush: How to prepare your site for AI agents
