Your site might look perfect in a browser and still be close to empty for an AI assistant. If your product details, prices, reviews or article text only appear after JavaScript runs, there is a real chance that ChatGPT, Claude or Perplexity never sees them when their crawlers visit.
Google spent years building a rendering pipeline so Googlebot can execute JavaScript. Most newer AI crawlers have not published anything similar, and the best public evidence suggests many of them read the raw HTML your server sends and move on. That gap matters in 2026, because AI answers increasingly decide which brands a buyer hears about first.
This guide explains what the evidence actually says, how to test what an AI crawler receives from your pages, and which rendering fixes (server-side rendering, static generation and prerendering) close the gap without rebuilding your whole site.
Key Takeaways
- A December 2024 Vercel and MERJ analysis found that the major AI crawlers it studied (GPTBot, ClaudeBot, PerplexityBot and others) fetched pages but did not execute JavaScript.
- Google renders JavaScript for Search, and Gemini uses Googlebot infrastructure, but that does not help with ChatGPT, Claude or Perplexity.
- The fastest test is to look at the raw HTML: if your main content is missing from
curloutput or "view source", assume most AI crawlers cannot read it. - Server-side rendering, static generation or prerendering of critical content is the durable fix; client-side rendering is fine for non-essential widgets.
- Re-test after every front-end release, because framework updates can quietly move content back to the client.
What the evidence says about AI crawlers and JavaScript
AI companies document who their crawlers are far more than how they process pages. OpenAI’s crawler overview explains the purpose of OAI-SearchBot, GPTBot and ChatGPT-User, and Perplexity’s bot documentation does the same for PerplexityBot and Perplexity-User, but neither page says whether those bots execute JavaScript. So the most useful evidence comes from server-side measurement.
The best-known study is The rise of the AI crawler, published by Vercel with MERJ on 17 December 2024. It analysed crawler traffic across nextjs.org and the Vercel network and found that ChatGPT’s crawlers, ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider did not render JavaScript. Some of them downloaded JavaScript files (the study reports 11.50% of ChatGPT fetches and 23.84% of Claude fetches were JS files) but did not execute them. The two exceptions were AppleBot, which used a browser-based crawler, and Gemini, which relies on Googlebot’s infrastructure.
“The results consistently show that none of the major AI crawlers currently render JavaScript.”
— Vercel, The rise of the AI crawler
Two caveats are worth stating plainly. First, the study is from late 2024 and crawlers change; a vendor could add rendering at any time without announcing it. Second, it measured specific networks rather than the whole web. Treat it as strong evidence of the default behaviour, not a permanent rule, and verify on your own site with the tests below.
How the main crawlers compare
| Crawler | Operator | Renders JavaScript? (Vercel/MERJ, Dec 2024) | What it means for you |
|---|---|---|---|
| Googlebot (Search, AI Overviews, AI Mode, Gemini) | Yes, via a rendering queue | Client-side content can be indexed, but rendering is a separate, later step | |
| GPTBot, OAI-SearchBot, ChatGPT-User | OpenAI | No | Only server-sent HTML is reliably visible to ChatGPT |
| ClaudeBot | Anthropic | No | Training and retrieval see the raw HTML |
| PerplexityBot | Perplexity | No | Answers and citations depend on raw HTML |
| AppleBot | Apple | Yes (browser-based crawler) | Closer to Googlebot behaviour |
| Meta-ExternalAgent, Bytespider | Meta, ByteDance | No | Same raw-HTML limitation |
Even Google says rendering on the server is worth doing. Its JavaScript SEO basics guide notes that server-side or pre-rendering makes a site faster for users and crawlers, and that not all bots can run JavaScript. If you want a refresher on Google’s crawl, render and index pipeline, our guide to JavaScript SEO covers it in detail.
Which pages are most at risk
Not every JavaScript site has a problem. A WordPress site that uses JavaScript for a slider is usually fine, because the article text is already in the HTML. The risk is highest when the content itself is assembled in the browser:
- Single-page apps built with React, Vue or Angular that ship an almost empty
<div id="root">and fetch everything after load. - E-commerce templates that inject price, stock status, variants or reviews through client-side API calls.
- Headless CMS front ends where the build was configured for client-side rendering only (see our headless CMS SEO guide).
- Tabs, accordions and "load more" sections whose text is fetched on click rather than included in the page.
- Help centres and documentation portals that render from a JSON search index.
How to test what AI crawlers can see
You do not need special tools. The goal is to look at the HTML before any JavaScript runs, because that is what a non-rendering crawler receives.
- Fetch the raw HTML. Run
curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://www.example.com/page/ > raw.html. Using a crawler-style user agent also reveals whether your CDN or firewall serves bots something different. - Search the output for key facts. Look for the H1, the first paragraph, the price, the product name and a review sentence. If they are missing, a non-rendering crawler cannot see them either.
- Compare with the rendered DOM. Open the page in Chrome DevTools and compare the Elements panel with "view source". Large differences mean heavy client-side rendering.
- Disable JavaScript. In DevTools, disable JavaScript and reload. What remains is roughly what GPTBot or ClaudeBot works with.
- Check status codes and redirects. A 200 response with an empty shell is as bad as a 404 for AI crawlers. Make sure redirects resolve in one hop.
- Confirm with logs. Check your server logs to see whether AI crawlers actually request the pages you tested (our new guide to finding AI bots in your server logs shows how).
- Ask the assistant. As a rough sanity check, ask ChatGPT or Perplexity a question that only your client-rendered content answers. Treat the result as anecdotal, because answers vary from run to run.
Fixes: from quick wins to full SSR
The Vercel study’s recommendation is simple: server-render critical content such as the main text, metadata and navigation, and keep client-side rendering for non-essential features. In practice you have several routes.
| Approach | How it works | Best for | Trade-offs |
|---|---|---|---|
| Server-side rendering (SSR) | HTML is generated on the server for each request | Dynamic pages: product, pricing, search results | More server work; needs caching |
| Static site generation (SSG) | HTML is built at deploy time | Blogs, docs, landing pages | Content changes need a rebuild |
| Incremental static regeneration (ISR) or similar | Static pages refreshed on a schedule or on demand | Large catalogues with frequent updates | Framework-specific; stale windows |
| Prerendering service | A headless browser renders pages and serves snapshots to bots | Legacy SPAs that cannot be refactored quickly | Extra cost; risk of serving different content to bots and users |
| Hybrid / progressive enhancement | Core content in HTML, interactivity added by JavaScript | Most sites | Requires discipline in templates |
Google describes dynamic rendering (serving bots a different, prerendered version) as a workaround rather than a long-term solution, so if you use a prerendering service, make sure the snapshot matches what users see. Mismatched content can create trust problems with both search engines and AI systems. For performance side effects, our page speed optimization team usually pairs SSR with edge caching so the server is not rebuilding the same page thousands of times.
What to put in the initial HTML
- The title tag, meta description, canonical and robots meta tags.
- The H1 and the full main content, including the opening answer paragraph.
- Prices, availability, key specifications and a sample of reviews for product pages.
- Internal navigation links as real
<a href>elements, not click handlers. - Structured data (JSON-LD) that matches the visible text.
- Author, publication and update dates for articles.
Common mistakes
- Assuming "Google can render it" means every AI can. Google’s rendering helps Search and Gemini; it does nothing for other assistants.
- Testing only in a browser. Your browser always runs JavaScript, so it hides the problem.
- Injecting JSON-LD with JavaScript only. Google may process it after rendering, but non-rendering crawlers will not see it.
- Blocking JS and CSS for bots to "save crawl budget". This breaks Google rendering and does not help AI crawlers.
- Serving bots a stripped-down page. Prerendered snapshots must contain the same facts users see.
- Ignoring 404s from old asset URLs. The Vercel data showed high 404 rates for AI crawlers (about 34% of ChatGPT and Claude fetches), often on outdated assets. Keep redirects and sitemaps tidy (see our crawl budget guide).
Where this fits in your AI search plan
Rendering is a foundation, not a strategy. Once AI crawlers can read your pages, the content still has to be worth citing: clear answers, well-structured passages and evidence. Pair this fix with our guides to answer-first writing, AI-friendly tables and lists and structured data for AI search. And before you invest, check that you are not blocking the crawlers in the first place; our guide to AI crawlers and robots.txt and the new Cloudflare AI Crawl Control explainer cover access settings. The Semrush study of technical factors in AI search (January 2026) also flags server-side rendering as a practical safeguard, although it did not measure JavaScript’s effect directly.
Related Guides
- Finding AI Bots in Your Server Logs: A Practical Guide
- Fake AI Bots: How to Verify GPTBot, ClaudeBot and PerplexityBot Requests
- Cloudflare’s AI Crawler Controls and Pay Per Crawl Explained
Frequently Asked Questions
Does ChatGPT render JavaScript when it crawls my site?
Public evidence says generally not. The Vercel and MERJ analysis (December 2024) found that OpenAI's crawlers fetched pages without executing JavaScript. OpenAI does not document rendering behaviour, so test your own raw HTML.
Does Google AI Mode see client-rendered content?
Google renders JavaScript for Search, and Google says pages must be indexed to appear as supporting links in AI Overviews or AI Mode. Client-rendered content can therefore be used, but server rendering is still faster and more reliable.
Is prerendering considered cloaking?
Not if the prerendered snapshot contains the same content users see. Problems start when bots receive different facts, prices or text than human visitors.
Do I need to rebuild my site with a new framework?
Usually not. Most modern frameworks support server rendering or static generation per page. Start with your highest-value templates, such as product, pricing and service pages.
How often should I test?
After every significant front-end release and at least quarterly. Framework upgrades and new components can move content back to the client without anyone noticing.
Conclusion
If your important content only exists after JavaScript runs, you are invisible to a large part of the AI search ecosystem. Test the raw HTML today, move critical content to the server, and re-test after each release. If you want a second pair of eyes, our technical SEO specialists can audit your rendering setup as part of an SEO engagement, or you can request a custom quote for a JavaScript and AI-crawler audit.
