What an AI Bot Sees When It Crawls Your Site: A 10-Minute Test

Last updated: 1 August 2026. AI crawler behaviour changes fast; this article is reviewed every two months.

If you want to know whether ChatGPT, Claude and Perplexity can actually see the content on your site, you don’t need an audit or a tool. Open any page, press Ctrl+U to view the page source, and search the raw code for your main text: a product description, the body of an article, a service description. If the text is there, AI crawlers can read it. If all you see is <div id="root"></div> and a list of <script> tags with no content, your page is a blank shell to most AI crawlers.

The reason is single and binary. As of June 2026, none of the major AI crawlers execute JavaScript: this includes GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Meta-ExternalAgent and Bytespider. The one meaningful exception is Gemini, which leverages Googlebot’s rendering infrastructure (Lantern, June 2026). A bot fetches the raw HTML your server returns in the first response, reads it as it arrives, and stops. Anything added to the page only after JavaScript runs in the browser does not exist for it.

This is why a page can rank number one on Google and still be invisible in ChatGPT. Googlebot is today effectively the only mass-scale crawler with full JS rendering, so it renders your content and indexes it. The AI bot gets the same empty shell that Googlebot has learned to work around. This article shows how to find out, in about ten minutes, which side of that line you are on, and what to do with the answer.

Why Googlebot sees your content and ChatGPT doesn’t

Googlebot behaves like a browser: it fetches the HTML, then runs a headless Chrome renderer, executes the JavaScript, waits for content to load, and only takes that fully assembled document into its index. An AI crawler behaves like curl: it fetches the first server response and reads it as-is. If that first response already contains finished text (server-side rendering, or SSR), both kinds of bot see the same thing. If the text is added later by a script in the browser (client-side rendering, or CSR, typical of single-page apps built in React, Vue or Angular without SSR), Googlebot copes and the AI bot sees the shell.

One myth is worth defusing immediately, because it produces false comfort. On 4 March 2026 Google removed a long-standing JavaScript SEO warning from its documentation, signalling the maturity of its own rendering (Lantern, June 2026). Teams read this as “client-side rendering is safe now.” It is safe, but only on Google’s side. For ChatGPT, Claude and Perplexity nothing changed: they still don’t render.

There is a nuance worth knowing so you don’t panic without cause. Content doesn’t have to be visible prose to count. Data embedded in the initial HTML as inline JSON, structured-data markup (JSON-LD) or a server-rendered component payload is still in the raw response and readable by AI crawlers (SearchOptimo, June 2026). What they miss is specifically the content the browser builds on the client after the page loads.

Test 1: view page source (Ctrl+U), 30 seconds

This is the fastest test and in most cases the only one you need.

1. Open a page that needs to be visible in AI (a product page, an article, a service description).
2. Press Ctrl+U (Windows/Linux) or Cmd+Option+U (Mac). This opens the raw HTML, exactly what the bot receives.
3. Press Ctrl+F and search for a piece of your visible text, for example a sentence from a product description.

The reading is simple. If the text is in the source, the AI bot sees it. If it isn’t, and instead of content you see an empty container and scripts, the content is injected by JavaScript and is invisible to AI crawlers. Important: view source shows what the server returns, not what you see on screen after the page loads. Use View Page Source, not the DevTools Elements panel, which shows the rendered DOM after JavaScript has already run (SearchOptimo, June 2026).

Test 2: disable JavaScript in the browser, 2 minutes

This test shows exactly what the bot sees, because it simulates a browser with no JS.

1. In Chrome, open DevTools (F12), then the command menu (Ctrl+Shift+P).
2. Type “Disable JavaScript” and confirm.
3. Reload the page without closing DevTools.

If the page still shows content after reload, you are safe. If you are left with an empty layout, a loading spinner or an “enable JavaScript” message, that is the picture the AI crawler gets. This test is stronger than view source because it also catches the in-between cases: a page that server-renders part of its content (say the header and menu) but injects the key material (descriptions, prices, reviews) with a script.

Test 3: fetch the page as an AI bot with curl, 3 minutes

If you want to see exactly what a specific crawler fetches, reproduce its request from the command line. This is the most reliable test for a specialist, because it bypasses the browser entirely.

curl -L -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" https://yourdomain.com/your-page

The -A flag spoofs the bot’s user-agent, -L follows redirects. Search the returned HTML for your main text. If it’s there, the bot gets it. If the response is just a shell, the content depends on JavaScript and the bot won’t see it. Repeat with other user-agents by swapping the string: ClaudeBot/1.0, PerplexityBot/1.0, OAI-SearchBot/1.0. The results should be similar, since none of them render JS. For a one-liner confirmation: curl https://yourdomain.com | grep 'your sentence'.

One distinction that’s easy to trip over. Automatic crawlers (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) browse the web on their own and build indexes. User-triggered agents (ChatGPT-User, Perplexity-User) fetch a page on demand when someone pastes a URL into the chat, and may behave differently toward access rules. For a rendering test that doesn’t matter; for a decision about blocking bots it does (we cover that in a separate article on robots.txt).

Test 4: compare against Googlebot’s render in Search Console, 5 minutes

This step shows how wide the gap is between what Googlebot sees after rendering and what the server returns in raw HTML.

1. In Google Search Console, use the URL Inspection tool.
2. Enter the page URL and open the tested-page view.
3. Check the crawled HTML tab and the rendering screenshot.

If there is a large difference between the raw HTML and the rendered view, it means Googlebot only reaches the content at the JS rendering stage. If Googlebot has to render to see the content, an AI crawler that doesn’t render won’t see it at all. This test doesn’t show the AI bot’s view directly, but the gap between Googlebot’s two views is a reliable warning sign.

Test 5: ask the model directly, 2 minutes

The simplest test, requiring no tools. Open ChatGPT, Claude or Perplexity and paste your page URL with the prompt: “Read this page and summarise the first paragraph.” If the model returns a specific summary of your content, the page is readable to it. If it says it can’t access the content, or gives a generic answer instead of specifics from your page, you have a rendering problem (Glenn Gabe, cited via semgence.pl, May 2026).

This test has one caveat: a user-triggered agent is sometimes smarter than an automatic crawler and may fetch content differently. Treat it as a supplement to tests 1 through 4, not as the only proof.

How to read the result: decision table

What to do if the test failed

If the key content turns out to depend on JavaScript, there is one direction for the fix: get the main content into the first server response. There are several paths, differing in cost and durability:

• Server-side rendering (SSR) or static generation (SSG). The cleanest solution: the server returns finished HTML with content. For modern stacks this usually means switching to the right framework mode (for example server components). Highest implementation cost, best lasting result.
• Prerendering. A separate layer serves bots a pre-rendered, static version of the page. Fast to deploy and good as a bridge, but it has to be maintained, and you must ensure the bot version doesn’t drift from the user version.
• Hydration with content in the initial HTML. The content is in the first HTML, and JavaScript only “revives” the finished document instead of building it from scratch. Requires correct framework configuration, but combines the best of both worlds.

The choice depends on your stack, dev resources and how urgent the issue is. We break down which approach makes sense when in a separate article on rendering architecture. If you’d rather diagnose this on your own site with a team that runs these audits every week, that’s what Insightland’s technical SEO services are for.

Which bots render JavaScript and which don’t (as of August 2026)

Source: Lantern, June 2026, SearchOptimo, June 2026. This category changes fast, check the verification date at the top of the article.

FAQ

Does ChatGPT read JavaScript?
No. OpenAI’s crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) fetch raw HTML and do not run JavaScript. If your content is injected by a script in the browser, ChatGPT won’t see it. The same applies to Claude and Perplexity.

My site ranks number one on Google, so it’s fine, right?
Not necessarily. Googlebot renders JavaScript, so it sees content an AI crawler will miss. A high Google ranking only tells you about visibility in Google, not in ChatGPT or Perplexity. These are two different channels with different access rules.

How do I check whether my site is visible to AI bots without a paid tool?
Fastest: view page source (Ctrl+U) and search the code for your text. If the text isn’t there, the content depends on JavaScript. To be sure, disable JavaScript in DevTools and reload, you’ll see exactly what the bot sees.

Does WordPress have this problem?
Usually not. Classic CMS platforms like WordPress, Joomla and Drupal generate HTML server-side, so the bot gets finished content in the first response. The problem mainly affects single-page apps (SPAs) and JS frameworks run without server-side rendering.

Do I need an llms.txt file for AI to see my site?
No. Visibility depends on whether content is in the raw HTML, not on an llms.txt file. That’s a separate, heavily overhyped topic we cover in an article on AI visibility myths. First make sure bots can see your content at all.

How is this different from classic SEO?
The foundation is the same: a bot has to be able to reach the page and read the content. What’s new is that AI bots don’t render JavaScript, so the “content in the initial HTML” criterion became hard and binary, where for Googlebot it was soft. As Insightland’s Joanna Jelenik puts it, this is evolution, not revolution: good technical SEO is still the base, there’s just a new, sharper rule on top.

AI visibility starts with one verifiable question: is your content in the raw HTML the server returns. You can answer it in about ten minutes by running the tests in this article on your own site. If the content is there, you have a solid foundation and can move on to the content and citability layer. If it isn’t, no amount of content optimisation will help until you fix the rendering, because the model simply has nothing to read.

Run the test on your most important page now, before moving on. If the result is ambiguous, or you want to check a whole site rather than individual URLs, a technical SEO audit will show exactly where content drops out of AI crawlers’ reach and in what order to fix it.

AI crawler behaviour verified as of 1 August 2026 against Lantern, SearchOptimo and semgence.pl. This category ages in weeks; verify data against the primary source before relying on it.