It is tempting to think search is being replaced by chat. In practice, assistants still need sources, and your website is the one source you control. This guide sets out what Google and the AI companies say they read, as of September 2026, and what that means for your site.

What has changed

Google now answers many searches with AI. In June 2026 it reported more than 2.5 billion monthly users of AI Overviews and more than one billion of AI Mode. ChatGPT, Claude, Perplexity and Copilot also search the web and link to what they used.

These features work by retrieval. The assistant turns a question into several searches, reads the pages it finds and writes an answer with links. It can only cite a page it was able to find and read, so the old question, "do we rank first?", becomes "are we one of the sources?".

The work behind that is familiar. Google's own guide says that optimising for generative AI search "is optimizing for the search experience, and thus still SEO".

What it does to clicks is disputed. Pew Research tracked 900 US adults in March 2025 and found they clicked a normal result on 8% of visits to a results page with an AI summary, against 15% without one. Google says total clicks from Search to websites have been "relatively stable year-over-year", and that clicks from pages with AI Overviews are of higher quality. Plan for fewer casual visits and more considered ones.

How assistants read the web

Three kinds of automated visitor matter, and each does a different job.

The distinction matters because the rules are separate. OpenAI states that a site which opts out of OAI-SearchBot "will not be shown in ChatGPT search answers", while blocking GPTBot only keeps pages out of training. Fetches made on a person's request are different again: OpenAI, Perplexity and Google all say these may not follow robots.txt, because a person asked for the page.

Rendering matters too. Googlebot runs JavaScript, but Google adds that server-side rendering "is still a great idea", because "not all bots can run JavaScript". A study by Vercel and MERJ in December 2024 found that none of the main AI crawlers ran JavaScript at the time. The safe choice is a page whose content is already in the HTML the server sends.

Structure beats volume

Google's guide to AI search names several popular tactics as things you can ignore: splitting content into "chunks", rewriting it for AI systems and chasing mentions. What it says will matter most in the long run is content people find "unique, compelling, and useful".

In practice that means a site built like this:

• One topic per page, with a title and a first paragraph that say what the page is about.
• Facts in text. Prices, dates, opening hours and specifications that sit only in an image or a PDF are easy to miss.
• Headings that describe their section, in order, and ordinary semantic HTML. It helps people using screen readers, and the browser agents that now read a page through the same accessibility information.
• Pages kept current. An assistant that quotes last year's price does you no favours.

A hundred thin posts do less for you than ten pages that each answer one question well.

Structured data

Structured data is a short block of code that states what a page is about in a shared vocabulary: this is an event, it starts on this date, a place costs this much.

The useful types for most businesses are Organization or LocalBusiness, Product, Event, Article and breadcrumbs. Some features have gone: FAQ rich results stopped appearing in Google on 7 May 2026, and review stars are not shown for reviews a business publishes about itself. The markup is still worth having, but for what it does, not for what it used to do.

Speed is still a signal

Google uses the Core Web Vitals in ranking: how soon the main content appears (good at 2.5 seconds or less), how quickly the page reacts to a tap (200 milliseconds or less) and how still the layout stays (a score of 0.1 or less). They are judged on real visits, at the 75th percentile.

Keep it in proportion. Google says relevance comes first and that good scores guarantee nothing. But in the July 2025 data of the HTTP Archive's Web Almanac, only 48% of mobile websites passed all three, so a fast site still stands apart from half the web.

Nothing published says assistants choose their sources by speed. The case for speed is simpler: fast, server-rendered pages are easy to crawl, and the people who do click through stay.

What about llms.txt?

llms.txt is a proposal from September 2024: a Markdown file at the root of a site that lists its most important pages for AI agents.

It is worth being precise about what it does. Google says its Search ignores the file, so having one "will neither harm nor help". None of OpenAI's, Anthropic's or Perplexity's crawler documentation says the file affects what their assistants cite. Where it is in real use is developer documentation, which coding agents read; those three companies publish one for their own docs.

So publish one if you have documentation or an API, and treat it as optional otherwise. We publish one on this site. It is never a substitute for a sitemap, which search engines do use.

Controlling what AI can use

You have more control than you did a year ago.

• Google's AI answers. Since 31 August 2026 every site has a setting in Search Console that includes it in, or excludes it from, AI Overviews, AI Mode and the AI features in Discover. Google says it is not a ranking signal. The older snippet controls also limit what AI features can use.
• Google's model training. The Google-Extended rule in robots.txt covers Gemini training and grounding. It does not affect Search.
• Other assistants. Allow the search crawlers you want to be cited by, and decide separately about the training crawlers. Apple offers Applebot-Extended for the same purpose, and Microsoft uses meta tags instead of a separate crawler name.

Remember that robots.txt is a request. Well-behaved crawlers follow it; it is not a lock.

Measuring it

• Search Console counts clicks and impressions from AI Overviews and AI Mode inside your normal totals. Its Generative AI performance report, worldwide since 31 August 2026, shows impressions in those features by page, country and device. It does not show clicks yet.
• Bing Webmaster Tools has had an AI Performance report since February 2026, showing how often Copilot and Bing's AI answers cite your site.
• Your analytics show visits referred by assistants, with one caveat: visits from an assistant's app often arrive without a referrer, so the real number is higher.

We found no trustworthy figure for what share of a typical site's visits come from assistants. Measure your own and watch the trend.

A short checklist

1. Check the basics: every important page is indexed, returns a normal response and has its content in the HTML.
2. Give each page one job and a plain summary at the top.
3. Put the facts in text: prices, dates, hours, locations.
4. Add structured data that matches the page: your organisation, products, events, articles and breadcrumbs.
5. Set a crawler policy: which search crawlers to allow, and whether to allow training.
6. Publish a sitemap and submit it.
7. Pass the Core Web Vitals on mobile.
8. Look at the AI reports monthly and at which pages get cited.

If you'd rather hand it over, our website development service builds all of this in from the first page.

Common questions

Do we need to write differently for AI?

No. Google lists rewriting content for AI systems among the things to ignore. Write clearly for people, and answer the question early in the page.

Should we block AI crawlers?

It depends on the crawler. Blocking a training crawler does not remove you from that company's search answers, and blocking its search crawler does. Most businesses want to be cited, so they allow the search crawlers and make a separate decision about training.

Does FAQ markup still help?

Not in Google, where FAQ rich results stopped appearing in May 2026. The markup does no harm, and a good FAQ section is still useful to readers.

Will AI answers replace our traffic?

The evidence is mixed. Pew's study found fewer clicks on results pages with AI summaries, and Google says overall clicks are stable. What is clear is that answers are built from sources, and a clear, fast, well-structured site is how you become one.

---

Sources, as of September 2026: AI features and your website (opens in a new tab), Optimizing your website for generative AI features (opens in a new tab), New controls for website owners (opens in a new tab), Google's common crawlers (opens in a new tab), OpenAI crawlers (opens in a new tab), Anthropic's crawlers (opens in a new tab), Perplexity crawlers (opens in a new tab), Applebot (opens in a new tab), JavaScript SEO basics (opens in a new tab), The rise of the AI crawler (opens in a new tab), Structured data policies (opens in a new tab), Google Search updates (opens in a new tab), Web Vitals (opens in a new tab), Page experience (opens in a new tab), Web Almanac 2025: Performance (opens in a new tab), llms.txt (opens in a new tab), Pew Research Center (opens in a new tab), Google on clicks from AI search (opens in a new tab), AI Performance in Bing Webmaster Tools (opens in a new tab).