Visibility in AI answers

Start with the server log

robots.txt tells you what you allow. The server log tells you who actually arrives, how often and for which pages. That difference matters more for AI systems than for classic search, because some of their requests are not crawls at all.

When someone asks ChatGPT to look at a page, the request comes from ChatGPT-User. OpenAI describes these as fetches for user actions and notes that robots.txt rules may not apply to them, because a person triggered the request. Only the log shows whether and how often that happens on your site.

The AI Bot Log Parser reads an excerpt of an Apache or Nginx access log directly in the browser, without uploading it, and shows which AI bots came and what they requested.

One decision per crawler purpose

The large providers now separate their crawlers by purpose, and the consequences of blocking them differ:

A site can therefore refuse training by OpenAI and still be eligible for ChatGPT search. With Google, one token covers both: if you block Google-Extended, the content is also excluded from grounding in Gemini Apps and in Grounding with Google Search on Vertex AI, while inclusion and ranking in Google Search stay unaffected. Because the trade-off differs by provider, a single rule for all AI bots rarely fits.

Crawler details as of 14 September 2026, from OpenAI's crawler documentation and Google's overview of its crawlers.

To see how your current robots.txt treats each of these bots, the AI Bot Checker tests it against a list of known AI crawlers.

What my own data shows, and what it does not

On minoka.de, around 50 visits arrived from AI assistants in the 90 days up to 13 September 2026. They landed mainly on free tools, most often the English domain age checker with 17 entries and the llms.txt tester with 10, and not once on a service page.

That is a small sample, and it counts visits, not mentions. It does match what Search Console showed at the end of August 2026: where a tool page answered a question phrased as a full sentence, minoka.de ranked in the top three, while questions looking for a provider found the same domain between positions 85 and 97.

For minoka.de, the pages people reached from AI answers were the ones that answer something. That is the working assumption I plan content around, and I keep checking it against new data.

Help machines connect the facts

Structured data is one place where a machine learns who is speaking. The person entry in this site's JSON-LD carries the identifier https://minoka.de/#person-sercan-kahraman, exactly as my profile on minoka.de does, so a system that reads both pages can connect them to one person.

The FAQ answers on this site follow the same logic. They are phrased in the third person, so that a sentence such as Sercan Kahraman speaks German, Turkish and English remains accurate when it is quoted without the question in front of it.

Where to begin

• Check robots.txt against the crawler purposes you actually want to allow.
• Read a week of server logs to see which AI bots come and what they request.
• Collect the questions your customers ask and check whether one of your pages answers each of them directly.
• Record a baseline before changing anything, so that a later difference can be measured at all.
• Optionally, publish an llms.txt file. It is a proposal, not an adopted standard, so treat it as a small extra and not a priority; the llms.txt Tester checks the format.

An audit can show whether those preconditions are in place and where a site is cited at the time of testing. A citation found once says little about the next request, so the comparison that counts is the same set of questions asked again after a change.

Working together

I offer GEO audits through minoka.de. The audit covers crawler access, the llms.txt file, presence in the Knowledge Graph and visibility in AI answers, and ends with a prioritised roadmap.

The groundwork underneath is described on technical SEO. Teams who would rather learn it themselves can request the course GEO Foundations.