Why ChatGPT Names Your Store on One Question and Skips You on Another
AI visibility is not one number, it is won question by question. We live-tested six Japanese beauty stores on six real buyer prompts: one store was recommended on a broad shipping question and completely absent from the prompt that literally describes its business model. The reason is structural: each question surfaces a different pool of third-party sources, and the AI recommends whoever those sources mention.
Most store owners ask "is my site visible to AI?" as if it had a yes-or-no answer. It does not. The same store can be the AI's recommendation on Monday's question and invisible on Tuesday's, with an identical website. We ran the test that shows it, on six real stores in one niche, with live web search, on the same day.
The test: one niche, six stores, six live buyer questions
We took six genuine Japanese beauty and skincare stores, all technically solid (AI-readiness scores 74 to 92), and asked an answer engine one targeted buyer-intent question per store, without ever naming a brand. Then we read who actually got named in each answer.
| Store | Readiness score | Named in the answers? | The telling detail |
|---|---|---|---|
| Shop A | 86 / 100 | Referenced in 6 of 6 | The niche's AI winner, present in every single answer, including other stores' questions |
| Shop B | 89 / 100 | Referenced in 5 of 6 | Named directly on its own core question, quoted in most others |
| Shop C | 92 / 100 | Named on its category question | The answer described its curation almost word for word |
| Shop D | 89 / 100 | Yes on one, no on another | Named on "cosmetics with worldwide shipping", absent on "skincare shipped directly from Japan", which is literally its model |
| Shop E | 74 / 100 | Linked, never named | Its site was cited as a source inside the answer, but the brand never appeared in the text a buyer reads |
| Shop F | 92 / 100 | Absent everywhere | Not in the answer, not even in the raw sources, despite the second-highest readiness score of the lot |
The headline case: right store, wrong question
Look at Shop D. Ask the engine "where can I buy Japanese cosmetics online with worldwide shipping?" and it is named as a pick. Ask "where can I buy authentic Japanese skincare shipped directly from Japan?", the sentence that could be this store's tagline, and the answer names two competitors instead. Same store, same website, same day, opposite outcomes.
The explanation is not mysterious. An answer engine does not evaluate your store and decide how good it is. It runs a web search for the question, gets back a pool of third-party pages, roundups, review sites, community threads, and composes its recommendation from what those pages say. Each question pulls up a different pool. Shop D happens to be mentioned in the lists behind the first question and missing from the lists behind the second. The citation follows the sources, question by question.
Niche maturity changes everything
Three days earlier we ran the identical protocol on six Japanese décor stores. Only 2 of 6 were ever named. In Japanese beauty: 4 of 6, with one store present in every single answer. Same test, same engine, wildly different visibility, because beauty has years of dense "best J-beauty" roundup and review coverage for the AI to lean on, and décor barely does.
Two practical consequences. If you are in a young niche, the roundups that will decide tomorrow's citations are being written now, and getting into them is cheap. If you are in a mature niche, the winners are already compounding, and the question-level gaps like Shop D's are where you can still take ground.
The measurement trap: named is not the same as linked
One more finding from this run, at our own expense. Shop C was named in the answer as "Shikō-style" branded prose, with a Japanese macron in the name, and no link to its domain. A naive detector that only checks cited URLs, ours included until that afternoon, counts that as "not cited". The opposite trap also exists: Shop E was linked as a source but never named, which a URL-only check would count as a win a buyer never sees. If you measure your AI visibility, make sure the tool reads the answer the way a buyer does: the brand named in the text is the citation that sells. We fixed our detector the same day, and the fix shipped before this article did.
What to do with this
Stop asking "am I visible to AI" and start asking "on which buyer questions am I visible". The work looks like this:
- Map your buyer questions. Write the five to ten questions a buyer would type when researching your category: product types, shipping model, price positioning, no brand names.
- Test each one live. Record who gets named, question by question. Your visibility profile will almost certainly be uneven, like Shop D's.
- Fix the gaps per question. For each question where you are absent, find the roundups and review pages that answer pulls from, and get mentioned there. Our free off-page gap finder maps the channels, and the off-page AEO playbook ranks the moves.
- Give it 60 days. Off-page changes lag four to eight weeks as engines re-crawl. Measure again on the same questions, not on new ones.
See your own question-by-question profile
CrawlBit runs this exact protocol on your store: a full AI-readiness audit, then live citation tests on real buyer questions for your niche, telling you where you are named, where you are only a source, and which third-party pages would flip the questions you are losing. Start with a free scan, the citation test runs from your dashboard.
Frequently asked questions
Because each buyer question surfaces a different pool of third-party sources: "best X" roundups, review platforms, community threads. The AI builds its recommendation from whichever pool that question pulls up. If the lists behind question A mention you and the lists behind question B do not, you are visible on A and invisible on B, even though your website never changed.
Start with five to ten: the questions a buyer would actually type when researching your category, without naming any brand. Cover your main product categories, your shipping or service model, and your price positioning. A single prompt gives a misleading picture, since visibility can flip completely from one question to the next.
Dramatically. In our live tests, only 2 of 6 Japanese décor stores were ever named by the AI, while 4 of 6 Japanese beauty stores were. Mature niches already have dense roundup and review coverage for the AI to lean on; younger niches barely have sources to quote. Test your own niche before assuming anything.
Yes. In our test one store's site was linked as a source inside the AI's answer, but the brand was never named in the text a buyer reads. Being the source without being the recommendation usually means the AI trusts your content but has no third-party corroboration to name you as a pick. It is a real, fixable gap.