A reported citation gap between ChatGPT and OpenAI’s API sounds bigger than it may be. The article examines a narrower question: whether the same prompt surfaces the same sources across those two outputs.
Table of Contents
An October 11 arXiv paper from researchers at NUS and Xi'an Jiaotong reported only partial overlap in a literature-review test. That matters for marketers and agencies because citation tracking can reflect product-layer differences, not just source quality, trust, preference, or intent.
What the citation gap claim actually means
The citation gap claim matters because a mismatch between ChatGPT citations and API outputs does not, by itself, show hidden favoritism or deliberate steering. It points to a narrower issue: the system may surface sources unevenly, and source selection does not always track source quality.
A LinkedIn post by Jiahao Chen, summarizing University of Sheffield research on GPT-4o literature reviews, describes a related pattern: ChatGPT cited papers even when they had been retracted or were otherwise problematic, and its scoring was not correlated with altmetrics.
Chen also notes that the model did not appear to synthesize outside signals that flagged those papers. That makes the practical takeaway clear for measurement work: citation visibility can be informative, but it is not a clean proxy for reliability, preference, or trust.
How the study compared ChatGPT citations with API outputs
The comparison setup remains thin here, but the Facebook post frames a literature-review test between ChatGPT and API outputs.
- To test the gap, the NUS and Xi'an Jiaotong researchers compared the sources ChatGPT cited in its interface with sources surfaced through OpenAI's API for the same prompts. That side-by-side design isolates differences, but it does not by itself explain why they occur.
- That setup matters because the NUS and Xi'an Jiaotong arXiv paper compared ChatGPT citations against what OpenAI's API surfaced, not against source quality itself. For measurement work, that makes the test a consistency check first, and a trust judgment only later.
What the researchers found, and where the match fell short
Partial citation matches matter because, as Search Engine Journal reports on Columbia’s Tow Center, ChatGPT can misquote and misattribute sources.
- The NUS and Xi'an Jiaotong arXiv paper reports incomplete overlap. OpenAI's API did not fully reproduce the sources ChatGPT cited in the interface for the same literature-review prompts. In practical terms, some visible citations were missing from the API-side source list.
- The arXiv paper from NUS and Xi'an Jiaotong reports only a partial overlap between ChatGPT's visible citations and sources returned through OpenAI's API. Practically, that means citation tracking can miss or misread exposure if it treats either surface as complete.
Why partial matches can happen without proving bias or intent
That narrower reading matters. A partial match can reflect how different system layers assemble, format, or expose sources, not a motive. It shows inconsistency between outputs, but it does not reveal whether any source was favored on purpose.
A related warning appears in a 2023 YouTube video titled “Yes, ChatGPT makes up citations for Literature Reviews,” which points to another failure mode: citation problems can come from generation errors themselves.
In that sense, a gap between interface citations and API results may signal incomplete retrieval, formatting differences, or post-processing, rather than bias. For anyone measuring AI visibility, the practical implication is simple: treat a citation mismatch as evidence of a tracking limit first, and as evidence of intent only if separate testing supports that stronger claim.
How much weight to give the findings, scope, limits, and open questions
Semantica AI's data suggests caution. AI citations are noisy and highly concentrated.
- The arXiv paper from NUS and Xi'an Jiaotong supports a measured conclusion: consistency is imperfect. Its real limit is scope, because open questions remain about prompt range, interface layers, timing, and reproducibility.
- Give the result moderate weight: the arXiv paper compares literature-review outputs, not every ChatGPT use case. That leaves open whether the same mismatch appears across other prompts, models, or citation features.
- Because the NUS and Xi'an Jiaotong arXiv paper compares literature-review outputs, its result carries moderate weight, not final proof. It suggests a consistency gap, while leaving task breadth, cause, and frequency open.
- Does this test measure one literature-review setup, or a broader citation pattern? Because the NUS and Xi'an Jiaotong arXiv paper compares overlap, its findings are useful but still leave interface-pipeline questions open.
When API results may diverge from the ChatGPT interface
- Timing and surface differences: Divergence is most plausible when the same literature-review prompt passes through different product layers or runs at different moments. The NUS and Xi'an Jiaotong arXiv paper compared those surfaces, so API logs alone may miss what the interface shows.
What this means for agencies tracking AI visibility and mentions
For agencies, the main takeaway is simple: mention tracking is not the same as citation validation. A brand, paper, or publisher may appear in AI output without a reliable source trail behind it. TrueStandard says language models can invent studies, DOIs, and cases with the same confident tone they use for real material, and its roundup places fake-source rates between 18 percent and more than 50 percent, depending on the model.
That turns raw visibility data into a risk signal, not a clean measure of earned authority. The cost can be real: TrueStandard cites the 2023 Mata v. Avianca case, where fabricated cases helped trigger a $5,000 sanction.
For reporting, that means logs should separate mention counts from verified citations, and reserve manual checks for high-stakes mentions.
ChatGPT’s citation gap with API output appears real, but only in a limited sense. The NUS and Xi'an Jiaotong literature-review comparison points to partial overlap, not proof of favoritism, intent, or source quality.
LinkedIn commentary from Jiahao Chen on University of Sheffield research adds a related caution: cited papers can still be retracted or otherwise unreliable. For agencies and marketers, the practical takeaway is straightforward.
Treat citation visibility as a useful signal, but not as a clean measure of trust, preference, or verified authority without separate checks.
