A reported citation gap between ChatGPT and OpenAI’s API sounds bigger than it may be. The article examines a narrower question: whether the same prompt surfaces the same sources across those two outputs.
An October 11 arXiv paper from researchers at NUS and Xi'an Jiaotong reported only partial overlap in a literature-review test. That matters for marketers and agencies because citation tracking can reflect product-layer differences, not just source quality, trust, preference, or intent.
What the citation gap claim actually means
The citation gap claim matters because a mismatch between ChatGPT citations and API outputs does not, by itself, show hidden favoritism or deliberate steering. It points to a narrower issue: the system may surface sources unevenly, and source selection does not always track source quality.
A LinkedIn post by Jiahao Chen, summarizing University of Sheffield research on GPT-4o literature reviews, describes a related pattern: ChatGPT cited papers even when they had been retracted or were otherwise problematic, and its scoring was not correlated with altmetrics.
Chen also notes that the model did not appear to synthesize outside signals that flagged those papers. That makes the practical takeaway clear for measurement work: citation visibility can be informative, but it is not a clean proxy for reliability, preference, or trust.
How the study compared ChatGPT citations with API outputs
The comparison setup remains thin here, but the Facebook post frames a literature-review test between ChatGPT and API outputs.
- To test the gap, the NUS and Xi'an Jiaotong researchers compared the sources ChatGPT cited in its interface with sources surfaced through OpenAI's API for the same prompts. That side-by-side design isolates differences, but it does not by itself explain why they occur.
- That setup matters because the NUS and Xi'an Jiaotong arXiv paper compared ChatGPT citations against what OpenAI's API surfaced, not against source quality itself. For measurement work, that makes the test a consistency check first, and a trust judgment only later.
- The arXiv paper from NUS and Xi'an Jiaotong compared ChatGPT's visible citations with sources returned through OpenAI's API, but that design tests overlap, not the full interface pipeline. That tradeoff matters because a partial match alone cannot show why the source lists diverged.
What the researchers found, and where the match fell short
Partial citation matches matter because, as Search Engine Journal reports on Columbia’s Tow Center, ChatGPT can misquote and misattribute sources.
- The NUS and Xi'an Jiaotong arXiv paper reports incomplete overlap. OpenAI's API did not fully reproduce the sources ChatGPT cited in the interface for the same literature-review prompts. In practical terms, some visible citations were missing from the API-side source list.
- The arXiv paper from NUS and Xi'an Jiaotong reports only a partial overlap between ChatGPT's visible citations and sources returned through OpenAI's API. Practically, that means citation tracking can miss or misread exposure if it treats either surface as complete.
- The arXiv paper from NUS and Xi'an Jiaotong found only a partial overlap between ChatGPT's visible citations and OpenAI API outputs. The tradeoff is important: that shortfall shows a consistency limit, but not how the interface selected, ranked, or filtered sources.
Why partial matches can happen without proving bias or intent
That narrower reading matters. A partial match can reflect how different system layers assemble, format, or expose sources, not a motive. It shows inconsistency between outputs, but it does not reveal whether any source was favored on purpose.
A related warning appears in a 2023 YouTube video titled “Yes, ChatGPT makes up citations for Literature Reviews,” which points to another failure mode: citation problems can come from generation errors themselves.
In that sense, a gap between interface citations and API results may signal incomplete retrieval, formatting differences, or post-processing, rather than bias. For anyone measuring AI visibility, the practical implication is simple: treat a citation mismatch as evidence of a tracking limit first, and as evidence of intent only if separate testing supports that stronger claim.
How much weight to give the findings, scope, limits, and open questions
Semantica AI's data suggests caution. AI citations are noisy and highly concentrated.
- The arXiv paper from NUS and Xi'an Jiaotong supports a measured conclusion: consistency is imperfect. Its real limit is scope, because open questions remain about prompt range, interface layers, timing, and reproducibility.
- Give the result moderate weight: the arXiv paper compares literature-review outputs, not every ChatGPT use case. That leaves open whether the same mismatch appears across other prompts, models, or citation features.
- Because the NUS and Xi'an Jiaotong arXiv paper compares literature-review outputs, its result carries moderate weight, not final proof. It suggests a consistency gap, while leaving task breadth, cause, and frequency open.
- Does this test measure one literature-review setup, or a broader citation pattern? Because the NUS and Xi'an Jiaotong arXiv paper compares overlap, its findings are useful but still leave interface-pipeline questions open.
When API results may diverge from the ChatGPT interface
API results may diverge from the ChatGPT interface, and YouTube reports that this can distort citation tracking.
- Timing and surface differences: Divergence is most plausible when the same literature-review prompt passes through different product layers or runs at different moments. The NUS and Xi'an Jiaotong arXiv paper compared those surfaces, so API logs alone may miss what the interface shows.
- Timing checks: API and interface runs may not line up exactly, especially if citation assembly happens in later interface steps. That means a mismatch can reflect different output paths or timing windows, not necessarily a different underlying source pool.
- Timing and layers: Matching prompts can still produce different source lists when the ChatGPT interface adds retrieval, ranking, or formatting steps beyond raw API output. That tradeoff limits one-to-one tracking, especially when testing snapshots are not captured at the same moment.
What this means for agencies tracking AI visibility and mentions
For agencies, the main takeaway is simple: mention tracking is not the same as citation validation. A brand, paper, or publisher may appear in AI output without a reliable source trail behind it. TrueStandard says language models can invent studies, DOIs, and cases with the same confident tone they use for real material, and its roundup places fake-source rates between 18 percent and more than 50 percent, depending on the model.
That turns raw visibility data into a risk signal, not a clean measure of earned authority. The cost can be real: TrueStandard cites the 2023 Mata v. Avianca case, where fabricated cases helped trigger a $5,000 sanction.
For reporting, that means logs should separate mention counts from verified citations, and reserve manual checks for high-stakes mentions.
ChatGPT’s citation gap with API output appears real, but only in a limited sense. The NUS and Xi'an Jiaotong literature-review comparison points to partial overlap, not proof of favoritism, intent, or source quality.
LinkedIn commentary from Jiahao Chen on University of Sheffield research adds a related caution: cited papers can still be retracted or otherwise unreliable. For agencies and marketers, the practical takeaway is straightforward.
Treat citation visibility as a useful signal, but not as a clean measure of trust, preference, or verified authority without separate checks.