An Audit Finds About 16 Percent of Cited Sources Look AI-Written
A 712-query audit of ChatGPT, Copilot, Gemini, and Perplexity says about 16% of unique cited sources were labeled AI-generated.
Direct answer
The audit submitted 712 English queries on politics, health, and the environment to ChatGPT, Copilot, Gemini, and Perplexity, collected 2,848 responses, and reports about 16% of unique cited sources labeled AI-generated after excluding the Possibly AI category.
The paper "Synthetic Sources?" audits four generative search engines: ChatGPT, Copilot, Gemini, and Perplexity. The sample is 712 English queries from Search Arena and Climate Q&A, split into politics (175, 24.5%), health (257, 36.1%), and the environment (280, 39.3%). Each query was submitted through the engine's user interface. Cited pages were then scored with Pangram, an AI-text detector. The authors say they have no ground truth for those labels, and they sampled 200 URLs classed as Likely AI or Highly Likely AI without finding an AI-use disclosure.
They collected 2,848 responses. 2,596 (91.2%) include in-text citations and 252 (8.8%) do not. Of the uncited responses, 120 are health queries, which the paper describes as 11.6% of health queries, against 8.2% of environment queries (92) and 5.7% of politics queries (40). They drop the "Possibly AI" Pangram category so that ambiguous scores do not inflate the AI-written share. After combining "Highly Likely AI" and "Likely AI," they report approximately 16% of unique cited sources as AI-generated, with environment queries showing the highest share of that labeled content.
Most flagged domains appear once
At domain level, 1,754 of 6,258 unique source web domains (28.0%) have at least one cited source labeled AI-generated. Among those 1,754 domains, 66.2% have one such source, 17.7% have two, and 16.0% have three or more. The paper treats that long tail as a design trade-off: a wider set of citations is harder to check for how the page was produced. The 16% figure is a detector label on fetched source text, not a count of pages that declared themselves synthetic.
Source: arXiv 2605.23684.
FAQ
- How often did a response have no citation?
- 252 of 2,848 responses, 8.8%, had no citations. The paper says 11.6% of health queries were in that group.
- How many domains had at least one AI-labeled source?
- 28.0% of 6,258 unique source domains, 1,754 domains, had at least one source labeled AI-generated.