Can software or people reliably tell AI writing from human writing?

Turnitin sells an AI writing detector to schools. Its current instructor FAQ says it aims to keep its false-positive rate “under 1% for documents with over 20% of AI writing.”⁹ The FAQ explains that figure as possibly flagging one in every 100 fully human-written documents.⁹ Annie Chechitelli, Turnitin's chief product officer, gave the same document-level figure on June 14, 2023. She also gave a sentence-level false-positive rate of around 4%.⁷ Workado, another vendor, marketed its AI content detector on accuracy. The Federal Trade Commission challenged that marketing on April 28, 2025.⁵ Alongside the software sits a common belief that a careful reader can recognise machine prose by its tells.

The 2023 and vendor evidence does not support reliable detection by software in general, although one 2025 study found a commercial detector, Pangram, matched the best human readers on 300 English news-style articles. OpenAI withdrew its own classifier on July 20, 2023, citing low accuracy.¹ Independent tests in 2023 found high error rates on essays by non-native English writers and on AI text that had been edited.²,³ In a 2025 study, four readers who rarely or never use LLMs for writing performed close to chance. A selected group of five heavy ChatGPT users performed much better.⁴ Turnitin itself tells instructors not to use its score as the sole basis for adverse action against a student.⁸

What do the vendors say about their own tools?

OpenAI released a classifier for AI-written text on January 31, 2023. It reported the results on an English challenge set. The classifier correctly identified 26% of AI-written text as “likely AI-written.” It labelled human-written text as AI-written 9% of the time.¹ OpenAI later added a notice to the page saying the tool had been withdrawn on July 20, 2023, because of its low accuracy.¹ Those figures describe OpenAI's own tool and no other vendor's.

Turnitin's under-1% false-positive figure carries a condition. It applies to documents with 20% or more AI writing, according to the June 2023 blog. The current FAQ says over 20%.⁷,⁹ The figure is not a rate for every human-written document. At the sentence level, the 2023 rate was around 4%.⁷ The FAQ also concedes that the detector may miss some AI text in order to hold its false-positive rate down.⁹ It publishes no false-positive rate for second-language writers. It says only that such writers were considered when the training sample was built.⁹ All of these figures are reported by the vendor. No independent audit of the current Turnitin model was found for this piece.

What did the FTC allege about one vendor's claim?

The FTC's complaint against Workado, LLC, formerly Content at Scale AI, was issued on April 28, 2025. It says the developers' best result for overall accuracy on a mix of human-created and AI-generated non-academic content was 74.5%.⁵ It says the same testing data showed the model correctly detecting AI-generated non-academic text “merely 53.2% of the time.”⁵ The FTC's April press release describes the 53 percent figure differently. The complaint's wording is used here.⁵

These are allegations issued with a consent agreement. They are not findings after a trial.⁵ The Commission approved a final order on August 28, 2025. The order bars Workado from advertising its detector's accuracy unless it has “competent and reliable evidence” that the product is as accurate as claimed.⁶ The order binds Workado alone. It sets no accuracy standard for detectors.⁶

What do independent tests show?

James Zou's group at Stanford ran essays through seven public detectors on March 15, 2023. The essays were 91 TOEFL essays written by non-native English speakers and 88 essays by US 8th graders. The detectors misclassified more than half of the TOEFL essays as AI-generated, with an average false-positive rate of 61.22%.² The authors call it a pilot study. Most of the detectors they tested used a GPT-2 backbone, so the results may not hold for current commercial products.²

Debora Weber-Wulff and colleagues tested 14 tools on 54 documents between March 7 and May 8, 2023. Their only generator was ChatGPT's February 13, 2023 model.³ When AI-generated text was machine-paraphrased, overall accuracy was 26%. Most of that text went undetected.³ On AI text that people had edited by hand, accuracy was 42%. The best tool reached 76% under the strictest of the study's scoring methods.³ The tools leaned toward calling text human-written. For that reason the paper reports few false positives on the unmodified human documents.³ Turnitin and PlagiarismCheck knew they were being tested.³ The OpenAI, Stanford and Weber-Wulff error rates are from 2023. Turnitin says its model has been updated since, and the other tools may have changed too.

Can people tell?

Jenna Russell, Marzena Karpinska and Mohit Iyyer had annotators judge 300 English non-fiction articles, each under 1,000 words, in five batches of 60. The four non-expert annotators judged only the first batch. Their paper appeared at ACL 2025.⁴ The four non-expert annotators, who rarely or never use LLMs for writing, labelled 56.7% of AI articles correctly on a 60-article batch. They wrongly flagged 52.5% of human articles, which is close to chance. They stayed confident in their judgments.⁴ Five annotators who frequently use ChatGPT for writing did far better.⁴ Those five were selected from a larger pool. The AI articles came from simple prompts. The authors say they cannot generalise beyond this domain.⁴ The record therefore does not show that people cannot detect AI text. It shows that ordinary readers in this test could not.

Are typos and casual language evidence of a human?

The same study bears on tells. Nonexperts believed humans are more likely than AI to write grammatically correct sentences. The experts knew the opposite: “humans actually make more grammatical errors.”⁴ One expert was fooled by o1-Pro output. The paper says that annotator relied too heavily on informality as a sign of human writing.⁴ The study did not test deliberately inserted typos against readers or software.⁴ No study in this record isolates typos or casual language as a variable. A 2025 study did test a prompt-based 'humanizer' meant to make AI text sound more human, and it lowered several detectors' scores, but it did not isolate typos. The record therefore cannot say whether the tactic protects a writer or simply degrades the writing.

Who bears a false positive?

In the Stanford test, the false positives fell on the non-native English writers. More than half of their essays were flagged.² Turnitin publishes no false-positive rate for that group.⁹ On August 16, 2023, Vanderbilt University applied Turnitin's stated 1% rate to the 75,000 papers it had submitted in 2022. It wrote that “around 750 student papers could have been incorrectly labeled.”¹⁰ That is the university's arithmetic, not a count of real accusations. Because Turnitin's figure is stated for documents with 20% or more AI writing (blog) or over 20% (FAQ), the calculation is an illustration and may not match the real number.⁹,¹⁰

What does the record say about acting on a score?

The plainest statement comes from the vendor. Turnitin's instructor guide says its model “may misidentify human-written, AI-generated, and AI-paraphrased text.” It says the score should not be the sole basis for adverse action against a student.⁸ An institution that disciplines someone on the score alone is using the tool in a way its maker has disclaimed. Vanderbilt disabled Turnitin's AI detector in August 2023 and gave its reasons in published guidance.¹⁰ The FTC's Workado order treats an accuracy claim as something a vendor must be able to prove.⁶ This piece did not review any employer, publisher or board policy, or any court docket, on a detector-based decision. What those bodies do with a score is outside this record.