How AI text detection works, and when it is wrong

Language models write with unusually predictable word choices and an even sentence rhythm, and they lean on a recognisable set of phrases. A detector measures these properties against a baseline of human writing for the same kind of document.

• Perplexity: how surprising each word is given the words before it. Machine prose is rarely surprising.
• Burstiness: how much sentence length and structure vary. People vary; models flatten.
• Marker phrases: constructions that models over-produce ("it is important to note", "delve into", "pivotal role").
• Style breaks: a passage whose statistics differ from the rest of the same author’s document.

The output is a probability that the span was machine-written. It is not a fact about the author, and it is never reported by ProofVerity as a single score for a whole document.

Most real documents are mixed: a human draft with a machine-polished paragraph, or a machine draft with human edits. A whole-document score hides that mixture and cannot be discussed. A sentence-level flag names the exact passage, which lets an author, a student or a source respond to something concrete.

Sentence-level detection also lets ProofVerity compare a passage with the author’s own baseline in the same document. A sudden change in rhythm and vocabulary is evidence in itself, and it is shown as such.

A 94 means 94: among all spans ProofVerity marks at 94%, about 94 in 100 were machine-written in evaluation. Calibration is done separately for each document type, because a lab report, a news brief and an admissions essay have different baselines, and a detector tuned on one will misfire on the others.

Every report states the false-positive rate for its document type next to the probability. The rate is the share of human-written spans that the detector would wrongly mark at that threshold. It is published so that a reader can decide how much weight a flag deserves.

All AI detectors make mistakes, in both directions. The situations below are where false positives concentrate, and what ProofVerity does about them.

• Non-native and formulaic writing reads "flat" to a naive detector. ProofVerity calibrates per genre and discounts short spans, and it never reports a whole-document verdict.
• Templates and boilerplate (methods sections, legal wording, lab protocols) are predictable by design. Style-break detection against the author’s own text separates a template from a machine paragraph.
• Heavily edited machine text keeps some rhythm signals but loses marker phrases; confidence drops and the report says so.
• Paraphrasing tools change words but rarely restore human variation, and they cannot make an invented reference exist. The citation check catches what the text check misses.
• Translated text carries the statistics of the translation engine. Reports note when a document appears to be a translation.

1. Look at the span, not the document. Which sentences are marked, and how many.
2. Read the evidence lines: perplexity against the human median, burstiness, marker phrases, style break.
3. Check the false-positive rate for this document type.
4. Look at the citations. Invented or misquoted references beside a flagged paragraph change the picture; verified ones do too.
5. Ask before you accuse. Drafts, notes and a short conversation settle most cases.
6. Record the decision: mark reviewed or dismiss, add a note, export the annotated PDF with its evidence appendix.

• It does not issue verdicts or "AI scores" for whole documents.
• It does not compare a text with other students’ work; it is not a plagiarism checker.
• It does not train on your documents, and it deletes them after 30 days.
• It does not hide its reasoning. Every flag shows the signals that produced it.