What an AI detector score actually measures

Australia’s higher-education regulator puts the problem in two sentences, and they are worth reading before any argument about accuracy. A detector score is not the probability that a text was machine-written. If a tool has a one per cent false-positive rate and a hundred students in a class all wrote their own work, one of those hundred will still come back with a high score — and the real probability that it was machine-written is zero.

That is a statement about what kind of quantity the number is, not about whether the tool is any good. The two get confused constantly, and the confusion runs in both directions.

The tools are better than 2023

The most-cited paper on this found that seven detectors flagged sixty-one per cent of TOEFL essays written by human non-native speakers, against five per cent for native writers, and traced the mechanism to perplexity: less lexical variation reads as machine-like. It was published in 2023 and it is still quoted as though it described the present.

It does not. A 2026 replication on the same TOEFL data measured twenty-three per cent with a current commercial tool, and found no measurable bias at all in Czech, with almost no correlation between detector output and entropy. A peer-reviewed evaluation the same year found no false positives at all for three of four tools on forty human theses. Anyone writing that detectors simply do not work is arguing against a version of the field that has moved.

Two caveats keep that honest. Twenty-three points of gap is still a gap. And forty documents is a small sample: no false positives in forty leaves the true rate anywhere below roughly seven per cent.

But it is still the wrong quantity

Accuracy improving does not turn a score into a probability of authorship, and three results explain why not.

• On a single passage, the best achievable performance is bounded by how far the machine and human distributions actually differ. As models get closer to human writing, that ceiling falls, and no cleverer detector raises it.
• You can get past that ceiling by aggregating many texts — but the number of samples required grows as the square of the shrinking gap. That is a statement about corpora, not about the page in front of you.
• Comparisons between tools are meaningless without a common operating point. Fixed at a five per cent false-positive rate, one shared benchmark found paraphrase barely dented some detectors while synonym substitution cost another thirty-six points, and one popular tool could not be tuned below a seventeen per cent false-positive rate at all.

The most direct evidence that these tools measure register rather than origin is what they do with a base model. In a preprint from earlier this year, text generated by a model that had never been instruction-tuned — machine-written by any definition — was confidently rated human by current detectors, while the aligned version of the same model was flagged. The instrument appears to be reading the assistant, not the machine.

Three things that get called detection

A post-hoc detector infers from the text alone. A provider watermark is placed at generation: SynthID-Text biases which token the model picks, and verifying it needs the tokenised text and the key. That makes it strong evidence about that provider’s output and no evidence at all about anyone else’s — and it is not a tool the public holds. A watermark gives a detector to the vendor, not to the reader.

The third is a reading of register, and it is genuinely measurable — at the level of a corpus. Analysis of fifteen million PubMed abstracts showed a sharp excess of a handful of words after 2022 and concluded that at least thirteen and a half per cent of 2024 abstracts had been through a model. The authors state the limit in the paper: the analysis is performed at the corpus level and cannot identify an individual abstract.

What our own score inherits

This is the part an article like this usually skips. Our score is a deterministic count of matched phrasings per hundred words against a taxonomy we publish, and it makes no claim about authorship — which is why it is capped rather than allowed to reach a hundred: the absence of a known habit does not prove a person wrote something.

But it runs on the same axis the detectors do. Corpus linguistics has documented for years that writers working in a second language use fewer and less varied lexical bundles. A score that penalises stock openers, hedges and filler phrases will therefore mark a non-native writer down more often, for the same reason the 2023 detectors did. We do not get to point at that finding and exempt ourselves from it.

What differs is what the number is allowed to do. Ours is not evidence of anything, it accuses nobody, and every mark it makes is a phrase you can look at and disagree with. A detector score arrives as a verdict with no working shown; a register score is a list of edits you are free to reject. That is a difference in accountability, not in statistics, and it is the only one we claim.

What the regulation actually requires

Article 50 of the EU AI Act, applicable since 2 August 2026, requires providers of systems that generate synthetic text to mark their outputs in a machine-readable, detectable format. The obligation sits with the producer, at generation. Deployers publishing AI-generated text on matters of public interest must disclose it, with an exemption where a person holds editorial responsibility.

Nothing in it requires, recognises or gives weight to third-party post-hoc detection. The techniques the regulation contemplates — watermarks, metadata, cryptographic provenance, logging, fingerprints — are all applied by whoever produced the content. Where the text says “detection”, it means reading a mark that was deliberately placed, not guessing at origin after the fact.

Which is the practical conclusion. If provenance matters to you, the answer is a signal placed at generation, or a record of who edited what. It is not a number derived from the finished prose, however good that number gets.