How this was calibrated

And how often it is wrong, including the number most detectors do not publish.

Why this page exists

Every tool in this category reports on other people's writing. Almost none of them report on themselves. The number that matters for a tool like this is not overall accuracy, it is the rate at which it flags human writing as machine written, and specifically the rate at which it does that to people writing English as a second language. That population absorbs most of the harm detectors cause, so it gets its own column below.

The rule applied here: any signal whose false-positive rate on non-native writing exceeds 25% is dropped or downweighted, however well it separates the corpora otherwise.

The method

Three corpora, one sample per file, 300 to 1,500 words each. Human writing from pre-2020 blog posts, forum posts, personal essays and Wikipedia talk pages, chosen from before generated text was common enough to contaminate the pool. A second human set written by people whose first language is not English. An AI set covering several models and several prompt styles, including formal, casual, "write like a human", and output edited by hand afterwards.
Run the analyzer over all three and record which signals fire on each sample, plus the score.
Report per signal: true-positive rate, false-positive rate, and the false-positive rate on the non-native subset alone.
Drop or downweight anything over the non-native limit.
Set the band thresholds where the two score distributions actually separate, rather than at round numbers.

The script is scripts/calibrate.mjs in the source. It reads from corpus/human, corpus/human-nonnative and corpus/ai, and writes the JSON this page renders. Anyone with a corpus can reproduce or contradict what is below.

Current status: not yet calibrated

The weights currently shipping are untested priors. They were set from the published literature and from reading a lot of generated text, not from measurement against a corpus. No calibration run has been published yet, so this tool cannot tell you its own false-positive rate.

That is a real limitation and it is stated here rather than buried. Until the corpora are assembled and this page fills in with measured numbers, treat the score as a rough index over the evidence underneath it, and treat the evidence as the actual output.

The pipeline that produces these numbers is written and working. What it needs is the corpus, and assembling one honestly, particularly the non-native subset, is the slow part. Fabricating the numbers in the meantime would defeat the point of the page.

What this cannot tell you, regardless of calibration

Whether a specific piece of text was written by a person. No amount of tuning changes that, because the properties being measured are properties that careful human writing also has. A high score on an edited, formal, well-proofread document is the expected outcome, not a failure.

It also cannot survive an editor. Swapping the em dashes, varying the sentence lengths and cutting the stock phrases will move a score from the heavy band to the few band in about ten minutes. Anything measuring surface style has that property.

Use it to find out what your writing looks like from the outside, or to check your own drafts for tics. Do not use it as evidence against a person.