How to Verify an LLM API Serves the Model It Claims

Ask a language model to pick a random number between 1 and 100 enough times and something strange happens: the answers are not random. One model keeps landing on 42 and 73. Another favors 47 and 57. The pattern is stable, reproducible, and different for every model.

A July 2026 paper turned that quirk into a verification method. One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions shows that these answer distributions form a reliable behavioral fingerprint, enough to check from the outside whether an API endpoint is serving the model it claims^[1]. No weights, no logits, no insider access. A few hundred one-word answers.

This guide explains why a model field is a claim rather than a guarantee, how single-token fingerprinting works, what the numbers mean, and where the method's limits are.

TL;DR

• The model field in an API response is unverifiable by the protocol. Nothing proves your request for a flagship model was served by it^[1].
• Language models cannot produce uniform randomness. Across the paper's probe set, the median answer distribution carries about 1.0 bit of entropy against the 6.64 bits a uniform 1-to-100 pick would have^[1].
• That failure is consistent, so it works as a signature. Same model, same skew, every time.
• The scale has usable daylight: same model against itself lands near JSD 0.14, same model across two providers near 0.23, two genuinely different models near 0.46^[1].
• Accuracy is strong, not perfect. Equal error rate is 10.6% at 8 probe units and 7.3% at the full 40, with an AUC of 0.971^[1].
• A mismatch is evidence, not proof. Quantization, a silent version update, or a hidden system prompt can all shift a distribution without anyone lying.

You cannot see what is behind an API

When you call a chat-completions endpoint you send text and get text back. The model field in the response says whatever the server chooses to put there. Nothing in the protocol proves the request was served by the model named in it, rather than by something a tenth the price^[1].

That gap matters more as more of the market sits behind intermediaries: aggregators reselling hundreds of models through one endpoint, regional resellers offering flagship access below official pricing, and third-party hosts serving open-weight models with quantization and serving-stack choices you never see^[1].

Most of these businesses are legitimate. The incentive to cheat is still obvious, and a separate audit of 17 shadow-API operators found several endpoints that did not pass verification against the models being advertised^[2].

It is also not only about fraud. A provider can quantize a model to cut serving costs, roll out a silent version update, or route traffic across mixed backends. If your product's quality depends on a specific model, "what am I actually getting?" should be answerable with evidence rather than trust.

Why random numbers give the game away

The method rests on a well-documented weakness. Language models do not compute, they predict, so when asked for a random number they reproduce the biases of their training data and preference tuning^[1].

Three clusters dominate:

• 42 is massively over-represented, the Hitchhiker's Guide answer echoing through decades of internet text.
• 7 carries centuries of cultural weight as the lucky pick humans reach for.
• 37, 47, 73 and other two-digit primes feel random to humans, so they dominate the human-generated "random" examples the model learned from.

The paper quantifies the collapse: the median answer distribution carries roughly 1.0 bit of entropy, where a fair pick from 1 to 100 would carry 6.64 bits^[1]. Where a fair die has a hundred faces, most models behave like a slightly weighted coin.

The key move is that the failure is consistent. The same model produces the same skewed distribution every time, and different models, including sibling versions in one family, produce measurably different ones. A bug becomes a signature.

The protocol, in four steps

1. Probe. Ask the endpoint a battery of one-word questions: pick a random number between 1 and 100, name a random color, flip a coin. The paper uses 10 tasks across 4 languages for 40 probe units, sampling each 30 times at temperature 1.0 with max_tokens=16 and reasoning disabled^[1].

2. Fingerprint. For each probe unit, tally the answers into an empirical distribution. The collection of those distributions is the endpoint's behavioral fingerprint.

3. Compare. Measure the distance between that fingerprint and a trusted reference for the claimed model, using Jensen-Shannon divergence in base 2, so the scale runs from 0 (identical) to 1 (disjoint).

4. Decide. Small distance means consistent with the claim. Large distance means the endpoint is behaviorally a different animal.

Two properties make this hard to game. It needs no special access, since anything answering chat completions can be fingerprinted. And there is no magic string to filter, because every probe is an ordinary harmless question drawn from paraphrase pools, so a dishonest middlebox cannot special-case the test without breaking normal traffic^[1].

Reading the distance number

A divergence value means nothing without the reference points, and this is where the method becomes practical.

There is real daylight between "same" and "different"^[1]. Note what the middle row implies: even an honest deployment of the same model by a different host drifts measurably, because quantization, serving stack, and hidden system prompts all leave marks.

Accuracy scales with probe count. At 8 probe units the equal error rate is 10.6%; at the full 40 it falls to 7.3%, with an AUC of 0.971^[1].

What the paper found in the wild

The palmyra-x5 case. The paper's most striking result concerns a model offered as a proprietary flagship whose fingerprint sat at a JSD of 0.141 from an open-weight 235B model, statistically indistinguishable from the ~0.140 you get comparing a model against itself. Behaviorally, the paper concludes, the endpoint was serving something functionally identical to the open-source model^[1].

Same model, different providers, sometimes suspiciously different. Of 34 same-model pairs served across different providers, 10 diverged beyond the 5th percentile of the impostor distribution^[1]. Some official-model third-party deployments drift far enough to look like different models. Verification is not paranoia even when nobody is lying about the name.

The research artifacts are open: the fingerprint dataset is published under CC-BY-4.0 and the reproduction code under MIT, both on Zenodo^[3][4].

Running the check yourself

The protocol is simple enough to implement directly. The shape of it:

import collections, math
from openai import OpenAI

client = OpenAI(api_key="...", base_url="https://your-endpoint/v1")

def probe(model, prompt, n=25):
    counts = collections.Counter()
    for _ in range(n):
        r = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": prompt}],
            temperature=1.0,
            max_tokens=16,
        )
        counts[r.choices[0].message.content.strip()] += 1
    total = sum(counts.values())
    return {k: v / total for k, v in counts.items()}

def jsd(p, q):
    keys = set(p) | set(q)
    m = {k: 0.5 * (p.get(k, 0) + q.get(k, 0)) for k in keys}
    def kl(a):
        return sum(a[k] * math.log2(a[k] / m[k]) for k in a if a[k] > 0)
    return 0.5 * kl(p) + 0.5 * kl(q)

Four practical notes on doing this properly.

Hold the sampling conditions fixed. Temperature 1.0, a small max_tokens, reasoning disabled. A fingerprint collected under different settings is not comparable to one collected under the paper's.

Use more than one probe. A single question is noisy. The error rate roughly halves going from 8 probe units to 40.

Compare against a reference collected the same way. The cleanest reference is the official endpoint for the same model, fingerprinted in the same session under identical settings, rather than a published table gathered months earlier.

Watch the auxiliary signals too. A fingerprint that disagrees with its own split halves suggests multi-backend routing. One-word answers that bill hundreds of completion tokens suggest padding. Inflated prompt-token counts suggest a large hidden system prompt.

A standard check of 8 probe units at 25 samples is about 200 tiny requests, which costs a fraction of a cent on a small model.

What a result does and does not mean

Be precise about the claims this method supports.

A mismatch is evidence, not proof. The method has an inherent error rate, around 10.6% EER at 8 probe units. Aggressive quantization, a silent model update, a stale reference, or a serving-side system prompt can all shift distributions with no intent to deceive. Treat a red result as a reason to re-run with more probes, test a second reference, and ask questions^[1].

A match is strong but not absolute. A sophisticated impostor could in principle mimic another model's distributions, though doing so across dozens of paraphrased multilingual probes while serving normal traffic correctly is harder than it sounds.

Reasoning models need care. Fingerprints are collected with reasoning disabled. Where an endpoint cannot disable it, confidence drops.

A distance is a property of an endpoint at a moment in time, not a verdict on a business.

FAQ

What is LLM fingerprinting?

Measuring the distribution of a model's answers to a battery of one-word questions, then comparing that distribution against a reference for the model an endpoint claims to serve^[1].

Why can't language models produce random numbers?

They predict rather than compute, so a request for randomness returns the biases of training data and preference tuning. Culturally loaded values like 42 and 7 dominate, collapsing the distribution to about 1.0 bit of entropy against a uniform ideal of 6.64^[1].

How much of a difference counts as a mismatch?

Use the paper's baselines rather than a fixed threshold: about 0.14 for a model against itself, 0.23 for the same model across providers, and 0.46 for genuinely different models^[1].

How many requests does a check need?

The paper's full protocol is 40 probe units sampled 30 times each. A lighter 8-unit check at 25 samples is roughly 200 requests and raises the equal error rate from 7.3% to 10.6%^[1].

Can a provider detect and defeat the test?

Not easily. Probes are ordinary harmless questions drawn from paraphrase pools, so special-casing them without breaking normal traffic is difficult^[1].

Does a failed check mean a provider is cheating?

No. Quantization, silent version updates, hidden system prompts, and stale references all produce drift without deception. A distance is a statistical observation warranting closer inspection.

Is the dataset available?

Yes. The fingerprint dataset is on Zenodo under CC-BY-4.0 and the reproduction code under MIT^[3][4].

Treating the model field as a claim

The useful shift here is small and specific. The model field in an API response is a claim, and there is now a cheap, open, statistically grounded way to check that claim from the outside, built on nothing more exotic than the fact that language models cannot say a random number to save their lives.

If you buy capacity through any intermediary, the right place for this is next to uptime monitoring: a periodic check against a reference you collected yourself, with the auxiliary signals watched alongside the headline distance. And the right way to read a red result is as the beginning of a conversation, not the end of one. To verify an LLM API is to gather evidence about an endpoint at a moment in time, which is worth doing precisely because the alternative is assuming.

References

1. Bruckner, Tomáš. One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions. arXiv, July 2026. arxiv.org/abs/2607.10252
2. CISPA researchers. Real Money, Fake Models — an audit of shadow LLM API operators. arXiv. arxiv.org/abs/2603.01919
3. LLM fingerprint dataset (models × tasks × languages). Zenodo, CC-BY-4.0. zenodo.org
4. Reproduction code for One Token Is Enough. Zenodo, MIT License. zenodo.org

Further reading

• reAPI. How to use Claude Opus 5. reapi.ai/blog/how-to-use-claude-opus-5
• reAPI. How to use GPT-5.6. reapi.ai/blog/how-to-use-gpt-5-6