How Many Guesses Does It Take to Crack an AI's "Random" Number?
AI Summary: The article quantifies the bias of language models toward specific numbers when asked to pick a random integer between 1 and 100 by extracting exact probability distributions from model logits, avoiding sampling noise. It compares base models with their instruction‑tuned counterparts across five open model families and two prompt formats, finding that a 1.5‑billion‑parameter instruction‑tuned model selects the number 47 in roughly 75 % of cases, and still does so over one‑third of the time with a reworded prompt. By converting the probability vectors into bits of randomness and expected guess counts, the study shows that these biases dramatically reduce the security of secret selections. The work also highlights the reproducibility advantages of open checkpoints and documents how post‑training stages, particularly RLHF, amplify the concentration of probability on favorite numbers.