Shannon's Linguistic Playground
The Shannon Text Generator is an interactive demonstration of an idea from Claude Shannon’s 1948 paper “A Mathematical Theory of Communication.” Shannon approximated English by a series of ever finer statistical models and let each of them produce random text. These Markov models are an early ancestor of statistical language modeling. Today’s large language models have moved far beyond them, but they still rest on the same idea: predict the next symbol from the ones before it. With this tool you can repeat Shannon’s experiment on any text and watch the output turn from letter salad into almost readable prose as the context grows.
Open the Shannon Text Generator
How it works
The generator reads the source text and builds a table: for every sequence of k characters that occurs in the text, it records which characters follow it and how often. The number k is the context length, also called the order of the Markov chain. Generation starts from a sequence of k characters, chosen at random from the beginnings of sentences or typed in by you. The next character is drawn at random from the characters that followed this sequence in the source text, so frequent successors are drawn more often. The window then moves one character forward and the step repeats until the requested length is reached and a sentence ends.
A model with context length k uses the statistics of (k+1)-grams. So k = 1 corresponds to bigram statistics and to what Shannon calls the second-order approximation, k = 2 to trigram statistics and his third-order approximation, and so on. With k = 0 the characters are drawn independently according to their frequencies in the source, Shannon’s first-order approximation.
Example 1
Let’s take the text of Little Red Riding Hood (also available in the generator as sample text).
k = 1 (bigram statistics): Hop Hoed Rig, Gof Alfr HOOOOD OOnoood he cl," “Whthe thend, Ricry d Shodorer.” ITLI wid y Re t sat tlood sme, “Wod kerandmod angrs Litll Rith s ithe, Alys s yoe u t in wid dmoosche peasthe he atersthaf m y idmod?
k = 2 (trigram statistics): All the he wice Riding Hood?” In and Red who shen spre litter alk a pas pas then Mr. He she Rid flothers of the mad caloaked Rid.
k = 3 (4-gram statistics): Everybody lived Riding Hood. In andmothe bed Riding to see you goings. Once and cake and crept under mout in, so picked ther.
k = 4 (5-gram statistics): Four Everybody loved had dropped at the village. “Little Red Riding Hood’s father lay in throught and gave brough then the grandmother grandmother.
k = 5 (6-gram statistics): At last he reached this basket, Little Red Riding Hood?” “I am going with a hood was so happily ever after.
Example 2
Let’s take the most popular baby names in Austria of 2010 (format boys / girls).
k = 1 (bigram statistics): Lon, Do, Pamobantieons, Liavid, Chameonuiaur, Chan, Olikterchas, Hast, Stz, Dat, Hel, Erdorichian, Lested, Anjan / Soph, Thilerina, Ma, Misa, Sohisa, Ole, Nia, Gria, Aliare, Ela, Idra, Che, Miesa, Soh, Olianna, Ma, Ma, Minzia, Lela.
k = 2 (trigram statistics): Luca, Antincenry, Marco, Dammed, Arthael, Lian, Louistias, Hugo, Car, Domastian, Chritz, Man, Sebas, Sebas, Fabias / Sophie, Sophie, Melia, Marah, Viktonie, Julia, Lena, Vanna, Johanna, Daria, Sophie, Leonia, Eda, Rosa, Antorie.
k = 3 (4-gram statistics): Lukas, Matthias, Niklas, Henry, Manuel, Liam, Simon, Jonardo, Oliver, Felix, Julian, Elian, Tim, Hugo, Damian, Erik / Sophie, Hannah, Clara, Melisa, Melina, Amelia, Sophie, Isabeth, Viktoria, Hanna, Stella, Pia, Melina, Lisabeth.
