The mystery of Zipf
In our recent Plus article Tasty maths, we introduced Zipf's law. Zipf's law arose out of an analysis of language by linguist George Kingsley Zipf, who theorised that given a large body of language (that is, a long book — or every word uttered by Plus employees during the day), the frequency of each word is close to inversely proportional to its rank in the frequency table. That is: $ P_ n \propto 1/n^ a $ where a is close to 1. This is known as a "power law" and suggests that the most frequent word will occur approximately twice as often as the second most frequent word, which occurs twice as often as the fourth most frequent word, etc. A famous study of the Brown Corpus found that its words accorded to Zipf's law quite well, with "the" being the most frequently occurring word (accounting for nearly 7% of all word occurrences — 69,971 out of slightly over 1 million), and "of" the second most frequent (3.5% of all words).
language, maths saved by: davidar
http://plus.maths.org/latestnews/may-aug08/Zipf
In our recent Plus article Tasty maths, we introduced Zipf's law. Zipf's law arose out of an analysis of language by linguist George Kingsley Zipf, who theorised that given a large body of language (that is, a long book — or every word uttered by Plus employees during the day), the frequency of each word is close to inversely proportional to its rank in the frequency table. That is: $ P_ n \propto 1/n^ a $ where a is close to 1. This is known as a "power law" and suggests that the most frequent word will occur approximately twice as often as the second most frequent word, which occurs twice as often as the fourth most frequent word, etc. A famous study of the Brown Corpus found that its words accorded to Zipf's law quite well, with "the" being the most frequently occurring word (accounting for nearly 7% of all word occurrences — 69,971 out of slightly over 1 million), and "of" the second most frequent (3.5% of all words).
language, maths saved by: davidar
http://plus.maths.org/latestnews/may-aug08/Zipf
More Bookmarks From This Site:
Machine prose
Given a piece of text in any language, the program called ADIOS - automatic distillation of structure - searches for patterns and structures which it then generalises to produce new and m...
language, maths saved by: davidar more details ...
http://plus.maths.org/latestnews/may-aug05/adios
Given a piece of text in any language, the program called ADIOS - automatic distillation of structure - searches for patterns and structures which it then generalises to produce new and m...
language, maths saved by: davidar more details ...
http://plus.maths.org/latestnews/may-aug05/adios
Speechless maths
Here you are, reading an article in a magazine about mathematics. This shows that know how to read and that you are interested in maths. But could you still do maths even if you could not...
language, maths saved by: davidar more details ...
http://plus.maths.org/latestnews/jan-apr05/speechless
Here you are, reading an article in a magazine about mathematics. This shows that know how to read and that you are interested in maths. But could you still do maths even if you could not...
language, maths saved by: davidar more details ...
http://plus.maths.org/latestnews/jan-apr05/speechless
Evolutionary maths
What is it that makes the human mind so unique and us humans so different from the other species we share this planet with? One thing that is universally present throughout human cultures...
language, maths saved by: davidar more details ...
http://plus.maths.org/issue44/features/varley
What is it that makes the human mind so unique and us humans so different from the other species we share this planet with? One thing that is universally present throughout human cultures...
language, maths saved by: davidar more details ...
http://plus.maths.org/issue44/features/varley
Mysterious Number 6174
math saved by: btocher more details ...
http://plus.maths.org/issue38/features/nishiyama
math saved by: btocher more details ...
http://plus.maths.org/issue38/features/nishiyama