Type-token ratio of written language

Take a look at Text 1 below, which is an extract from something I wrote a while ago.

If we count the number of words we get a total of 87. The number of words in a text is often referred to as the number of tokens. However, several of these tokens are repeated. For example, the token again occurs two times, the token are occurs three times, and the token and occurs five times. The following table shows all the tokens in Text 1, together with their frequency of occurrence.

We see, then, that of the total of 87 tokens in this text there are 62 so-called types. The relationship between the number of types and the number of tokens is known as the type-token ratio (TTR). For Text 1 above we can now calculate this as follows:

The more types there are in comparison to the number of tokens, then the more varied is the vocabulary, i.e. it there is greater lexical variety.

Type-token ratio of speech

Now take a look at Text 2. This is an extract from a transcribed conversation between two people, P and A.

As before, I have set out the tokens and their frequency of occurrence in tabular format in the Table below (I have ignored pauses such as (…), (2.1), the repetition of initial /w/ in line 10, and the inserts er, oh and mm).

We can now calculate the type-token ratio as before:

Interpretation

You will see that the number of tokens in each of the texts is almost the same (87 in Text 1 and 88 in Text 2). However, the type-token ratios are different: 71% for the written text (Text 1) and just 51% for the spoken text (Text 2). We can say, therefore, that the vocabulary is less varied in the spoken text than in the written text. Or, to put it another way, the written text shows greater lexical variety. A high TTR indicates a large amount of lexical variation and a low TTR indicates relatively little lexical variation. This finding, that the type-token ratio of speech is less than that of written language, is typical.

A major difference between speech and written language is that speech in conversation is produced in real time. There is limited time to think about, and plan, what one wishes to say. Consequently, speakers tend to select words from a relatively restricted vocabulary. In contrast, an author of a written text has much more time to plan and select just the right vocabulary items that best communicate his or her meaning.

As with lexical density, the type-token ratio can also be used to monitor changes in the use of vocabulary items in children with under-developed vocabulary and/or word finding difficulties and, for example, in adults who have suffered a stroke and who consequently exhibit word retrieval difficulties and naming difficulties.

Reference

A few people have contacted me to enquire about a reference for TTR in order to include it in a report, a written assignment, or similar. Unfortunately, there is no reference for TTR as such. It is a well-known measure of lexical variation which is used in many linguistic analyses. If you search the internet for ‘type token ratio’ you will find several of these. I do not know who was the first person to use a measure of  TTR in a study but  (rather like lexical density) it is now well-known and, as it is in the public domain, no one really references its use anymore in articles, reports, and so on.

However, the book that I often refer to for definitions is:

• Biber, D., Conrad, S. and Leech, G. (2002) The Longman Student Grammar of Spoken and Written English Harlow: Longman. [ISBN: 0 582 237262]. I have found this to be a useful reference text, as it is a corpus-based reference work, i.e. the findings are based on an analysis of real world written and spoken texts.