SLTinfo logo

Entropy and Redundancy in Human Communication

Improving communication

The encode-decode model of communication implies that, in order to improve communication two influencing factors should be managed:

  1. noise (physical and semantic) should be reduced
  2. capacity (sender-receiver and channel) should be increased

In addition, to more effectively transmit a message there should be an appropriate balance between so-called entropy and redundancy. Shannon (1948) – whose work was highly influential in developing so-called transmission models of communication – discusses these terms in an explicitly defined, mathematical way. However, the terms have come to mean something like the following when applied to human communication:

  • entropy: refers to messages which convey highly unpredictable information to the receiver
  • redundancy: refers to messages which convey highly predictable information to the receiver

The proportions of entropy and redundancy in messages will differ for different speaker-hearers. The relative amounts will be influenced by such things as the type of information conveyed (e.g. highly complex, simple and straightforward), the form in which it is transmitted (e.g. spoken, written), the amount and type of knowledge that can be assumed to be shared by the speaker-hearers, their cognitive levels, and so on.

Entropy too high

Entropy too high

Sperber and Wilson (1995) claim that listeners/receivers proceed on a principle of least effort. That is to say, when attempting to understand what someone is saying, listeners try to balance the effort required to activate any background knowledge needed to interpret the message with resulting cognitive benefits. The cognitive benefit for the listener may simply be straightforward understanding of the message. Alternatively, benefits could be effects of enjoyment, mental enrichment, stimulation, and so on. Typically, listeners seek to invest minimum effort for maximum cognitive effects. If entropy is too high (i.e. there are too many highly unpredictable messages), then the effort to decode the messages becomes too onerous. The receiver will at best only partially decode the messages and, at worst, not decode them at all. Under this scenario, the amount of entropy has exceeded the receiver’s capacity to decode the messages.

Redundancy too high

Redundancy too high

If redundancy is too high (i.e. there are too many highly predictable messages) then this requires almost no effort to decode the messages, as there is little in the way of information that needs to be decoded. Listeners may expend little interpretative effort and, consequently, may overlook particular intended meanings or inferences.

A word of caution

As with most models, there are problems with the encode-decode model of communication. Sometimes concepts from one field of study appear to be over-extended when applied to another. For example, some researchers (e.g. Lin, 1971; Lawyer, 2001) have attempted to describe differences between world languages in terms of the relative amounts of redundancy and entropy in both their spoken and written modes. I am not overly convinced by the arguments, however, as this seems to me to be an incongruous application of the concept of entropy – which was originally explicitly applied as a measure of disorder in a physical system. [For the technically minded, entropy = (Boltzmann’s constant k) x logarithm of number of possible states = k log(N).] Nevertheless, transmission models do offer a useful way of beginning to think about how humans communicate and what might interfere with successful communication.

References

Lin, N. (1973) The Study of Human Communication Indianapolis: The Bobbs-Merrill Company.

Lawyer, G. (2001) Entropy, Redundancy, and Communication SpråkTeknologi Höst [WWW] http://www.nada.kth.se/kurser/kth/2D1418/uppsatser01/Entropy.htm (Accessed 04 April 2011).

Shannon, C. (1948) ‘A mathematical theory of communication’ Bell System Technical Journal Vol. 27, pp. 379–423, 623–656, July, October, 1948. [The paper has appeared in a number of republications since. You can download a pdf version of the paper from Bell Labs at http://cm.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf (Accessed 04 April 2011).]

Sperber, D. and Wilson, D. (1995) Relevance (2nd edn) Oxford, UK: Blackwell.