Information Theory Research

By Tara Howes
 

Darnell, Donald.  "Information Theory: An Approach to Human Communication."  Approaches to Human Communication.  Ed. Richard Budd and Brent Ruben.  New York: Spartan, 1972.



Review of Information Theory

Information Theory began during World War II when two men,  Norbert Weiner and Claude E. Shannon, were concerned with the need for communication and the technical problems that arose because of the war.  Although both men's thinking were close to each other,  it is Shannon's work that takes the name "Information Theory."
Shannon's system contained 5 main parts, an information source, a transmitter, a channel, a receiver and a destination. Four other parts were later added. These are a message, the transmitted signal, the received signal, and a noise source. The information source is that which "produces a message or sequence of messages to be communicated to the receiving terminal." The tranmitter is that which "operates on the message in some way to produce a signal suitable for transmission over that channel." The channel is "merely the medium used to transmit that signal from sender to receiver. The receiver ordinarily performs the inverse operation of that done by the transmitter, reconstructing the message from the signal and the destination is the person (or thing) for whom the message is intended." The message is the thing chosen from many other choices. One way to look at the message is

 

 
 


"What one does is only one
                           Of the several things he might have done.
               One must know the things rejected
            To appreciate the one selected."




The message that is received by the destination person is not always the message that has been sent. This is because the noise source, which is anything that may affect the message while it is being sent, may change the message in some fundamental way. Therefore, it is important that the person receving the message must be able to distinguish the origianl message from the one that he or she has received, or the system has failed. From this we learn the most important part of information theory is "how to send a message through a communication system with maximum efficieny and a tolerable level of error or equivocation in the presense of noise." Information theory only deals with the "technical problem of getting symbols or signals from one place to another without distortion." It is not concerned directly with interpreting these messages.

 

 

          Information theory uses a logarithm to measure the amount of information being transmitted. The source transmits one bit of information, which stands for binary digit.  For example, if you are trying to guess a number that a friend is thinking of, each yes or no answer to the question you ask the person is one bit of information.



New Information on Information Theory
 

          Information in a mathematical sense has to do with surprise and uncertainty.  It is the amount of the things a system such as an information source can do and the probability of these occurences happening.  It is the opposite of predictability.   Entropy is another word for the unpredictability of information.

          One way to test the discrepancies in information is to perform a test called clozentropy.  It is a mix of a cloze procedure (a fill in the blanks type test) and information analysis.  Here's how it works.

          1. First select a block of text (song lyrics, a poem) and take out words from here and there.

              Ex. _____ is an animal that goes woof. (And its not a fish!)

          2. Next, have people fill in the blanks.  A fairly big number will work best, since this is a scientific experiment.

          3. For a readability test of the passage, give a point for each fill-in that is correct,  and no points for a               response that is incorrect.

         4.  To test entropy, add up the frequency of each different response, then divide that number by the total number of responses.

               Ex.  _____ is the King.

              Answer 1: Elvis                                             Frequency 2                                2/4= .5
              Answer 2: George                                         Frequency 1                                1/4= .25
              Answer 3: Me                                                Frequency 1                                1/4= .25
              Answer 4: Elvis                                ________________________
                                                                                     Sum=        4

         5. Next, put the number you get into this equation:    -Epi log2 pi*

         6. The value you get from this computation reflects the number of different responses and the frequency they appear.

         7. Now, compute the difference between this entropy value and the information value of each different response.

        8. This will give you the compatitbility score, and "it measures the degree to which a particular response stands out or fits into the array of responses".

*The author of the research article specifically says, and I quote: "One need not be concerned with the computations in detail."  The author of this article was very vague with the variables in the equation to figure out -Epi log2 pi.  He did not explain where the variable came from or how to figure them out.  The above is a representation of what I learned from the mathematical section of his article.  The example above is used only to show others how a cloze test is done.
 




An Application of Information Theory to A Decision Making System
(A Critique of the U.S. Legal System)

        The following is the author's own application of information theory to a decision making system (DMS).  It states that if one "starts from a different point of view in analyzing a situation, one is likely to come up with a different conclusion.
           The American jury trial system is a large and well known decision making system.  By applying Information Theory to it, the author concludes it allows for too much error because it was designed without Information Theory.  Ideally, the jury would be able to give 12 bits of unbiased, unanimous information (guilty or not guilty).  However, because each juror is allowed to deliberate his/her decision not only intrapersonally, but interpersonally, there is room for influence from the other members of the jury.

"If, because of bias, sequential dependency, of dependency among components of the system, or an combination of these, the total information capacity of a DMS is less than the amount of uncertainty in the situation, then the system is incapable of rendering a satisfactory decision."

If, for instance, there was one very intelligent, very well-spoken, very biased member on the jury with 11 passive members, a person could get only one bit of information from the whole group (i.e. the biased jurors opinion).  Basically the author is saying by getting each jurors vote independently from the others, and allowing the decision to be made without a unanimous vote (the majority wins), the jury trial system would be more reliable.
 

Now that you've seen how the 5 parts of Shannon and Weiner's theory affect communication, see
how the rules of the Coordinated Management of Meaning theory affect communication,
and how meaning is created from that communication here.

This page was last updated by Tara Howes on 03/16/00.

Back to the main page