INFORMATION THEORY

a mathematical representation of the conditions and parameters affecting the transmission and processing of information. The theory stems from pioneering work done by the American electrical engineer Claude E. Shannon, who published his seminal work in 1948. Since then, information theory has developed rapidly, affecting not only the design of communications systems but also such areas as automation, information science, psychology, linguistics, and thermodynamics. The basic elements of any general communications system include a source of information; a transmitting device that transforms the information, or message, into a form suitable for transmission by a particular means; the means, or channel, over which the message is transmitted; a receiving device, which decodes the message back into some approximation of its original form; the destination or intended recipient of the message; and a source of noise (i.e., interference or distortion), which changes the message in unpredictable ways during transmission. It is important to note that information as understood in information theory has nothing to do with any inherent meaning in a message; it is rather a measure of the predictability of each transmitted messagein fact log2 (1/p) where p is the probability of a given message being received. The expected value of a transmission is referred to as the entropy, H, or average information of the set of messages. A mathematical characterization of the generalized communication system yields a number of important quantities, including the rate at which information is produced at the source, the capacity of the channel for handling information, and the average amount of information in a message of any particular type. To a large extent the techniques used with information theory are drawn from the mathematical science of probability. Estimates of the accuracy of a given transmission of information under known conditions of noise interference, for example, are probabilistic, as are the numerous approaches to encoding and decoding that have been developed to reduce uncertainty or error to minimal levels. a mathematical representation of the conditions and parameters affecting the transmission and processing of information. Most closely associated with the work of the American electrical engineer Claude Elwood Shannon in the mid-20th century, information theory is chiefly of interest to communication engineers, though some of the concepts have been adopted and used in such fields as psychology and linguistics. Information theory overlaps heavily with communication theory, but it is more oriented toward the fundamental limitations on the processing and communication of information and less oriented toward the detailed operation of particular devices. George Markowsky Additional reading John R. Pierce, An Introduction to Information Theory: Symbols, Signals & Noise, 2nd rev. ed. (1980), is an entertaining and very readable account of information theory, its applications, and related fields. While it displays the relevant mathematics, most of it can be read profitably by people with a weak mathematical background.Steven Roman, Coding and Information Theory (1992), is meant for an introductory college course. This book assumes that the reader has a basic knowledge of probability; an appendix reviews necessary ideas from modern algebra.Aleksandr I. Khinchin, Mathematical Foundations of Information Theory, trans. from Russian (1957, reissued 1967), is a mathematically challenging, but elegant treatment of information theory, intended for the advanced reader.N.J.A. Sloane and Aaron D. Wyner (eds.), Claude Elwood Shannon: Collected Papers (1993), is a very interesting volume that shows the breadth and depth of Shannon's work. While most of the paperssuch as The Mathematical Theory of Communication and Communication Theory of Secrecy Systemsrequire mathematical sophistication, somesuch as The Bandwagon, Game Playing Machines, and Claude Shannon's No-Drop Juggling Dioramado not. George Markowsky