Changes

The well known entity in the theory of information, seemingly founded by [[Claude Shannon]], is the [[information entropy]] defined as

The well known entity in the theory of information, seemingly founded by [[Claude Shannon]], is the [[information entropy]] defined as

  H = - sum { p(i) log [p(i) ] }

<math> H = -\sum_i p(i) \log_2 p(i) </math>

where the index i  runs through all positive integers up to n, for instance (i = 1, 2, …, n). This entropy may calculated for any statistical frequency function, such that the sum { p(i) } = 1. The maximum in H (= log(n)) is obtained when all p(i) are equal. Mathematically, H is also equivalent to entropy and disorder in physics.

where the index i  runs through all positive integers up to n, for instance (i = 1, 2, …, n). Note that the logarithm is taken to base 2. This entropy may calculated for any statistical frequency function, such that <math>\sum p(i) = 1</math>. The maximum in H (= log(n)) is obtained when all p(i) are equal. Mathematically, H is also equivalent to entropy and disorder in physics.

H may also be interpreted as an [[average information]] based on a certain axiom that when an event occurs with probability p(i), then we get the [[self-information]] - log [p(i) ]. This information is only about probability and has no mental meaning.

H may also be interpreted as an [[average information]] based on a certain axiom that when an event occurs with probability p(i), then we get the [[self-information]] - log [p(i) ]. This information is only about probability and has no mental meaning.

A reasonable postulate is that the sum of information from any two different messages should equal the total information in the joined messages. And one way to accomplish this is to let the self-information equal the negative logarithm of the probability. Thus, the information in a pair of tosses becomes

A reasonable postulate is that the sum of information from any two different messages should equal the total information in the joined messages. And one way to accomplish this is to let the self-information equal the negative logarithm of the probability. Thus, the information in a pair of tosses becomes

  -log(0.5) - log(0.5) = -log(0.25)   or   log(2) + log(2) = log(4)

<math> -\log(0.5) - \log(0.5) = -\log(0.25) </math>  or <math>\log(2) + \log(2) = \log(4) </math>

Another reasonable postulate is that when a message increases in length, then its information increases. For instance, a triple of tosses gives messages with probability = 0.125 and a quadruple with probability = 0.0625. and  

Another reasonable postulate is that when a message increases in length, then its information increases. For instance, a triple of tosses gives messages with probability = 0.125 and a quadruple with probability = 0.0625. and  

  -log(0.0625)  > -log(0.125).   

<math>  -\log(0.0625)  > -\log(0.125).  </math>

And an event occurring with probability = 1 is a certain event, which will carry no information. Consequently -log(1) = 0.   

And an event occurring with probability = 1 is a certain event, which will carry no information. Consequently -log(1) = 0.   

If all these properties should be maintained for all possible divisions of messages written in different alphabets, then the negative logarithm of the probability is a very good candidate for being a measure of information.  

If all these properties should be maintained for all possible divisions of messages written in different alphabets, then the negative logarithm of the probability is a very good candidate for being a measure of information.  

More generally - the word “we” may be followed by “are”, but hardly by “am” – some part Y of a message may depend on another part X, making the average information H(Y|X) less than H(XY), i. e. the average information when X and Y are statistically independent. We have

More generally - the word “we” may be followed by “are”, but hardly ever by “am” – some part Y of a message may depend on another part X, making the average information H(Y|X) less than H(XY), i. e. the average information when X and Y are statistically independent. We have

  H(Y|X) = H(XY) – H(X)

<math>  H(Y|X)=H( XY ) - H(X)</math>

and the only functions satisfying this formula are those proportional to the entropy H, Åslund, 1961. Then, entropy may be seen as [[average information]].

and the only functions satisfying this formula are those proportional to the [[entropy]] H, Åslund, 1961. Then, entropy may be seen as [[average information]].

== The continuous case ==

== The continuous case ==

In the continuous case the discrete frequency function is replaced by a continuous probability density function, p. d. f., f(x), and H may be written

In the continuous case the discrete frequency function is replaced by a continuous probability density function, (p. d. f.) , f(x), and H may be written

  H = integral{ f(x) log[ f(x) dx ] dx }.

<math>  H = \int f(x) \log f(x) dx .</math>

A problem here is that H becomes uncertain because of the infinitely small dx. A way to evade the problem is to define H in such way that H equals the logarithm of the volume – in analogy to the cardinality n in the discrete case - covered by a uniform p. d. f. over the volume. Thus H may be defined as

A problem here is that H becomes uncertain because of the infinitesimally small dx. A way to evade the problem is to define H in such way that H equals the logarithm of the volume – in analogy to the cardinality n in the discrete case - covered by a uniform p. d. f. over the volume. Thus H may be defined as

  H = integral{ f(x) log[ f(x) ] dx }.

<math> H = \int f(x) \log f(x) dx .</math>

The “volume” covered by a Gaussian p. d. f. is proportional to the volume of the ellipsoid of concentration and equal to the square-root{ (2 pi e)^n determinant(M) }, where M is the moment matrix of the Gaussian.

The “volume” covered by a Gaussian p. d. f. is proportional to the volume of the ellipsoid of concentration and equal to <math>\sqrt{2\pi e^n |M|} </math>, where M is the moment matrix of the Gaussian.

The [[average information]] may be maximized by a certain genetic algorithm, [[Gaussian adaptation]], keeping the [[mean fitness]] – i. e. the average probability of becoming a parent to new individuals in the progeny – constant (and without any need for any knowledge about average information as a criterion function).  This is illustrated by the figure below, showing Gaussian adaptation climbing a mountain crest in a phenotypic landscape.

The [[average information]] may be maximized by a certain genetic algorithm, [[Gaussian adaptation]], keeping the [[mean fitness]] – i. e. the average probability of becoming a parent to new individuals in the progeny – constant (and without any need for any knowledge about average information as a criterion function).  This is illustrated by the figure below, showing Gaussian adaptation climbing a mountain crest in a phenotypic landscape.