Entropy

Entropy is a measure of disorder or information content in a system, first postulated by Lazare Carnot in 1803.

The second law of thermodynamics states that entropy will always increase over time within a closed system, defining a closed system as one in which neither matter nor energy may enter or leave.

Entropy is undeniable and yet creates perhaps insurmountable difficulties for many modern theories of physics. For example, it renders time asymmetric, resulting in an arrow of time that is difficult to reconcile with the theory of relativity. Entropy casts doubt on whether physical laws or the speed of light are invariant and perpetual.

Perhaps because entropy confounds so much other contemporary theory in physics, there is very little theoretical work on entropy today.

Definitions

Thermodynamic definition

In classical thermodynamics, if a small amount of energy dQ is supplied to a system from a reservoir held at temperature T, the change in entropy is given by

<math> dS=\frac{dQ}{T} </math>

For a measurable change between two states i and f this expression integrates to

<math> \Delta S=\int_{i}^{f}\frac{dQ}{T} </math>

Statistical mechanics definition 1

If a system can be arranged in W different ways, the entropy is

<math> S= k_B \log W </math>

where <math>k_B</math> is Boltzmann's constant.

Statistical mechanics definition 2

Label the different states a thermodynamic system can be in by <math>i=1,2,3\ldots N</math>. If the probability of finding the system in state i is <math>p_i</math>, the entropy is

<math> S=-k_B \sum_i^N p_i \log p_i </math>

This definition is closely related to ideas in information theory, where the definition of information content is very similar to the definition of entropy.

Entropy in information Theory

For a discrete random variable, entropy is defined as

<math> H(X)=-\sum_{x\in \mathcal{X}} P(X=x)\log_2 \left(P(X=x)\right) </math>

For a continuous random variable, the analogous description for entropy, which in this case represents the number of bits necessary to quantize a signal to a desired accuracy, is given by

<math> h(X)=-\int_{x\in \mathcal{X}} f_X (x) \log_2 \left(f_X (x)\right) </math>

See also

• History of the development of entropy