Bayes' theorem

From Conservapedia

Bayes' theorem is a result in probability theory that shows how to invert a conditional probability. An extended form of the theorem provides a rigorous mathematical way of learning from uncertain data. This latter form is the basis of Bayesian statistics.

Let A and B be two uncertain events. We denote the probability of A by p(A) and probability of B by p(B). We denote the event that A and B are both true by AB, and the probability of AB by p(AB). In general, the fact that A has already occurred will affect the chances that B will occur, and vice versa. We denote the probability that B will occur, given that A has occurred, by p(B|A). We call this the probability of B, conditional on A. The event AB can occur in two ways. A can occur first, then B occurs, or B occurs first and then A occurs. The rules of probability require that the probability of two independent events both occurring is the product of the probabilities. Thus

                    p(AB) = p(A)p(B|A) = p(B)p(A|B)

This can be rearranged to be

                            p(B|A) = p(B)p(A|B)/p(A)

This is Bayes' theorem. The theorem is attributed to the Reverend Thomas Bayes (1702-1761) an English nonconformist minister, but it is likely to have been known to earlier mathematicians such as Pierre Laplace.

The theorem has many applications.

Example 1.

Suppose a particular disease afflicts 1% of the population. Suppose that a test for the disease is 95% accurate. Suppose that someone tests positive for the disease but there is no other evidence that they have the disease. What is the probability that they have the disease?

Let A be the event that the test result is positive. Let B be the event that the person actually has the disease.

Before the test result is known our probability for the person having the disease is p(B) = 1% = 0.01. The probability that the person has a positive test result, giving that they have the disease, is 95% = 0.95. The denominator term, p(A) is a little more complex since A can occur in two different ways. If the person has the disease, they will test positive with probability 0.95. If the person does not have the disease they will test positive with probability 5%. We denote the probability that B is not true by B'. The laws of probability require that B' = 1 - B. Thus

p(A) = p(B)p(A|B) + p(B')p(A|B') = 0.01 x 0.95 + 0.99 x 0.05 = 0.059

Hence

p(B|A) 0.01 x 0.95/0.059 = 0.16

In other words, there is only a 16% chance that a person testing positive actually has the disease.

An extended form of Bayes's theorem is obtained by noting that it applies to probability distributions as well as to events. Let y be a (vector valued) observable quantity that we want to use to estimate some unknown, unobservable (vector valued) quantity θ. Prior to seeing the data y, we summarise our knowledge about θ by a probability distribution p(θ). Assume that we have a model of the relationship between y and θ. Call this p(y | θ). We can use Bayes' theorem to update our knowledge of θ by incorporating the information contained in the observed data y.

We have

             p(θ | y) = p(θ)p(y | θ) / p(y)