Bayes Theorem¶

Introduction¶

Let us think about probability of a single event - a probability that coin turn up heads for example. Probability (1/2) is the number of outcomes that qualify as event "coin turn up heads" (1) divided by the total number of all possible outcomes (2). To calculate probability of multiple events occuring in sequence we multiply the probabilities of each event in sequence. Consider an event A occuring given that another event B (evidence that B) has occurred - probability of it is known as conditional probability. We need to calculate the probability of A conditinal on event B occuring knowing that probability of B conditional on A occuring. This is the case for applying bayesian inference. What is that? Bayesian inference is a statistical inference that uses Bayes' theorem. It is being applied to geostatistics, genetics, linguistics, image processing, machine learning and other fields.

Events¶

An event is an outcome. Let's think about system with n possible outcomes. Examples of single events:

• getting a tail when flip a coin,
• getting 5 when roll a die,
• have disease when receiving a positive medical test result,
• flag a message with some particular words in it as true spam.

Besides events there are other important concept in Bayes theorem: Priors, Likelihoods, Evidence (denominator in Bayes Rule)

Examples¶

To illustrate the problem, a contingency table is used. Here I list some examples of problems that can be faced using Bayes' Theorem:

1. I enjoyed approach by 3b1b to without touching to formula and visuals, just think of an example of "maybe-librarian-maybe-farmer" to get an intuition about when you need to use it.
2. Example of two containers(bowls/bags/boxes) containing k types of objects(Cookies, M&M's, colored stones etc). What is the probability that you pick red stone out of the container X given that each of the two containers have a particular distribution of red and blue stones.
3. Drug screening. Find probability that positive tested individual took drug knowing that 0.5% of the general population comsume drugs. Previous knowledge about drug tests: test is 97% sensitive (true positive) and 95% specific (true negative). Case 1: false positive drug test 💉😀 Case 2: true positive drug test 💉🤢
4. The medical test. The probability that patient actually have this disease knowing that medical tests are not 100 accurate.
5. Spam Filtering. Event - message is spam. The test - the message contains some flagged words. TODO: Figure out the probability a message is spam given that it contains certain flagged words.
6. Making a prediction. Technically calculate a degree of belief in getting some value for continuous variable C.

Closer Look at Box Problem¶

Here I look closer at a trivial example 2 listed in Examples section.

Find the probability that fruit taken from box E, given that selected fruit is a lemon. For this example we stick with two equally possible outcomes. The probabilities of selecting boxes E and F are (1/2) and (1/2), respectively.

Box E: LLLLLLLLLL Box F: AAAAALLLLL

Imagine that each probability represents a box. Box E is filled with 10 lemons. To demonstrate this, we will paint Box E yellow. In Box F there are also fruits of two different types: 5 apples and 5 lemons.

Now imagine, you closed your eyes, shuffled the boxes, and selected a fruit from one of them - and it was lemon. What box the lemon came from, what box would you prefer to beliece as origin? We see that 'lemon' Box E has a greater probability of being selected than Box F knowing that we got lemon.

When your got lemon something disappeared: the probability of selecting an apple. We wipe away the portion of Box F that represents the apples.

We see now at the picture below on the right that there is different amount of parts of getting lemon from in each box. There are two parts in box E versus one part in box F, which makes totally 3 parts. It is primitive explanation but by dividing by possible 3 parts, we can see that Box E has a probability of 2/3 , or 0.66 of being selected, Box F has a probability of 1/3, 0.33 of being selected.

Now, we solve this same problem with the formula.

Formula¶

$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$

Where:

• P(A) is the prior probability of A, without account for any information about B.
• P(A|B) is the conditional probability of A, given B ('|' means given) called the posterior probability it depends upon the occurence of B.
• P(B|A) is the conditional probability of B given A. It is also called the likelihood. P(B) is the prior probability of B.
• P(B) is the probability of B is the sum of the conditional probabilities based on whether A has occurred or not. To apply the fomula to our lemon box case we need to figure out again what we are trying to find. #### Firgure out each Parameter in Formula ##### 1. What is the Problem We want to know the probability of origin is box F given that we got lemon P(E|L). ##### 2. Use Relevant Variables Names in your Formula

  A --> Event that apple is selected
  L --> Event that lemon is selected
  E --> Event that box E is selected
  F --> Event that box F is selected

We need to calculate P(E|L) using formula:

$P(E|L) = \frac{P(L|E)P(E)}{P(L)}$

3. Find Values, then Pass as Parameters¶

• P(E) = .5 * the probability of drawing from Box E? Remember, this probability is independent of all other events. Since there are only two boxes and the probability of selecting from either is equal, the answer is .5

P(L) = .75 To answer this we ask the following: What is the probability that we will select a chocolate chip cookie? Remember, this probability is independent of all other events. There are 20 cookies total in both boxes, and 15 of them are chocolate chip. So, 15/20 is .75 P(CC Cookie | Box A) = 1 To answer this question, we ask the following: What is the probability of selecting a chocolate chip cookie given that we have selected from Box A? Since there are only chocolate chip cookies in Box A, the probability is 1. * A probability of 1 represents a 100% probability of something occurring.

$P(L|E)$ = 2/5 An apple selected fiven that we pick up from box E

$P(E)$ = 1/3

Putting above values in the Bayes's Formula P(A) = P(E)P(A|E) + P(F)P(A|F) = (1/3)(2/5) + (2/3)(3/4) = 2/15 + 1/2 = 19/30

Understanding Bayes Theorem¶

In email classification example we have some prior probability of an event A occuring (email is spam) given some additional piece of information (email contains word 'viagra') about probability is a marginal likelihood. Marginal because divided by tables margins.

the goal of Bayes rule¶

This rule is mainly the way to measure the relationship between P(A|B) and P(B|A)

$ Pr(A|B)=\frac{\Pr(B|A)\Pr(A)}{\Pr(B|A)\Pr(A)+\Pr(B|\neg A)\Pr(\neg A)}$

Visualisation to figure out Bayes Theorem¶

Let's assume yellow (A) and blue (B) rectangles are two probabilistic events A and B. Marginal probabilities P(A) and P(B) are connected by product rule wich is represented in picture below.

This illustrates the product rule, you may see that the intersection is interchangeable, wich makes derive Bayes' theorem: $ \Pr(A|B)=\frac{\Pr(B|A)\Pr(A)}{\Pr(B|A)\Pr(A)+\Pr(B|\neg A)\Pr(\neg A)} $

this picture illustrates the concepts of union, intersection and complement. The non filled area is a complement, union is a smaller yellow-green-pink rectangle and intersection is green rectangle.

References¶

https://seeing-theory.brown.edu/bayesian-inference/index.html https://www.statisticshowto.com/probability-and-statistics/probability-main-index/bayes-theorem-problems/ https://drive.google.com/file/d/1KvM-ZxAnsrfgWT__QhQmOcHW3jnK400t/view https://www.statisticshowto.com/probability-and-statistics/probability-main-index/bayes-theorem-problems/

https://napsterinblue.github.io/notes/stats/bayes/basics_tables/ https://jakevdp.github.io/blog/2014/03/11/frequentism-and-bayesianism-a-practical-intro/