Naive Bayes classifier

Here is a worked example of naive Bayesian classification which is an application of Bayesian inference to the document classification problem.

Consider the problem of classifying documents by their content, for example into spam and non-spam E-mails. Imagine that documents are drawn from a number of classes of documents which can be modelled as sets of words where the (independent) probability that the i-th word of a given document occurs in a document from class C can be written as

p(w_i | C).

(For this treatment, we simplify thing further by assuming that the probability of a word in a document is independent of the length of a document, or that all documents are of the same length).

Then the probability of a given document D, given a class C, is

p(D | C) = product over i of p(w_i | C)

The question that we desire to answer is: what is the probability that a given document D belongs to a given class C?

Now, by their definition,

p(D | C) = p(D and C) / p(C)

and

p(C | D) = p(D and C) / p(D)

Bayes' theorem manipulates these into a statement of probability in terms of likelihood.

p(C | D) = (p(D) / p(C)) · p(D | C)

Assume for the moment that there are only two classes, S and not-S.

p(D | S) = product over i of p(w_i | S)

and

p(D | not-S) = product over i of p(w_i | not-S)

Using the Bayesian result above, we can write:

p(S | D) = (p(D) / p(S)) · product over i of p(w_i | S)

p(not-S | D) = (p(D) / p(not-S)) · product over i of p(w_i | not-S)

Dividing one by the other gives:

p(S | D) / p(not-S | D) =

p(not-S) · product over i of p(w_i | S)
p(S) · product over i of p(w_i | not-S)

Which can be re-factored as:

(p(not-S) / p(S)) · product over i of p(w_i | S)
                                      p(w_i | not-S)

Taking the logarithm of all these ratios, we have:

ln(p(S | D) / p(not-S | D)) =

ln(p(not-S) / p(S)) + sum over i of ln(p(w_i | S)/p(w_i | not-S))

This technique of "log-likelihood ratios" is a common technique in statistics. In the case of two mutually exclusive alternatives (such as this example), the conversion of a log-likelihood ratio to a probability takes the form of a sigmoid curve.

In real life, the naive Bayes approach is more powerful than might be expected from the extreme simplicity of its model; in particular, it is fairly robust in the presence of non-independent attributes w_i. Recent theoretical analysis has shown why the naive Bayes classifier is so robust.