Hello and welcome back to the.
Course on Machine Learning.
This is Kirill Eremenko, and in today's tutorial we're.
Talking about the Naive Bayes classifier.
This is a very interesting machine learning algorithm.
And today we're going to get to.
Know it on a very intuitive level.
And in line
with the super data science mission, which is making the complex simple.
We're going to break down this complex topic into simple steps and bite.
Sized pieces of information.
I've got some very exciting slides prepared ahead, so.
Let's dive straight into it.
All right.
So here we've got the Bayes theorem.
And this is something we talked about in the previous tutorials.
So by now we should be quite. Comfortable with the concept.
How are we going to apply it to create a.
Machine learning algorithm.
Well let's have a look here.
We've got a data set.
So it has two features.
It has X1 and x2.
And there are two categories category.
One which is red and category two which is green.
But instead of working with these abstract terms, we're going to convert them into
something that.
We can understand a bit better,
something that's a bit easier to operate with or to talk about.
So we're going to call the y.
Variable x2 variable salary.
And the x one variable is going to be h.
So basically we're representing observations or people.
That are part of our data set in terms of their age and salary.
As you can see we have 30 people here on this chart.
And the categories we're going to replace them with walks, meaning that person.
Walks to work and green will be. Drive.
That means that person drives to work.
And so now we get to a problem to the machine learning challenge that.
We're going to be solving.
What happens if we add a new observation, a new data point into this set?
How do we. Classify this new data point.
So as you can tell, this is a supervised machine learning algorithm
because we're classifying something based on previously known classes.
And so the question is is this person going to be.
Classified as a person. Who walks to work?
Or is this.
Person going to be classified as a person who drives to work?
And the Naive Bayes.
Algorithm is going.
To help us solve this challenge.
All right. So how are we going to approach this?
We need a plan of attack.
It is going to be quite a complex approach.
But at the same time we're going to break it down into.
Steps and it'll all make sense.
It'll be very easy to understand.
So our plan of attack.
We're going to take the Bayes theorem and we're.
Going to apply it twice.
First time we're going to apply it to find out
what is the probability that this person walks given his features.
And x over here is the features or represents the features.
Of that data point.
So let's go back to the visualization here.
So here.
You can see that this is.
Our new data point. That person has a certain age.
So let's say the age of that person maybe is like 25 years old.
And then they have a salary.
So let's say this salary is $30,000 per year.
So those are features of this observation.
Right now we're only working with two variables
just for simplicity's sake so we. Can visualize.
Things age and salary.
But in reality there could be many many many more features.
There could be features on how many what what industry they work in or how many
years of education they have, or how long they've had a driver's license for.
And the things like kind of how far away they live from work.
So there could be lots of variables.
But at the same time, right now
we're only going to be dealing with two age and salary.
And regardless of how many variables you.
Have, they will be called.
And we're going to call them features.
So given the features of X, so given the age of.
25 and the salary of $30,000, and we'll talk in more detail
about exactly what we mean by features just in a moment.
And so therefore this part represents that person that we're trying to classify.
What is the likelihood of a person with those features x.
So we know that we are taking somebody for those features that we have
in our new data point.
What is the likelihood of them working.
And then we've. Got the right side.
So we're going to talk through each one of these as we calculate them.
But for now let's just give. Them their names going from right to left.
So this one on over here is called the prior probability.
And we're going to calculate that first because it's the easiest to calculate.
Next one is the marginal likelihood.
And we're going to calculate that second.
The third one is a livelihood.
That's just the names that they have.
And we're going to calculate that.
Third and finally what we're looking for is called the posterior probability.
We're going to calculate that force.
All right.
So that's our plan of attack for step one.
This is all still step one to calculate the probability that somebody walks
given those features x that we see in our new data point.
Next we're going to have step two where we're going to calculate the probability
that somebody drives given those features x that we see in our new data point.
And again here we'll have the probability which will calculate first,
then the marginal likelihood then the likelihood.
And then we'll get to posterior probability.
And finally we're going to compare the probability that somebody walks
given features x versus the probability that somebody drives given features x.
And then from there we'll decide which class to put that new data point in.
So as you can see the Naive Bayes classifier is a probabilistic type
of classifier
because we first calculate
the probabilities and then based on probabilities, we're assigning a class.
All right.
So are you ready to perform these steps.
It's going to be lots of fun.
We're going to take it nice and easy nice and slowly
so that we understand everything. And after
this you're going to be very comfortable with the Naive Bayes classifier.
Step one. All right. So here we have our visualization.
Let's move it all to the left a little bit so we.
Can make some space.
Now we're going to.
Calculate the first probability in our Bayes theorem.
We're going to calculate the probability that somebody walks right.
Just the overall probability. And what does that mean.
That is the probability that somebody walks without knowing anything about them.
So we're just saying
we're going to add a new observation to our data set into here.
But we don't know their age and we don't know their salary.
We're just going to put it somewhere. Into our data set.
What is the probability that this person.
That we're adding to our database walks to work?
Well, it's very easy and straightforward.
From here we don't have much choice.
The only thing that we can do is calculate the number of.
Read observations,
the number of people that actually walk and divide by the overall number.
So probability that a person walks to work.
Without any other knowledge.
Is the number of workers, number of people at work,
which is these red dots divided by the total number of observation.
The green dots.
So the gray dot is in participating in these calculations.
So here we have probability of somebody walks is 1010
red dots divided by 30 dots overall.
All right.
So that was easy.
We've calculated the prior probability.
Next we calculating the marginal likelihood.
And this is where things get interesting.
So how do we calculate the marginal likelihood.
Let's have a look.
Here's our dataset again.
And the first thing we're you're going to do is we're going to select a radius.
And we're going to draw a circle around our observation like that.
Now this radius you need.
To select on your own.
You need to decide for your algorithm.
This is going to be like an input parameter or an algorithm.
You can select less. You can select it more.
It's up to you.
Now what does this radius do.
Well what we're going to do is we're going to first of all let's
just to make things easier, we're going to remove.
Our dot for now just so that it's not confusing us.
And then we're.
Going to look at all the.
Points that are inside this radius.
And what.
We're saying here is.
That all of the points inside the circle.
Are we're going to deem them to be similar in terms of features, to the.
Point that we had the point that we had.
Remember, it had an age of, for example, 25 and a salary of $30,000 per year.
So now we're going to draw a radius around it.
And let's say anybody between the ages of 20 and 30 and in the salaries
of $25,000 to $35,000,
anybody that falls in that circle, again, it's not a square.
It's not just a square, it's a circle.
anybody who falls somewhere, somewhere in that.
Vicinity is going to be deemed similar to.
The new data point that we're adding to our data set.
So, as you can imagine, this radius is actually going
to have a big say in the way your algorithm works.
Well, let's say we have this radius and.
This is how it all played out.
We have three red dots one green dot in there.
All right.
So now what do we do.
How do we calculate the probability of x.
And what is the probability of x.
Well the probability of x is the probability of a new point.
That we add to our data set.
Being similar in features to the.
Point that we actually are adding to. It.
So basically is the probability of.
That new point that we're adding.
Or like any random point that we add, is the probability
of that any random point to fall into this circle.
And p of x is calculated as the number of similar observations.
So the number of observations that already. We can see in the circle.
So one, two three, four divided by the total number of observations which is 30.
So p of x is four divided by three.
Once again just to reiterate, p of x tells us what is the likelihood of any new.
Random variable that we add to this data set falling inside this circle.
And it is four over 30 because we only have four, based on prior knowledge,
we can tell that this four here and this student also is four with three.
All right.
So that wasn't hard at all as well.
We calculate the marginal likelihood.
So so far we got this one and we got this one.
Next we're moving on to the likelihood.
And this is probably the most complex one.
What is the likelihood that somebody
who walks exhibits features X.
Well actually after we've spoken about the marginal likelihood calculating.
The likelihood won't be as. Complex.
So let's have a look.
So there's our chart.
And now what we're.
Going to do is we're going to.
Draw the same circle again.
And once again we're going to remove the.
Gray point for now.
And we're going to color our circle.
And so.
Anything that falls inside the circle is deemed to be similar to the.
Point that we're adding.
So the question is what is the probability.
That a randomly selected data point from our data.
Set will be similar to.
The data point that we're adding.
So basically, what is the likelihood that a randomly selected data point will be.
From this circle?
Given this vertical pipe means given that
that person walks, that we know that that person walks to work.
The other way to think about
this is we're only working with people who walk to work.
So we're only working with the red dots which represent people who walk to work.
So let's forget about the green dots there like that.
Now they're faint and we're not even talking about them at all.
We're only talking about the red dots.
So the question is, given that we're only.
Working with the red dots, what is the likelihood that a
randomly selected data point from our data
set, from the red dots, is somebody.
Who exhibits features similar to.
The point that we are adding to our data set.
So basically, what is the likelihood that a randomly selected
red dot falls into this gray area, into this circle?
That's what the question we're asking.
And that is also very simple.
Now that we know how all of this works, it's basically
the number of similar observations among those who work.
So the number of red dots that actually fall inside this red circle.
In this gray circle, that's three divided by the total number of workers.
So people and total number of people who walk to work.
And that is three over ten.
There we go.
So that's our P of the likelihood of somebody exhibiting the feature similar
to that data point that were about to add, given that we're only selecting among
the red dots.
So that's three over ten.
And that was our likelihood.
So now if we plug all that in so there we go.
That likelihood is done.
So if we plug all of that.
In we'll get our posterior probability.
So three over ten times ten over 30 and divided by four over 30.
So if we calculate that it'll give us 0.75
75% is the probability
that somebody that we put into the place where we're putting. X.
Is should be classified as a person who walks to work.
That was step one was pretty intense, right.
Pretty exciting to calculate this value now.
And the next step is step two.
That's step one. The next step is step two.
To do the same thing for the likelihood that somebody with.
Features X.
Will be classified or should be classified as a person who drives to work.
And here I'm going to throw you a challenge.
I'm going to challenge you to pause this video.
Or rewind back to find out, to have the image in front of you
and do these calculations yourself, to.
Actually go through the.
Same steps and perform those. Calculations.
If you'd like to see and compare to my calculations.
Then I'm going to put in another video after this one.
So another tutorial after this one in the course.
So you can just go to the next tutorial and compare.
Otherwise I'm just going to show you the result now.
So the result is one over 24 likelihood.
Or let's start from the right.
Prior probability is 20 over 30 and marginal likelihood remains
unchanged for over 30.
Likelihood changes to one over 20.
So the probability of somebody who exhibits features
X being a person who drives to work is 25%.
So that was step two.
Now we're going to do step three.
We're going to compare.
The probability of somebody with features X
the probability of them being a person who walks to work versus the probability
of somebody who features X being a person who drives to work.
So it's 75% versus 25%.
And therefore the first is greater in the second.
And therefore it is more likely that that person.
With features X is.
Going to be a person who walks to work than the person who drives to work.
So still a 25% chance that that is a person who drives to work.
But percent chance that it is a person who walks to.
Work is great, 75% and therefore, we're going to classify
this point as a person who walks to work.
There we go. That is how the.
Naive Bayes algorithm in machine learning works.
I hope you found this tutorial useful.
I was I'm pretty excited and pre proud of these slides,
and hopefully this is a step by step and simple.
Explanation of a complex concept.
And I look forward to seeing you next time.
Until then, enjoy machine learning.