All right.
Let's do this.
Logistic regression intuition.
And you can probably already tell by my voice that I'm pretty excited.
There's some very interesting slides coming up.
And this is quite an important topic.
But at the same time it is quite challenging.
So a quick heads up, there will be some math.
And I've done a few run through of this presentation already.
And I really I will try my best to convey everything in the simplest way possible.
So let's get into it.
We already know about the linear regression.
We know that there is a simple linear regression,
and it has a very short formula with one independent variable.
And we also have looked into the multiple linear regression
which has many independent variables.
So we already know how to deal with this type of challenge.
So when we have a scatterplot like that where on the horizontal axis
we've got the independent variable
on the vertical axis we've got the dependent variable.
And this is an example we looked at salary versus experience.
How do we create a model here.
So we use a simple linear regression.
It puts a line through our data.
And that line is modeling our observations.
So we can basically forecast things and
compare our actual observations to our model and so on.
But so we know how to deal with challenges like that or problems like that.
But your company had hired you as a data scientist.
What they do is they send out email offers to customers
with like a proposal to buy certain products.
it might be a clothing store, it might be grocery store or something like that.
So, what they do is basically they send out
a, offer in, in email,
to a lot of customers to purchase certain products.
And here you've got a sample of those customers that they contacted recently.
You've got their age.
And also you have a variable whether or not they took action.
So did the person take up an action, perform an action.
Did they take up an offer? Did they buy a product?
Did they open up an email, respond to our email and so on.
So was the action taken or not?
And the very black and white very different.
But at the same time, like even though we don't know what to do,
we don't know what's going on here. It's not what we're expecting.
But at the same time,
intuitively, we can see that there is some sort of correlation.
We can see that the observations on the bottom
there are a bit more to the left.
Observations on the top are a bit more right, implying kind of that probably
older people are more likely to take action,
based on this offer, and the younger people are more likely to ignore it.
So can we somehow model this?
How about we try our, the existing method
in our toolkit, which is the linear regression?
Let's run a linear regression.
And that's what it looks like.
As you can tell,
it doesn't look like the best approach doesn't look like the best method.
to solve this problem.
So let's look into this in a bit more detail.
We're going to draw
the other horizontal line over here.
Instead of trying to predict exactly what's going to happen for
for any given person, let's imagine a person and let's say we want to predict
for that person knowing their age, we want to predict whether they will
take up the offer or not.
But instead of predicting exactly whether they're going to take it up or not,
how about instead we will,
predict the probability we will we will state a probability
or a likelihood of that person taking up that offer.
And if you think of it in that way right away, things start becoming clearer.
Right away.
You can see that, okay, so this chart is actually from 0 to 1.
And I also know that probabilities are from 0 to 1.
Oh that's interesting.
So basically I could fit in probabilities between 0 and 1.
The fact that the red dots
the red observations are already either 0 or 1 and nowhere in between.
Well, that's simply because we already know the results.
We already know that they're either there or there.
But for something that we're predicting, it kind of makes sense to say, well,
I don't know for sure.
I don't know 100%. He'll take it up or not.
But I know maybe, maybe with an 80% chance he'll take it up or not.
Right.
And when you think of it that way, the linear regression line,
or at least that part that's in the middle between 0 and 1, it makes sense, right?
Well, it makes some sense because that is basically
it's telling you that anybody between those ages of,
for instance, where it's crossing the horizontal line for the first time,
it might be where it's crossing the horizontal axis, it might be like 25,
or let's say 35, and we're crossing the vertical,
the horizontal axis for one, it might be, let's say 55.
So those people between 35 and 55,
they,
anything in between, any person that falls in between that age,
there's a probability of them taking up this offer.
And that probability is increasing as we move to the right,
as we take more and more older people, that probability is increasing.
So the part of the linear regression in the middle kind of makes sense.
And we can do something with it.
But the parts that don't make sense at all are the ones at the top
and at the bottom.
Because a probability can never be less than zero, I can never be above one.
So what is the linear regression?
Trying to give us a hint about here?
Well, what it's probably saying what we could interpret as is that
people above that age that
nominal age, we said 55 above that age, they
they are very, very likely to take off or actually more than, more than 100%.
So basically, they're definitely taking it up.
Anybody below 35 on the other side, on the left,
they're definitely not taking it.
So essentially what we're saying is
if we ever take that approach, then we would have to replace this linear
regression line with a line that looks like that.
So let's just cut those bits off and replace them with horizontal parts.
And that would
be a very
basic, but it still would be an attempt at creating a model for this situation.
So we would still be able to use this to make some sort of predictions
and assumptions.
that's a lot that's, talk about
the correlation between the action and the age of a person.
So that's a very basic, understanding.
And that's kind of the start of our,
understanding of intuition behind logistic regression.
So let's see what the actual scientific approaches.
So here we've got the line that we looked at
and it is described by this equation.
Now this part is going to be this is the most fun part.
So bear with me
if you apply to this equation a sigmoid function which looks like that.
So you put the y into the
sigmoid function in purple, and then you solve
for y from the purple box and you put y back into the blue box.
Then you will get the green box.
So basically your linear regression will start to look like this.
And this is the formula for logistic regression.
And what that will do to a chart which is most importantly
this visual part.
It will convert it from
the chart that we see at the top to this new chart,
which is actually the logistic regression function.
So if at this stage you're asking yourself
what just happened, then you're not alone.
The first time I saw this or I learned this, this was the expression on my face,
if you if you told me if you told you come through with all of that,
that's super great.
That means you'll fly through this section.
But if you're confused right now, not a problem.
I was the same when I was in your shoes.
So let's, take this step by step.
Let's look at it step by step.
Exactly what happened. So there's our graph.
There's our independent variable.
There's our outcome.
Yes or no.
So that's the y the dependent variable.
And there are our observations in our data set based on these observations.
And plus using this formula
which we're going to take as given this is the logistic regression formula.
Using this formula.
And these observations we come up with this line.
And what is important to understand here.
It's not a magical line.
This line for the logistic regression is the same
as a slope or a trend line for a linear regression.
So basically what this line is doing is
it is using the formula is following the formula.
And it's the best fitting line that can fit these data sets.
So basically we're doing exactly the same thing as a for linear regression.
But it just looks different that's all.
So there there are heaps of these lines that can you can draw that look like that.
But only one of them is the best fitting line.
So the point of the logistic regression is to find that best fitting line.
And this is it.
So we found the best fitting line that follows that equation. And
it fits these
variables that we or these observations that we had in our data set.
After that we can forget about the equation.
We forget about the variables. We've got our line.
So this is our logistic regression function
we found in same thing as with the linear regression.
We've created the model. We've built the model. You can see it.
This is the model in front of you right there.
Now what can we do with this logistic regression.
Well we can use it to predict probabilities.
And we've already touched on probabilities that the lie between 0 and 1.
And that instead of predicting for sure that something will or will not happen,
how will we predict probability?
So let's look at Oh by the way,
probability here is called p hat.
So that that's a little sign about the p.
It gives it the name p hat.
And anything you see in the info hat in this section
just basically means that it's,
something we're predicting.
So, and that's, that's the way to remember it, that that picture P hat.
So we're predicting this probability
okay.
So let's take four random values for the independent variable for x.
We're going to say 2030 4050.
Let's see what happens with the variables.
So let's put them on the X line.
Those are the dots.
And I specifically put dots not x's or crosses
because it doesn't mean that they're on the horizontal.
The bottom line doesn't mean that the probability is zero or that
their dependent variable is zero.
No, they're just there because they're on the x axis.
We just plotted them.
There has nothing to do with the vertical axis.
Now let's what you need to do to find the probabilities is
you need to project these values onto your curve.
Once you project them you get these blue
light blue blue dots or blue
observations which plotted basically.
So these are the fitted values.
As you remember in Gretel you have in red.
You have the actual and in blue you have the fitted values.
So these are your fitted values.
And now if you project them,
if you want the probabilities you need to project them to the left like that.
And let's have a look at these probabilities.
So the person who's 20 years old the probability of taking up
this offer is very low, perhaps 0.7%.
So less than one percentage.
To take up this offer, the person who's 30 years old, the percent.
The probability is higher.
It's about 23%. To take up this offer.
The person who's 40 years old, their probability take up.
This offers 85% according to this model,
and the person who's 50 years old, their probability is 99.4%.
So that's the first thing that you can get out of a logistic regression.
That's what we're going to be using.
Very we're going to be using it very actively when we're talking
about building geo demographic segmentations, because you use
this probability as a score.
And we'll talk about this more.
So you can actually rank people who is the most likely to take up your friend.
Who's the least likely to take your offer up.
So it's actually even better than just having a one or a zero.
You have a probability.
So you can, order people by this probability.
Anyway, the you might want to say, well, I don't want the probability,
I want a prediction as because this is a regression.
I want a prediction for the, the y value.
So okay. We can do that.
Can we get let's get rid of those, probabilities.
Now can we get the Y the actual obviously we can't get the actual
because the actual is something that we can only observe in, in our data
set or in real life.
We can only get a prediction for the actual.
So y hat, as it has suggests, is a predicted value
for the dependent variable.
How did you get y hat?
Well, the approach is very arbitrary.
You have to select a line.
Let's wait for that okay.
So you have to select a line.
In this case we're going to take 50%.
you can select it anywhere, but 50% is usually selected
because it's in the middle.
And it's therefore you have symmetry and anything below this line.
So anything that falls on the curve below this line will be projected
downwards onto the zero line, which, which makes sense.
So it's basically saying if your probability,
your predicted probability of taking up this offer is less than 50%, let's say
it's 40% or 20%,
then we're just going to say that you're
not you're probably not going to take up this offer.
And so that's what's happening.
The person with 0.7%, the person with whatever it was, 27%, 23%,
their, predicted their probabilities are not zero, but they're below 50.
So you are if you're if you're
if you do require a Y hat, so a predicted
value, a yes or no value, then make sense that
if something is below 50%, you're probably going to say that
they're not going to take up the offer now.
And you think about, oh yeah, so there you go.
Both of them. Y hat so zero.
Now anything above the horizontal line that we've selected, the 50% line.
It is agreed that all of those values
that fall onto the curve above that line are projected upwards.
They're projected onto the yes line, the one line.
So the person that's had a probability of, 85% is projected
upwards, and the person had the probability of 99.7% is projected upwards.
Also makes sense. Right.
So if, if, somebody has got a chart, you're predicting
that somebody is probability of taking up an offer is 85%.
Then if you have to say yes or no, then you're probably going to say yes.
You're going to say yes.
This person will take up the offer
if you just if you have to choose one of the two.
So those are our predicted y hat values.
In this case they're both one for those two variables.
And those are the two things,
you can get out of, the logistic regression.
So you get them,
probabilities, which are important.
Also, you can get the Y hat.
So the predicted values for the dependent variables, once again, it's important
to think of it as it's doing exactly the same thing as a linear regression.
It's
It's fitting this line even though it's not a straight line.
And, and the values are not scattered.
Everything looks bizarre in its uniformity or in the way it is structured.
Its structure makes it look very bizarre, but still, it's it's a pretty much
the same way we agreed on a line or a formula for a curve,
and we're trying to fit the best curve to our data.
Once we've done that, we've got we've got a model,
we've got the, coefficients, which we'll talk about later,
and we can start drawing conclusions
or insights from, this model.
And some of the insights are we can get a probability of somebody taking action
or on of the event occurring and or basically of the answer being yes.
So it's not a yes no, it's a probability.
So 85% or 20% or whatever.
So that's when we projected to the left onto the y axis.
And also we can get a predicted value for the dependent variable
based on where we select this arbitrary line 50%.
You can select it anywhere you like.
You can select higher or lower.
depends on your knowledge about the problem at hand.
And as you as you understand, depending on where you selected,
that will significantly affect, your variables.
So I really hope this, explanation was,
trivial enough and yet insightful enough
for you to gain an intuitive understanding of logistic regression.
I look forward to seeing you then.
And until next time, happy analyzing.