Hello and welcome back to the course.
Today we're talking about p-values and statistical significance.
A quick heads up.
This tutorial is borrowed from another course of R's
called statistics for Business Analytics and Data Science A to Z.
If you hear a references to a z
score and hypothesis testing and other parts
that are relevant to that course, but not relevant to the course
in which we're in right now, then please ignore them.
The focus of this tutorial, what we want to get out of it is,
p values,
what they mean, how to feel about them, and what statistical significance
significance means and what hypothesis testing is all about.
So with that in mind, here we go.
Hello and welcome back to the course on statistics.
Today we've got a very exciting topic, statistical significance.
And it's very exciting because I know that from my experience from my career,
it's a question that is always asked, or at least you ask yourself
this question, is my results statistically significant or not?
Or are these insights statistically significant or not?
And if you don't understand statistical significance,
then it's a question that you dread from your manager,
or the person that you're reporting to because you cannot substantiate
your results.
You found some stuff, but you don't know if it's right or wrong.
And in this section, we're going to be exploring this concept in a lot of detail.
And we're going to be referencing it a lot.
And you will get a great understanding.
Hopefully you'll be able to build a strong
understanding of statistical significance.
However, in this tutorial, what we're going to focus on is in on the
intuition behind statistical significance, what how you can link,
how you feel about an experiment to what actually statistical significance is.
So let's have a look.
All right.
So here we've got an experiment a coin toss
I know it's always a coin toss, but a coin toss is,
in all fairness, a great example to get started with
because it is so simple, though, there's only two possible outcomes.
And you're already done this so many times in your life that, you know,
kind of what to expect.
And that's why it will help us build the intuition.
So, there are two possible
versions or two possible situations, kind of like
we're going to start talking about hypothesis testing right now
when we're talking about hypothesis testing, we're thinking about two possible
alternative universes, if you will, if you want to put it that way.
So one possible universe is that in that universe it's a fair coin
and that scenario in that environment,
but probably like the the way to put it, as in the universe.
So in that universe, the it's a fair coin.
So that's, that's our original assumption.
That's why it's got h zero.
That's the null hypothesis that we're starting off with.
And then H1 is the alternative universe or the alternative hypothesis.
Sometimes it's also called h a.
And in this universe this is not a fair coin.
So we want to understand which universe do we live in or which situation
we're actually dealing with, or what is the truth.
That's what we're trying to assess. Is it a fair coin?
Is the null hypothesis correct, or is it not a fair coin?
And then the alternative hypothesis is correct.
And so the way we're going to go about this is we're going to assume
that H0 or the null hypothesis is true.
So we're going to assume that we live in the first universe.
And then based on our experiments we're going to see if we can contradict that.
If we can come to a contradiction and say,
oh, actually our assumption was incorrect.
And we'll talk more about that towards the end of this tutorial.
But for now, let's assume that you've got a coin that you're tossing
that some or somebody is going to toss just now,
and you're assuming it's a fair coin.
We live in that universe,
so the coin is tossed the first time and the result is tails.
The question is, how do you feel about this?
Forget it.
Forget about those statistics for a second.
How do you feel about this?
Do you feel that this is a fair coin or is the coin is rigged?
You probably feel that it is likely a fair coin.
This is a totally probable outcome in that universe that we have assumed
we live in.
In the H0 universe, the probability of that is 50%.
So it could have been heads, could have been tails. No problem.
Then the coin is tossed a second time and it's tails again.
How do you feel about this now?
Do you feel that the coin is rigged or, do you feel that it's a fair coin?
Well, let's look at the probability of this happening in our universe
that we're living in.
In the H0 hypothesis, the probability of that happening
if this is a fair coin is 25%.
So no big deal could happen.
Then the coin is tossed again and again.
It's tails.
Okay.
So what how do you feel about that now?
What do you feel about, having three coins?
three times the same coin is tossed and you get every single time it's tails.
Well, you know, it might be a bit a bit suspicious, but it's fairly okay,
because in our universe in which we live in, in the H0 universe
where the coin is assumed to be fair, this could have happened,
and the probability of that happening is, 12%.
But then the coin is tossed again.
And guess what? It's tails again.
All right.
So are you getting a bit suspicious now?
Do you feel that if this were a fair coin, this would be quite hard
for this to happen?
Well, in the universe that we assume we live in, in the
H0 universe, where the null hypothesis is assumed to be true,
or when the null hypothesis is true and we assume we're in that universe.
Well, in that case, this could have happened with a probability of 6%.
All right.
And then the coin is tossed again and again it is tails.
Now, how do you feel about it?
How do you feel about seeing somebody toss a coin
and them getting tails every single time.
Five times in a row.
Well, you feel a bit correct me if I'm wrong,
but I assuming you feel a bit uneasy about that.
Did you feel that something might be going on here?
Something is a bit not right or like it's a bit
suspicious that this coin is
has had tails five times in a row and your feeling is correct.
Your feeling is natural
because the probability of this happening if we assume that it is
a fair coin, the probability of this happening is only 3%.
So if we assume that we live in a universe
where the null hypothesis is true, where the coin is a fair coin,
we would have only seen this with a likelihood of 3%, meaning that,
you would have to watch this experiment on 33 different occasions.
Like you watch it today, you watch it tomorrow, you watch it another day.
You would have to watch it on 33 different days or 33 different times.
So this experiment of five coin toss would have to be conducted only once.
You would see this result. So it's a very low probability.
And that's why your feeling of this is some something's going on here
is completely valid.
It's completely justified.
But then let's see what happens.
the coin is tossed again, and this time it's guess what?
Tails again.
Yeah, I know you're probably expecting heads.
Now it's tails again.
What do you think is going on here?
How do you feel about this coin?
Do you feel that it's a it's still a fair coin.
Do you feel that our assumption that it's a fair coin is correct,
or do you feel uneasy about that assumption?
Do you feel that something might be suspicious here
and if you feel that something might be suspicious, here you are again.
Very well.
Just for even more than so than previously,
because probability of that happening is about 1%, 1% only.
And what is going on here is that the probability of this
happening is dropping as you get more and more tails.
And in terms of this course further down, we'll be operating with the term p value.
In fact, we've already seen p values when we looked at the Z-score table
previously when we were looking up the z scores.
Well, in that table in the middle, what you have is actually a p value.
So you can see that the p value is dropping.
That's simply the probability of this happening
given that we're in a universe where the null hypothesis is true.
And this is very important to understand that if we're in this universe,
this top universe, then this is what the p value looks like.
It's very unlikely that you'll get six tails in a row.
However, if you think about it for a second, if we were in this universe
where it's not a fair coin, where it's, for example, it's a weighted coin,
or both sides of the coin are tails that, if you have a coin like that,
if we were in this universe, then these p values would be completely different.
It wouldn't be like this.
The p values would be 100%, 100%, 100%, 100%, 100, 100%.
So in if we lived in this second universe, we wouldn't get that feeling.
We wouldn't get suspicious over here or even more suspicious over here.
We would totally be comfortable with this because we know
that the coin has two tails on each side.
There's a tails on the coin, and therefore we would be totally comfortable with one
one tails in a row.
Two tails in a row, three or 4 or 5 six tells in a row.
We wouldn't get any uneasy feeling about it if we lived in this universe.
However, the way hypothesis testing works is that we assume we live
in this universe over here at the top, the null hypothesis universe.
And then we want to see will we get whatever our experiment is?
Will we get uneasy and uneasy feeling about it?
And this uneasy feeling in mathematical terms,
you can't just go up to your boss and say, or your manager or your client.
You guys just go and say, I had an uneasy feeling about this.
I don't think it's true.
Of course,
you have to put in mathematical terms, and that's
where statistical significance comes in.
So statistical significance is basically this line over here is simple as that 5%.
So that's where you started getting this uneasy feeling just is just after
the 6% you started being very suspicious about things going on.
And that's about right.
That's because as soon as something is 5% less likely to happen,
that's the alpha here, then that is like 5% is like one out of 20.
So that's when we draw the line and we say, okay, so that is it's so unlikely.
I am so unlikely to see this by random that I'm going to reject this
null hypothesis.
I'm going to say that because I'm seeing this
and it was so unlikely to happen at random.
This cannot be true.
And that's where you say that I'm confident
I'm rejecting this null hypothesis with 95% confidence
that I'm 95% sure that we don't live in this universe.
There's a 5% chance that we do.
But I am 95% sure that we don't live in this universe.
I'm going to reject that hypothesis, and I'm going to state
that we live in a universe where this coin is actually rigged,
and you can set this confidence level to whatever you like.
You can set it to 10% and you'd be rejecting over here.
You can set it to 1% you to be rejecting over here.
So it depends on your experiment.
Most of the time 95 is a good level to go with, but
sometimes, especially in medical trials and things like that, where it's,
people's lives depend on that the results
or the confirmation or not confirmation or like whether there is whether or not
we reject the null hypothesis in those cases, you might want to.
So sometimes it's required that you set it to 99%.
So it depends on your experiment and on your
on what you're doing with those results.
But generally speaking this is what statistical significance is all about.
It's the point where in human intuitive terms, you get an easy
about the null hypothesis being true, and you get super suspicious about it.
In mathematical terms, it's just where you draw the line and you say, okay,
I have enough confidence, or like this is sufficient
level of confidence for me to reject the null hypothesis.
So I actually I'm going to state that we live in this alternate universe.