All right.
Now I've done a little bit of EDTA or exploratory data analysis.
We're going to do one more step and that is building a correlation matrix.
You might be thinking what is a correlation matrix.
In layman's terms it's a big table of numbers telling us how each independent variable is related to
each other.
Okay.
These columns here.
How do they interact with each other.
Are they a positive correlation or they are negative correlation.
Let's not talk about it for much longer because we want to see it.
We want to see what is happening.
So let's go make a correlation matrix and to do so we can use DFT score.
We hit shift tab here compute power wise correlation of columns excluding and a slash null values.
Well power wise correlation fascist basically saying Compare each of these columns to each other.
Well let's see it in action.
If you're looking at this for the first time quite confusing because there's a lot of numbers going
on here okay.
You can see the columns are on the rows as well as the columns and the diagonal here is 1 1 1 1.
There's a lot of decimals and some are negative and some are positive.
What we might do though instead of staring at this and exploring it for a little bit we might make this
a little bit more visual so let's make our correlation matrix a little prettier.
We're going to do that using C Bond's heat map so we'll set it up in a variable so call matrix equals
DFT.
Cool.
Wonderful.
And then we'll set up a figure and an ax using what is a penalty not subplots.
That's a fig size I kind of know the fig size because I've seen this pop before.
So I want it to be 15 width and height and then we're going to go X equals s and s dot heat map would
help if I typed map and then I'm gonna pass it call matrix and we want to annotate it so we'll set a
not to equal true.
And we also want a line with line weights.
Yeah I believe that set of zero point five.
And then what else do we need.
There was a couple more things format equals we only got two decimal places.
So see here these like six so twos two is gonna be enough for us.
And then see map to give it some nice colors will go yellow green blue.
Wonderful.
Let's let's check it out.
Invalid syntax Oh forget the comma here.
Classic Wonderful.
Okay.
So what is this showing us.
We've got some cut off things here that's that pesky map plot lib three point one point one area.
That's right.
We can fix it so might go bottom.
You might not see that if you're using a lighter version of that plot lib then what I'm using at the
moment we need to adjust the y axis limitations so we're going to get Y lamb.
Wonderful.
And then we're going to go X still set y land that's that's it that's how we change it we to move the
bottom up and we want to move the top down.
That makes sense.
So bottom plots zero point five and top minus zero point four.
Does this work wonderful that's looking a little bit better it's not cut off and this little adjustment
here this is from the research that I found by just just searching for it.
So map plot lib seaborne first I think that my history might have had it there plot lib There we go
first and last row cut off.
That's an example of searching for your problem when you have it right.
When you run into an issue it's just about trying to figure it out.
So they've got a similar problem to what we just had cut off and then come down here you can find this
is the lines of code that I'm using here and in fact they're making a correlation matrix as well.
Beautiful yeah.
Unfortunately map plot lib three point one point one broke seaborne heat maps.
Well that's all right.
We've got our heat map we've got our correlation matrix.
Again this is just the same version of this a more visual format.
I look at this and I understand this one a lot better than I do this one.
So what is actually going on here.
Well we can see on the diagonal we've got a lot of one values.
But what this means the diagonal is we might zoom out one tick so we can see the diagonal.
Is giving a perfect correlation which means the age column is exactly equal to the age column which
makes a whole bunch of sense right.
And the sex column is completely equal to the sex column and so on and so on so on all the way down
the diagonal here.
But then we have some other a lot of other actually little cells here.
We can see on the side you got 0 up to 1 and some of them are positive like this one here.
And some of them a negative we just compared chest pain type which was C.P. to our target.
Let's investigate that.
What's the correlation here.
I got C P.
It's got a positive correlation so zero point for three.
Remember the maximum value is 1.
But this is telling us how.
Chest pain is related to the target column.
Well if we're looking at this and we did some research on a correlation matrix I'm going to tell you
because I've done this research myself a higher positive value means a potential positive correlation
and a higher negative value means a potential negative correlation or a decrease.
What this is saying is as C.P. goes up the target value also increases now.
Does that make sense.
If we come up here so if we look at this plot that we did before.
So as CPI goes up the target value also increases.
So remember our target value is 0 0 1 0 for no disease 1 for disease now were most interested here in
the correlation because if you looked at there's no disease is highest at zero so that means no disease
the target is zero.
And chest pain type is also zero.
Okay.
So according to our correlation matrix as C.P. goes up because it's a positive correlation the target
value should also go up.
So let's investigate disease.
This is a 1 value and this is 0.
So blue is one salmon is zero.
Salmon.
What a beautiful color as our chest pain type goes up 0 1 2 3.
Does the blue column for one of target value go up if we look here it does go up slightly and this drops
off dramatically and then it goes up here it goes down here but the ratio of disease to no disease is
still much higher.
So that's where the positive correlation comes from it comes from chest pain type increasing or LCP
column as that increases our target column meaning disease or no disease also increases.
So that's a positive correlation that's why it's positive here.
Now let's have a look at one of the negative columns so you've got a negative correlation here.
It's about the same ratio but just in a negative way.
That means if we look here x sang we'll check our data dictionary for that in a second as X and goes
down.
So should the target value let's not take this correlation matrix word for let's reproduce this plot
except using the x and column of what we do that we might jump up here.
What is X Ang exercise induced and Gina if you wondering what and Gina is we've checked out that before
a condition marked by severe pain in the chest.
So this means that if you exercise do you get hot pain and let's have a look exercise induced angina
one equals yes zero equals no.
So do you get hot pain when you exercise if you do.
That's a one.
And if you don't that's a zero there's a negative correlation according to our correlation matrix I'm
saying correlation a lot I know.
So X sang compared to the target value.
Do they have disease or not is negatively correlated.
That means as X and goes down the target should also go down.
Now there's only two values here for each of these so this is actually a pretty straightforward example.
So if we think about this based off what you've heard about in terms of heart disease if someone was
telling you that when they exercise they get chest pain so they would have one.
Would you predict closer to someone having heart disease or not if they had exercised induced chest
pain.
What would you be more inclined do they have heart disease or they don't well for me I would be more
inclined to say well based off the rest of the data if they're getting exercised induced chest pain
it could be something going on with their heart right.
They could be heart disease.
So this is where there's a negative correlation.
This is where the negative correlation comes into play.
Let's go down here back to our correlation matrix there's a negative correlation of negative point for
4.
So that means as X and goes down the target value goes down.
Based off what the call function is seeing in our data exercised induced angina when I compare that
column to the target column I'm noticing that when X Ang is lower.
So is the target column based off what we're looking here.
It's and remember it's only a point 4 4 correlation it's not a perfect want like negative 1 correlation
but just it's trending towards.
If someone didn't have chest pain so didn't have exercised induced angina the trend is negative.
So it's trending towards them not having heart disease does that make sense.
And you could go through all of these values.
So this is pretty flat like this is almost basically zero.
There is no correlation between FBI s and the target.
While our data dictionary telling us FBI s so fasting blood sugar so this is saying is the fasting blood
sugar above 120 milligrams per deciliter.
I believe that might be deciliter one of those shoes Eric was false.
So let's basically saying they're perfectly even.
There's no real positive or negative correlation it just it is what it is.
And so that's the kind of insight that you can start to form with this correlation matrix and that's
a kind of insight that we want to sort of figure out on our own if we want medical experts and we could
only work with the data that we have.
This is the purpose of ADFA.
This is the purpose of what we've gone through here comparing different variables looking at the distribution
comparing more different variables here.
And this process we could keep going with this Ryan.
We can keep going until we really really understand the data but what we want to do next is model driven
EDTA.
So what that means is building a machine learning model to derive insights of how these independent
variables here contribute to the target variable.
Now correlation matrix is one of them but we're more interested in trying to predict for the future
the correlation matrix only deals with the data that we have.
We're more interested in building a machine learning model that has predictive power based on the data
that we have to predict on things in the future.
So that's what we're going to look at next.
If you a little bit stuck here of what the correlation matrix is.
Don't worry it's a pretty in-depth concept too to understand but where you can probably go next is by
just searching what is a correlation matrix.
So you're here.
What is a correlation matrix.
You could go to this first link a correlation matrix is a table showing correlation coefficients between
variables.
Well that sounds like a good resource start to read up on a little bit more.
That looks a little bit like with the graph that we made but we've covered most of the crux of a correlation
matrix.
It's really just going.
Okay.
Looking at two variables and going is the relationship positive a.k.a. as one increases so does the
other.
Or is the relationship negative as one decreases so does the other but let's have a look at our workflow
see where we're up to we've been through some data analysis now it's time to get onto the machine learning
in the modeling section.
Let's see where we're up to here.
If we look at our new project workflow we've covered this we've looked at some exploratory data analysis
using num pie pan as a map plot lib.
Now it's time to get into the machine learning part of things using psychic learning.
We'll start to test our little theories about how what we found out an exploratory data analysis we'll
start to see does our machine learning model agree with what we've found here.
Just exploring the data on Alan.