All right.
So here we're going to be visualizing our results.
So I'll add a quick markdown sale here visualizing the results and in the next few sales we're gonna
be creating some charts scrolling to the top of our notebook.
We should have already added Our input statements for Matt Gottlieb and Seabourn hand.
We should have also added this percent.
Matt plotted in line the charts I want to create are the joint conditional probability that an email
is spam vs. the join conditional probability that an email is legitimate.
So in this first cell here that's some chart styling info.
And by that I mean things like the y axis label the x axis label stuff like that.
So y axis underscore label will be equal to single quotes P parentheses x pipe spam and the x axis on
a score label will be equal to single quotes p x pipe non spam the first chart we're gonna create is
gonna be not very fancy it's gonna be your standard map popular chant and it's gonna look like this
with peal T dot figure I'm going to set the size of my chart first so I'll see fake size is equal to
parentheses eleven by seven.
This will give me a nice size for the screen cast BLT dot X label parentheses it's gonna be equal to
that string that we have at a cell above.
So x axis underscore label comma font size is equal to 14 and peel T dot y label.
It's gonna be you guessed it y axis underscore label comma font size equals fourteen.
Now let me actually plot this chart so I mean he's a scatter plot here peel T dot scatter parenthesis
joint at a school log on a school ham.
So this is going on our x axis comma joint and a score log on a score spam is gonna go on our y axis
and peel teed up show will generate our chart we'll see what it looks like what it looks like.
This pretty wild huh let's zoom in a little bit on this area here and set the limits of the chart.
Also I'm going to change the color of my dots here to navy blue coming back up I want to set the scale
or the limit of the axis with PDT dot excellent parentheses square brackets minus fourteen thousand
two one and PDT dot why Lim is gonna go from also minus fourteen thousand to one to change the colour
of the dots all I need to do is apply the color argument and provide a color name here.
And one example of a legitimate name is Navy for navy blue.
If I refresh the cell then you can see that our chart now looks like this.
Not bad right but I think we can make some improvements.
Let's have a think about how our naive bayes classifier made his predictions.
Scrolling back up to this very crucial line of code here we can see that our algorithm decided on which
category an e-mail belonged to simply based on which probability was higher.
And this allows us to plot something on our chart called a decision boundary that decision boundary
is going to divide our chart into two regions one region.
It's gonna be for spam and the other region is gonna be for our legitimate emails now.
Any guess of how our decision boundary will look like on our chart.
Well let's think about it.
If the probabilities for one of these e-mails is calculated and it comes in in the top left corner should
be classified as spam whereas non spam well in this case it has a very high probability that the email
is spam and a very low probability that the email is non spam right.
Therefore according to our logic and our decision rule it should be classified as a spam email.
What about the lower left and the lower left.
We have the opposite case here.
An email is much more likely to be a real email than a spam email.
Such be classified as non spam.
So by that logic if we were to draw our decision boundary a line to separate these two regions then
it would go diagonally up through the middle of our chart right.
This is gonna be our decision boundary and this is how we can visualize the logic that our classifier
uses to separate what it believes to be a spam message from a non spam message.
So let me add a very quick section heading out it reads the decision boundary and what I'm going to
do is I'm going to copy this cell here and I'm going to paste it below and then what I'll do is I'm
going to create some data for a line and I'm going to do that up here actually to plot a line on a map
flip chart.
You need some data points and I'm just gonna call these line data and when to generate them using num
pi.
So with NDP dot Lin space parentheses I can generate these data points and I'm going to set a starting
and an ending point for these data points.
The start is gonna be equal to negative fourteen thousand which is what we're using for our axes right.
And I'm going to stop at say 1 and then I'll decide on how many data points I actually want to generate
I'll generate say a 1000 of them.
I think they'll be good enough.
Let me add the and u m equals 1000 to make this a named argument.
Now let's shift enter on the cell and I can come down here and modify the second chart all I got to
do is use PDT dot plot and supply an x value so it's going to plot all the X's under line data and it's
going to plot all the y values under line data as well so our x's and our Y's are going to be equal.
This will give us a diagonal line and to contrast that with the navy blue.
I'm going to use the orange color colors equal to orange.
Let's see what this looks like.
There we go.
Okay so we've plotted our probabilities now but this chart is still not very clear.
Let me ask you this.
What kind of problems do you spot with this chart.
Well for starters there is a lot of data in this top right hand corner and we can't actually see what's
going on.
The second problem is looking at this chart.
We cannot identify which emails were classified correctly and which e-mails were classified incorrectly.
All we see is our data plotted here and our decision boundary in the middle.
So first things first let's tackle that first problem I described.
This was the problem of over plotting when we have many many data points and we put them all together
on a chart like this.
They're so close together that we're faced with this over plotting problem we can't actually visually
distinguish the relationship on the chart with so much data.
If you wanted to make this chart a bit more clear how do you think we can improve this visualization.
Hit pause on this video and maybe try and track down two things that we can do to make this chart a
bit more clear.
I'll give it a few seconds to do just that.
All right.
So first we're going to use some techniques that we've already introduced in the past listen.
We're going to introduce some transparency to our data points.
And we're also going to reduce the DOT size on this chart.
And this is actually really really easy to do because all we need to do is add a few more keyword arguments
here.
The transparency we can set with alpha.
So if I set the alpha value to zero point five and refresh myself then I can see that the chart now
looks like this and we can better appreciate some of the density in certain areas to make the dot size
smaller.
On the other hand all we need to do is specify a value for this argument called S for size.
And if I said s is equal to say 25 then I get something like this I get smaller dots and I've improved
my situation somewhat but you know there's still a bit of a problem here in this top right corner.
We've got a lot of data here.
And even though we've made a picture a little bit more clear we're not quite there yet.
So let's do something else let's have two charts side by side instead of a single chart most of the
action seems to be happening up here.
So you can create a little zoomed in version of that area and place it next to our chart showing the
whole thing to do that in my plot lib we're going to use something called a subplot so I'll take the
cell copy and paste it below and now I'll modified of it.
So for my figure's size I want to change it to say 16 and seven.
So when I go a bit wider here because I want to have two plots and then I'm going to create my first
subplot.
So with BLT dot subplot parentheses and I'll supply three values the number of rows in my subplot there'll
be one row the number of columns.
So I want two charts.
Next to each other.
So I'll have a two there and an index for the first chart.
So this will be index 1.
If I copy this line and come down here and paste it in then I'll use INDEX TO HERE.
SO THIS WILL BE chart number two.
That's going to go here up here.
We're going to be working with chart number 1.
So tell you what.
Let's keep chart number one as it is when I want to do is I'm going to copy these lines of code come
down here.
Paste them in and then to set the scale for that second chart.
I'm not going to go between negative fourteen thousand one I'm going to go between minus two thousand
and one for both my x axis and my y axis and the other thing I'm going to do on my second chart is I'm
going to change the size of my dots.
I want to make them even smaller so as equals 25.
Probably too big.
So I'm going to go with a different number here a much smaller one say three.
And now let me hit shift enter and see what this looks like.
There we go.
Now I've got two subplots I've got them side by side and I've got this zoomed in version here where
I've got many many small dots to represent each data point So yeah these are some very very easy and
very handy techniques to apply to try and tackle this over plotting problem but let's move on.
Let's try and make our charts even better and to do that we're going to use Seabourn instead of matte
plot lib to create our charts.
One of the reasons I quite like using Seabourn is because it makes creating beautiful charts very very
simple.
There's not a lot of code you have to write.
And because we want to do a better job with our data visualization see what makes it very very easy
to add colour and make our charts a bit more clear so let me come down here and I want to use a cell
here to add some information about chant styling and the first thing I'll do is I'll set the style of
our seaborne charts.
So I'll see S.A. dot set up a school style single quotes white grid and the other thing I'll do is I'll
create a variable called labels which will hold onto a string called actual category.
The thing about some of these seaborne charts is that they really like working with data frames so we
can create one here call it summary and a score def that will hold on to all our data that we're interested
in plotting.
So PD dot data frame capital the capital F parenthesis curly brace y axis label colon joint underscore
log on to scroll spam comma x axis label colon joint on a squat log on a school hem you can see here
what I'm doing is I'm creating a dictionary and this dictionary is what's going to provide the columns
for our data frame y axis label it's gonna be the column name of all our spam probabilities x axis label
it's gonna be the column name for our ham probabilities and labels it's gonna be the column name for
y on a school test so it's gonna be the column name for the true values the actual values in the testing
dataset now we're ready to go let's create our first seaborne chart to do that we're going to use C
bonds l m plot so S.A. stopped and then plot parenthesis X equals X axis label.
So this is all the x values in our data frame Y is equal to y axis label common data is equal to summary
underscore D F.
So there you have it.
We're using our data frame and we're specifying which columns of our data frame make up the y values
and the x values.
Now let me specify the size of the chart that I want.
Size is equal to say six point five.
Then I'll just say as an S start PDT dot show.
Let's see what we get.
So we get something like this.
What you see here is a regression right.
We've got a regression and we've got a warning here so let's modify our arguments here.
And the first thing I'll do is I'll get rid of the regression line.
So say fit and school R E G is equal to False and that will remove our regression line and also removes
this warning.
So this is how our chart looks like without the regression line using l m plot.
Now I know what you're thinking you're thinking man this looks a lot worse than my plot led.
Why are we doing this.
Well you'll see in a minute.
So um let's add some keyword arguments for our scatter plot and we can do this by providing a dictionary.
So with the scatter underscore K W S is equal to curly braces.
Single quotes Alpha colon zero point five comma s and single quotes colon twenty five.
You'll recognize these because these argument names are what we've used with map plot lib right.
And uh similarly we can also scale this chart differently right.
With peel T dot X limb parentheses square brackets C minus two thousand and one and peel T dot while
in parentheses square brackets minus two thousand coma 1.
Let's take a look now.
I've actually got an error here a syntax error.
You might already be spotting it but uh I came down here on a new line without inserting a comma to
separate my arguments.
So adding the comma and refreshing the cell allows me to generate my chart as I intended.
There we go.
So this is looking a little better but I think it's still missing our decision boundary.
So let's add that as well so I'll add this line here.
After X limb and while I'm all that PDT dot plot parentheses line data comma line data comma color equal
to single quotes black I think will look nicely.
Yeah.
This makes our chart look very clear.
So right about now we've kind of caught up with that plot live in terms of looks but has promised we
want to add some color to this chart right.
The way we're going to color this chart is using a keyword argument called Hugh so I'll just copy the
same pasted below.
So we have the comparison between the two.
And I'll add this Hugh argument here.
And for the Hugh we're going to specify the true values in our summary unescorted.
Yes the true values of our data in our data frame are y underscore.
Test values.
That is.
So that's gonna be Hugh is equal to labels and heading shift into this is what we get we get some green
values up here we get some blue values done here and we even get a little legend here which has the
column name of our data from actual space category and 0 the blue ones are the non spam messages and
one are the spam messages.
The nice thing about this alleged plot function is that we can actually change these markers as well.
So you know not only can we get two different colors we can actually get different markers.
If you were to head to the mad plot lib documentation you can see that you have a whole menu of markers
to choose from.
So if we want to go with circles or triangles or squares or whatever stars you've got a whole lot of
options here.
So I have to set some markers as well.
So we've got color and shape as a differentiator.
And for the markers I'll go for Marcus is equal to square brackets single quotes 0 comma single quotes
X so now my spam messages will be marked with all X's little crosses and my non spam messages will be
marked with little circles like this.
Now there's actually two things I still don't quite like about this chart.
One thing is that I'd like to have a little bit more control over my legend here on the right.
I'd like to customize the legend for this chart.
The other thing that bugs me are actually the default colors that Seabourn has chosen for me.
At the time of recording the colors that were chosen here are blue and green.
Now that might not seem like a big issue but it could be a problem for approximately one and 12 men
or one in 200 women.
And the reason is is that certain color combinations actually not ideal for people with color blindness
and you can take this into account when you're creating your own graphics or your own charts for projects
that you have to show people or that you want to make look beautiful.
And one rule of thumb that I follow is I tend to avoid the red green combo the green blue combo and
the green brown combo.
If I if I remember because these are some of the common combinations of colors to avoid in your charts
if possible.
So if on the slide you have trouble seeing this like w the 42 and the 71 I've uh I've got bad news for
you you might actually be colorblind.
So I hope you're not finding this out on the course but you know you're not you're you're not alone
and very very famous and influential people are also colorblind.
So you know if you're creating your charts and you're presenting to the likes of Keanu Reeves or Mark
Zuckerberg or Bill Clinton they're gonna have difficulty differentiating between the reds and greens
in your chance.
So it's probably not what you want to pick and that's why I suspect it's not a coincidence that the
predominant color of Facebook is actually blue now Seabourn actually has a pretty nice choice of color
palettes that we can use in our projects and some of the default color palettes are found on this part
of their documentation if you scroll through here you'll see that these color palettes have names like
HLS or such two or paired or what have you and we can try them out in our notebook here by adding an
argument here to our little plot function called palette.
And if we set palette equal to say HLS and refresh ourselves we get this look right here.
If we set our palette to set capital S too then we get this look here orange and green.
And if we set it to see head then we get this look here.
Dark blue and light blue I actually quite liked HLS so I'm going to stick with us.
And now I'm going to add my own legend the way you do this is with Petey dot legend.
And there we can specify what should be inside the legend.
So I wanted to have three things I wanted to explain the decision boundary the non spam messages and
the spam messages.
So within the parentheses I'm going to add another set of parentheses and I'll add three strings right
decision boundary column single quotes non spam comma single quotes spam and after this first closing
parentheses I'll add a comma and I'll specify the location where I want this legend to be.
So LLC c is equal to lower space.
Right.
And now I'll have to specify as a font size.
So I'll go with font size is equal to 14 and I'll refresh myself.
Now what I get is I have the legend that I've created right here.
I have got the default legend a little bit to the right.
So we need to remove this default legend here and we can do that simply by specifying an argument under
Ellen plot called Legend set that equal to big capital F and A S E false.
Don't forget the comma afterwards.
Otherwise you get the same error I had earlier.
And if you refresh your chart should look like this now coming back to the colors were of course not
limited to using the built in pallets from Siebel.
Let me show you how you can create your own custom palette.
The process is actually quite simple.
You just have to stick the hex codes that correspond to the colors that you want into a Python list
and then feed that list into this Elm plot function as an argument now for this example.
I want to take some inspiration from some famous artists.
So I want to go to a website called Color list dot com and the lady behind this Web site basically took
a bunch of artists looked at their paintings and now the hex codes for us to use.
So I'm gonna go to r and I'm going to look for the troubled American painter by the name of Mark Rothko
and scrolling through here I can see the color palette of two of his paintings.
But looking what's available here I might actually change my mind.
I'm going to go with somebody else instead maybe see Mondrian.
See if she's got Mondrian.
There we go.
Mondrian.
This is a whole lot more promising.
Let me take the red and the blue from the Broadway boogie woogie from Mondrian.
I'll just copy this hex code here come back into my Python notebook what I'll quickly do is I'll copy
the cell pasted below.
So I have my own copy and then I'll create a variable here called My of colors and I'll set that equal
to two empty square brackets with some single quotes inside and this is where I'm going to paste my
hex code for my red turn and then I'm going to come back take the maybe the light blue and I'll come
back here and I'll put this one here as perhaps the second hex code in my list so I've got my red and
my blue and now all I have to do is change the value here of my palette argument from HLS which was
the inbuilt seaborne color palette to my underscore colors and then I'll refresh the cell.
Brilliant.
So now we can see that the red is showing up here and the bottom triangle and the blue is showing up
here on the top triangle and you know what the thing is I reckon red actually suits spam emails better
than does Blue so I'll have to do to change which color corresponds to which category is change the
order in my list.
So I take my second hex code and put it as the first text code and then I can refresh myself to see
how we're doing now.
Brilliant.
So this is a lot more interesting tell me thing I'll do now is I want to zoom in a little bit on this
top right corner.
And the reason is is that there seems to be a lot of this sort of red fringing going on there seems
to be some crossover.
So this area looks very very interesting to me and I'm going to change this value here on my axis from
minus two thousand to 500 and we'll see how that looks.
Now not bad not bad at all but perhaps maybe a little bit too faint.
So when I take my alpha value we change it to say seven and I think that is a lot more clear.
We can see the red Xs in this blue area much much better now how I know that a lot of these modern artists
on exactly known for their data visualization and I don't really get modern art but I do think that
sometimes these guys do have some pretty cool color combinations and I do think we can draw some inspiration
from that.
So yeah check out color Lisa and let me know if you have any other favorite resources for your projects
in the comments on this video.