All right.
Welcome back.
This is a two part lesson again and this lesson we will introduce open CV and start working on doing
our first part of the data processing by the end of the lesson.
We will have done the first five steps of our roadmap.
We're going to get up to here.
Well we've got a cropped image.
So as you can see I've got my Python server running on port eight thousand and I've got my local host
up here in Google Chrome with the developer console on the side.
The first thing we're gonna do is include the open see the library in our project and we've seen one
way of doing it which is using a C.D. And here with tensor flow jazz and all we had to do for that was
include this link.
Now I want to show you a different way of including a third party library and it's gonna be more similar
to how we're including our drawing that J S file right here.
So the first thing I'm going to do is I want to right click and create a new folder in my Explorer pane
and I'm going to call this folder vendor and then what I'm gonna do is I'm going to go and download
the open see the javascript file you can find this file under docks thought open C dot org and here's
the version number.
And this is the name of the javascript file open CV dot J S I want to put this link into the course
resources of course but the important thing to do is to save this thing as open CV dot J.S.
once you've saved the file in your Downloads folder all you need to do is need to drag it into your
project folder.
So that's where we credit this vendor folder just a moment ago and we're just gonna drag and drop it
in.
Alternatively you can even drag and drop it directly into the explorer pain and the s code that should
also add it right here in Windows Explorer or in Mac finder so now that we've heard of this entire library
of code and you can see it's quite a bit to our project all we need to do is add a script tag once again
to be able to use our library of code.
So once again it's a script tag that has a source and the source is going to be the relative path to
our open CV file and the relative path is going to be under vendor and then open CBD at J.S..
And you can see the autocomplete here suggesting the file already.
So I don't even have to type this out.
Now don't forget to close the script tag and we're ready to go the next thing we're gonna do is we're
gonna split out the logic where we pre process our data from our index thought each to Mal file and
from the drawing that j ust file where we've got our drawing logic.
So we're gonna right click and create a new file in our root directory and we're just gonna call this
processing dot J.S. and the first thing we're going to do is we're going to take the code that we've
written here in our index that each terminal where we've got our function where we load our model and
we're going to move it over by cutting it and pasting it into processing.
J.S. so let's save this file.
Let's save our index that HMO file and then let's go into chrome and refresh our site now.
What you should see here in the developer console is an error.
You should see an error that reads uncorked reference error load model is not defined.
Now I want to pose a challenge to you.
Can you diagnose and fix this error.
I want to give you a few seconds to pause the video and give this a go.
Ready.
Here's the solution.
The hint the error message is giving me is that the problem is on line number forty five and looking
at our vice code I can see that on line forty five is this load model function so load model is not
defined what that means is that our website doesn't know what load model means even though we've defined
it right here in the processing Doc J S file.
The reason we're getting this error is because our index that each team alpha cannot talk to our processing
file yet.
The part that we've missed out on is adding a script tag to link the two.
So we can do that right below where we've added our drawing not J S file so script is our c is equal
to and then it's processing Doc J.S. There you go.
If I close the script tag save come back into chrome refresh then what we should see is that era.
Go away.
Fantastic.
So now that we've got this working let's add a new function here I want to minimize the load model function
and then I'm going to format this document here.
It's gonna right click and click on format document that cleans up my indentation.
And in order to add the function I want to use the snippet here from this code.
So the function that I'm gonna create is gonna be called predict image.
That's what we're gonna do our pre processing.
And it's not going to take any inputs.
So I want to delete the parameters here inside predict image.
I'm just gonna add a log statement first and it's gonna read processing.
Dot dot dot dot and try and call this function when we press the button.
So inside the index that each time l file we're not just going to clear our canvas but before we clear
the canvas we're gonna call this predict image function.
There we go.
So let's save our changes.
Come back here and let's try and refresh and see if we can spot that log statement.
All right.
So when I press the button and I get an error predict image is not defined.
So we've seen this before right.
What's going on.
Do we have the same problem as before.
Well no because we've added the script tag right here.
So the index that each team l file and the processing that just file should be linked.
Right.
The spelling of our function name here also matches the spelling of our function name here.
The reason I'm getting this error is because I've got unsafe changes in my processing that J S file.
That's why it doesn't recognize this function.
So once I saved this file and come back here and refresh and click the button I've got my log statement
showing up in the console.
Now that we've got that working let's take the next step namely reading and being able to display an
image using open CV.
So what we want to do is if we draw something on the canvas we'll be able to load this image into open
CV.
The key to getting this working is to have a look at the documentation.
So on this page right here you'll see that there are really two key methods.
The first method is called I Am read.
And the second method for displaying an image is called I Am show.
So what we're gonna do is we're gonna use I am read to read our image from our canvas scrolling down
in the documentation.
We can even see this code snippet right here.
So it says let's image equals c v dot I am read.
And then in the parentheses it has image source.
This is the format of the code that we're gonna be using inside our predict image function.
Let's add this very line.
Let's image as equal to c v dot.
I am read.
And then in the parentheses we're going to add the image source and the source for us is our canvas
and that's it.
That's how we load an image.
I think to understand with this line of code is first off we're creating a variable and then we're giving
that variable a value here.
This let keyword is one that we haven't seen before.
But it's very very similar to const or var The only difference is that lat allows us to create this
image variable locally within this predict image function unlike the canvas which is a global variable.
We've created it here in our drawing that J S file and we can access it anywhere in this file and we
can even access it inside the processing dot j ust file image in contrast is a local variable meaning
it only exists inside the predict image function that's what that led keyword is saying for more information
about the differences between let const and var I would recommend heading to this page here from W three
schools on this page you can learn all about the three key words let const and var and the differences
that they imply for the scope of the variable so with var we've often got global scope but there's something
like led we have something called block scope CV is our open CV library and we're accessing our I am
read function from the open CV library that's what this line of code means in a nutshell but what I
reckon is that we should really verify that we've successfully loaded our image so let's create a second
canvas on our website where we can preview our output so what I want to do is I want to create some
canvases just below this check and button that preview what it is that our code is doing.
The way to create a canvas is outlined here on this page here from W3 schools.
So previously what we've done is we've used get element by I.D. to pick out a canvas from our age html
file but we can actually also just create a canvas object using JavaScript straight up and we can do
that with document don't create element and then canvas in the parentheses.
So let's try this out when I add a little comment here and it's gonna read.
Testing only meaning we're actually going to delete this later.
And here what we're gonna do is we're gonna create a constant called let's call it output canvas and
that is going to hold on to our canvas element that we're going to create using document don't create
element.
And then in the parentheses it's going to be single quotes and canvas.
Now that we've got our testing canvas we can use CV dot I am show in order to output what it is that
we've got.
So CV dot I am show.
And then when we put between the two parentheses.
Well for that we should check the documentation once again I am show only needs two things.
It needs a canvas output so it needs to know where to display the image and it needs the image itself.
So the one that we read from the image source so let's add that between the two parentheses.
The place we want the image to show up is our output canvas.
And of course our image itself is actually called image.
That's this one right here.
Now that we've got that we can display it on our website and we're gonna do that simply by appending
our output canvas to the body of our document.
So the code for that looks like this document.
What body don't append child so the canvas that we're testing here is just gonna be appended to the
body using the append child method.
So now that we've got our testing code finished let's try it out.
What I want to do here is going to zoom out a little bit.
So I'm just going to make the website slightly smaller and I want to draw something in my canvas.
Click check answer.
And lo and behold the canvas.
The second one for testing appears below the button and I can draw again and click and you should see
that you're able to generate these testing canvases side by side as they're appended to the body of
the document.
All right.
So where does this leave us.
Well we've successfully managed to load our image the next step is converting the image to black and
white and to a single colour channel.
Now I know that on our Web site it already looks like we're dealing with a black and white image but
this image is actually still in our GP format with three colour channels not in grayscale with one colour
channel which is what we need.
So how do we convert this to grayscale.
Well the open CV documentation is our friend.
Once again what we see on this page here which is entitled changing colour spaces as part of the open
CV tutorial on javascript we see that there is this function here called convert colour CBT colour and
if we scroll down a little bit we see this code snippet right here in this code snippet you can see
the function being called on this line.
And if I hit try it then we actually see this image here being converted to black and white to grayscale
and this looks like exactly what we want.
So let's examine this CBT colour function in a bit more detail so we can see it has one two three four
inputs.
What are those.
Well the parameters for this function are fourfold and they're described in detail up here.
So we need a source image which is our input.
Then we get the output here which is the second parameter.
In this case they're calling it DST for maybe destination then we need to supply a colour space.
So there are a number of different conversions you can make.
But the one we're actually interested in is this one here from our GP to grayscale.
The final thing that we can supply are the number of channels in the destination image we can see that
in the code snippet they've supplied a zero here.
And what this means is that if we put in the number zero then the number of channels is derived automatically
from the code.
So if you're converting to grayscale it's gonna be a single channel but you don't have to memorize that.
You can just leave it a zero and open CV will actually be smart enough to figure it out.
So let's copy this line of code here and in our document insert a few more lines here and pasted in.
Now let's modify the code that we've just pasted in to suit our needs.
So what's our source image called.
Well it's image because that's what we're reading from the canvas.
What about the destination.
What should the output be called.
Now that's a little bit more tricky.
So what should our destination be.
What should our output be.
Well how about this.
How would we overwrite or update our image.
So our image is going to be both the input and the output for this function.
And what this will do then is it will update our image to grayscale.
And then when we show it down here we should see a grayscale image now at the moment our canvas is already
black and white.
So in order to test this let's change the line that we're drawing to a kind of a toxic green like some
sour children's candy or the color Psaki racing colors to do that.
We're going to go into our drawing that J S file I'm going to change the line color to a different hex
code the one I'm after is b c f f 0 0 are you look this up.
This is a lime green color so if I save my changes here save my changes in processing and then refresh
my page then I should be able to draw here in a lime green color.
If you're not seeing that just like me then maybe head over to tools.
Clear your browsing data and make sure you're not working with a cached version of your Web site.
If I refresh this again and I draw on my canvas then the lime green color is coming through.
Nicely perfect.
But what about our grayscale conversion.
Well let's hit the button check answer and indeed we see a gray where we previously had the lime green
so the black and white conversion works.
We have gray instead of green no perfect but one of the things that we should maybe do is we should
increase the contrast between the line that we're drawing and the background so similar to how we are
drawing a white line on a black background.
Let's convert our lines to white programmatically the way we can do this with open CV is we use something
called Image threshold.
What this will do is it will make all the pixels above a certain threshold a different color.
The function that we're after is called CV dot threshold.
Once again it takes an input has an output and then we can specify a threshold value so this is the
cutoff value and the maximum value to use for the different threshold types.
Now I know this isn't a great description but what this will do is basically it will convert anything
above this threshold to the maximum value.
That's what this function does.
Let's take a look at the code snippet below for an example.
So here we've got a threshold that's going to convert everything above the value of 1 7 7 to 200.
So if we try it out we get something like this.
If we change the output to 2 5 5 and refresh then the contrast becomes even more marked and this is
perfect for our purposes because what this will allow us to do is make our edges a lot more defined
which is what we need for the next step.
So coming back into V is code let's add the logic to make all the pixels above a certain threshold white
and make the rest black.
We're gonna use the CVR dot threshold function and then we're going to apply for inputs our image and
then our image again.
So the first input was the source image.
Second is our output our destination and then we're gonna select a threshold value.
I want to go for 1 7 5 which is a kind of gray.
I can actually visualize this for you using a keynote right here.
So if I said all three are GV sliders to 1 7 5 then you can see what I'm setting as the threshold.
So anything above this threshold I wanted to be white I wanted to be equal to two hundred and fifty
five.
So that means that two under fifty five is going to be the next argument to my threshold function.
It's gonna be the value that anything above 255 is set to the last thing I need to provide is how the
threshold is calculated and if we look at the documentation him we see there is a number of different
ways but we can use this one here that they're using in the docs as well.
The binary threshold it's either gonna be one or the other.
So let's go with that and add the semicolon at the end.
So if we save our changes and come back to our website here what I'm gonna do next is I want to show
you a side by side comparison.
So when you use a screen capping tool here and I'm going to take a screenshot of these testing canvasses
done here capture them and then what I'll do is I want to refresh.
I want to draw something on the canvas press check answer.
And now if I pull them up here side by side you can tell that previously we had a kind of light gray
and thanks to our threshold logic that we've added all the lines are now stark white.
But that's not the only difference.
The other thing you actually see is that there are more jagged edges on the canvas here than before.
And that's because there's this marked differentiation between the black and white pixels.
So all the pixels that previously had a bit of a shade.
Now have a stock value.
And this is useful for allowing us to figure out what the bounding rectangle should be.
Now in order to calculate the bounding rectangle what we first need to do is work out what the contours
are of what's been drawn on our canvas in order to figure out how to calculate what the contours are
open C.V. once again has a fantastic tutorial and here on the Getting Started page for the contours
we've actually got a wonderful example that they've got right here on how to find the contours in the
first place.
Looking at the codes that are below.
Let's see if we can identify the most important lines in the code.
So we've got fine Quantas right here and find contours has one two three four five inputs and the key
inputs are of course the source.
And then this thing called contours and then this thing called hierarchy and those to come from up here
the first one this is sort of vector.
And the second one is some sort of matrix.
So it's these three lines here.
Let Qantas equals new CV dot match vector let hierarchy equal new CVD up Matt and speed up find contours
that are the lines that we're interested in as you might have guessed from the previous examples the
output of this function here are the contours.
So instead of having some sort of destination here the contours that are being calculated by find contours
are the second parameter here and the hierarchy is the information about the image topology.
The next thing that we need to specify in this function are a mode and a method.
So some sort of way of retrieving the contours and some sort of approximation method.
And we're not gonna dive into detail here as to like the pros and cons of the different methods.
The example code works just fine.
And to prove it we can hit the tried button here and it even draws what the controls are of this image.
So let's copy these three lines of code and come back into v as code and paste them in.
Now of course we'll have to modify them so our source of course is not called as our C it's called image
but the other two we can live as they are.
One thing I want to do of course is a format that the documents so we get this nice clean indentation
and that puts us in a great place for the next step namely calculating the bounding rectangle and actually
want to leave this as a challenge to you.
So what I'd like you to do is find the pages in the open CV documentation that allow you to figure out
how to write the code in order to calculate the bounding rectangle for whatever was drawn on the canvas
and then what I'd like you to do is use this bounding rectangle to crop the image.
In fact in order to complete this challenge you actually only have to write three lines of code but
you're gonna have to find out for yourself which two lines of code are going to create the bounding
rectangle and then what line of code will help you crop the image.
Now mind you you don't actually have to draw the bounding rectangle on the canvas like I've got here
on the slide.
All you need to do is crop the image to whatever part of the canvas.
Our lines were drawn and I'll give you a few seconds to pause the video hunt around in the open CV documentation
and give this challenge a go
ready.
Here's the solution.
You're gonna have to combine two of open CPS functions.
The first one is this bounding rect method and the second one is this region of interest method with
those two functions in hand you can complete the challenge.
So how does this bounding rectangle work.
This is part of the contours tutorial by the way.
And this is the reason why we worked out the contours first in the code snippet.
You can see that in order to figure out what the bounding rectangle is you need these two lines of code.
You need to get some contours from the controls object and then you're gonna use these contours which
are abbreviated here with S.A. to calculate the bounding rectangle.
Right here.
So these contours which is a subset of all the contours feed into the bounding rect method and allow
you to work out the bounding rectangle.
Now what about the cropping.
Well this is where are you going to use this bounding rectangle.
Right.
If we come over here to the tutorial on the basic operations and open CV we can see that the region
of interest here can be set using a rectangle.
Right.
So we can crop the image using the bounding rectangle that we've calculated from the controls.
That's the trick.
So let's implement this in V.S. code.
We're gonna use the contours which are the output from fine contours and we're going to store the contrast
that we're interested in this subset in a variable called S.A..
So it'll be contour start to get zero.
Just like it was outlined in the documentation and then we're to create a rectangle using these contours.
So we see the bounding rect parentheses S.A. and then we're gonna use this rectangle to crop our image.
So let's update our image right.
This is what we usually do.
We are going to set image equal to image dot Ah.
Oh I reached of interest.
And then as an input the region of interest method needs a rectangle and that's it.
Those are the three lines of code that allow us to crop our image and just to test it out and see it
in action.
I'm when I save the work here.
Come over here refresh and then what I'll do is I'll draw something maybe in this corner right here.
Click check.
Answer and you can see that all the parts of the canvas that were not drawn on were cropped.
If I draw something very large then it's going to crop very very little around the edges so the bounds
of the rectangle are just where our lines touch the edge.
So this is fantastic right.
Any shape you want you can draw and it'll crop it to the edges.
So I hope you enjoy that challenge and this brings us almost to the end of this lesson.
There's just a couple more things to do and that's doing a little bit of cleanup actually.
So I'm not gonna leave this in this lime green here.
I'm going to set my line color back to white.
So I want to come here in the drawing not just file and change this back to six FS which is the hex
code for white and then what we're gonna do is we do a bit of cleanup on the open CV front.
So if you've had a look at the basic operations on images page then what you see actually are a lot
of these notes.
Don't forget to delete CV mat when you don't want to use it anymore for example and this is almost below
every example.
So a lot of these objects here like Matt Matt Vec S.A. are being deleted at the end.
So we're gonna go with this recommendation as well.
We're gonna delete the open see the objects that we're not using anymore in order to free up memory
and kind of comply with the recommendations in the documentation.
So what are the things that we should delete.
We want to add a little comment here that reads clean up and then I'm going to delete my image so I'm
not going to need my image anymore my image object and I'm gonna do that with image not delete parentheses
and a colon but there's a couple of other objects as well namely these controls here.
These controls here and the hierarchy so I'm gonna delete those as well.
Contours don't delete S.A. don't delete and hierarchy delete delete and that's it.
Now we're ready to tackle part two of data processing.
I'll see you in the next lesson.