We're almost done pre processing our data.
There's just a few more steps to go.
We've just added the padding.
SO NOW WE HAVE TO FIND THE CENTER OF MASS.
Shift the image and normalize our pixel values.
So how we gonna find the center of mass.
If we head over to the documentation under the control of features and we read this first sentence then
we see image moments help you calculate some features like the center of mass of the object.
So this looks like a clue right.
Scrolling down a little bit we actually see that we've got this bit here where we can extract useful
data like the centroid the centroid is just another word for the center of mass that we're actually
after and what we've got here is the code and the calculation for how to calculate the center of mass
and what we see here is that the center of mass really has two coordinates.
It's got an x coordinate and it's got a y coordinate open CV really makes this easy to work out it just
takes a few lines of code.
Now looking at this equation we can see that what we'd really need are three things we need this m 0
0 we need the M 1 0 and we need the m 0 1.
These things are called Image moments and the code to work these out is given above in the sample but
what is an image moment a moment sounds like a really scary word.
When you first hear it but it's just a number that's it.
A moment is some kind of weighted average.
And if the words weighted average make you think of these statistics and probability that we talked
about in this course then you'd be spot on because guess what the mean is actually the first moment
of the probability density function.
So if you think about it this way it's not so strange that the center of mass is also a kind of average.
And we can even see how it's calculated in the documentation right here.
So what is m 0 0 0 0 is the mass for the image as a whole.
So we've got a black and white image and that means that m 0 0 is really just the area of our image.
M 1 0 is how much mass is on the horizontal or in the x direction m 0 1 is how much mass is in the y
direction or on the vertical and if we look at the fraction of these then we get the coordinates for
the center.
And that's really all there is to it.
Let's scroll up and take a look at the code to work this out.
What we see here is that once again the process involves first finding the contours and then using these
contours we can figure out what the moments are.
And once we have the moments we can figure out the center of mass.
So let's implement this in this code.
I'm going to come down here and add a little comment here that just reads center of mass.
Just to delineate where we are in the code and here we're gonna find our contours and we did this last
time with C the DOT find contours and it had a couple of inputs that it needed and we can even scroll
up and we might be able to actually just copy paste what we have here and change it a bit.
So I'm just gonna copy this line and say as the typing him by pasting it in and then what we can do
is use these contours that we have as an output from this function to get some specific contours right.
So S.A. is gonna be equal to contour start get and in the parentheses we'll grab the first set of contours
so we've done this process once before.
This is the second time we're doing it and even though it might look like we're repeating ourselves
we're actually working with a different image at this point right.
You have to think about how all those code runs we're continuously updating and manipulating our image
object so when we're finding the contours on line 53 we're dealing with a very different image than
when we were doing it on line 21 so now that we have the controls that we're after we can grab the moments
and I'm going to store these in a variable called moments and I want to set that equal to CV dot moments
and in the parentheses out to supply the contours from the line above and a boolean and this one is
gonna read false.
Now why is that.
One reason is that I'm pretty much doing what we've got here in the documentation but with the second
parameter is actually referring to is whether or not we have a binary image but that's not the case
for us.
We're working with a pretty much a regular image not anything in binary format.
So that's what we are sticking with the default false.
All right.
So now we have our moments object that we can use this thing to figure out the x and the y coordinate
of our center of mass.
And we're gonna go over the same formula that's outlined here in the documentation so I'll critic constant
here called C X and this is where I'm going to store the x coordinate of the Central Mass.
And in order to calculate the x coordinate I'm gonna grab my moments object and there I'm going to grab
m 1 0 so the mass in the horizontal in the x direction and I'm going to divide this by the mass of the
image overall.
So that's an 0 0 and then what I can do is I can do the same thing for the y coordinate so that C Y
and it's very very similar in the calculation except that we're going for an 0 1 instead of the n 1
0 as above.
And the denominator of course is the same as before.
That's our calculation.
Let's print out this information into the console so that we can look at it and get a better feel for
what it is that we're actually working with because yeah we talked a little bit about what these things
mean but I think seeing is believing and seeing a few examples really helps build an understanding of
what it is that we're actually working with.
So let me log these three values so when I use the tactics and here I'm going to say that m 0 0 is equal
to.
And when I use the string interpolation moment stopped and 0 0 and then it's simply going to put the
x coordinate and the y coordinate in this console log as well.
All right.
So now we can save and refresh our site.
You're gonna need your HDB server running as always in your folder and you just hit refresh.
And now what we're gonna do is we're gonna go through some examples.
So I'd like you to make some predictions with me about what the value of m 0 0 0 is gonna be what the
value of c y and C X is gonna be.
So I'm going to draw a big fat zero in here big fat circle and I'd like you to make a prediction.
Do you think that m 0 0 will be high or will it be a low value.
Remember that this is gonna be a proxy for the area right the mass of shape that I've just drawn here.
And you also have to remember that the largest possible value that we can have in our area for our image
in the center is 20 pixels by 20 pixels.
So that gives us a value of 400 at the upper bound.
Now what do you think we're going to come in.
Do you think we'll come in close to 400 or we're gonna come in at 0.
Or do you think we'll come in somewhere in the middle.
I'd like two hundred so maybe get a pen and paper and just write down your guess the other guess I'd
like to make is where you think the y coordinate and the x coordinate will come in for the Central Mass.
So here the numbers are gonna be between 0 and 28.
Right.
So the whole canvas is gonna be 28 by twenty eight right including the padding.
So make a guess where you think X and Y are gonna come in all right.
My hat check answer here.
And let's take a look at the log statement.
What are we saying.
Well the first thing that we can see is that n 00 the overall mass or the area of our image is going
to be quite large.
Actually it's quite close to 400.
It's well above 200.
I mean not quite at 400 but it's fairly big.
And that makes sense right.
The circle that I drew had a fairly large area.
What about the x and y coordinate of our center of mass.
If we look at these two numbers then what we see is that they're actually very close to 14.
Right.
So the center of mass for the circle that I drew was pretty much in the middle of the canvas.
So that makes sense right.
Let's try a different number.
I want to draw a one here and I'd like you to make the same predictions do you think that m 0 0 will
be high or low in this case.
And then also what do you think the center of mass is going to come in for this.
All right.
So we had check answer and what we see is that we're much closer to zero for the mass of the image.
So this no one here has a much lower mass than the circle.
The zero that I drew earlier but if we look at the x and y coordinate of our center of mass we're still
in the middle.
And I think that makes a lot of sense right.
Let's try a different number.
I want to draw a nine somewhere to go over here and drawing nine.
And for this number I want to ask you a slightly different question do you think that the x coordinate
of the Center will be greater or less than 14.
What about the y coordinate for the center of mass.
Do you think it'll be greater or less than 14.
Remember 0 0.
The Origin is going to be in the top left corner it's going to be over him.
All right.
Let's find out.
So what do we see.
We see that the x coordinate is pretty close to the middle.
Right.
It's almost that 14 but the y coordinate is way less right.
So we have that number nine which has that circle at the top.
And that means that the mass of the image is closer to the top of the canvas than to the bottom.
That's why the y coordinate is less than 14.
It's higher than the midpoint.
So I hope that helps demystify how these moments behave and how this center of mass behaves.
You can have a look at a couple of other examples on your own.
Like say maybe the number six or the number seven and see if you can predict how these three numbers
change for those numbers as you're drawing these shapes on the canvas.
The next thing that we have to do having found the center of mass is shifting the image we have to shift
our digit so that the center of mass is in the center of it image.
And the reason we have to do this is because this was done to the training data that we used for our
neural network.
So what this means for us is we have to figure out how to shift our image.
We have to figure out what the shift should be in the horizontal direction.
And we have to figure out what the shift should be in the vertical direction.
So how many pixels left or right and how many pixels up or down.
So let me ask you this if sea X the x coordinate for the center of mass is at 15 then what should the
horizontal shift be that we need to do in order to center the image on the horizontal.
Well our midpoint is at 14 right.
So if sea X comes in at 15 then we need to go one pixel to the right.
Let's write this down in code.
So when a critic constant you call it X shift and once that that equal to 14 minus C X.
All right.
So that's going to be the difference between the midpoint and our center of mass or at least the x coordinate
for the center of mass Now this isn't necessarily banned but I think we can improve this a little bit.
What if we use our image object instead.
So instead of this 14 here what I'm going to do is I'm just gonna grab the image and pick the number
of columns in the image and divide those by two and subtract the x coordinate for the center of mass
from this value.
So I think this is an improvement but once again we can't shift our image by fractional pixels so we're
gonna have to round this whole thing and we're going to use the math functionality him and round this
whole thing to the nearest integer.
So with this line of code we have this shift nailed down in the x direction.
And as you can guess in the y direction it's very very similar except we're not going to grab the image
columns.
We're gonna grab the image rows and instead of subtracting C X we're gonna subtract c y.
All right.
So now let's figure out how to shift the image around using open CV.
If you had to this page right here the one on geometric transformations of images and you scroll down
a little bit then you can find something called walk a fine.
So you've got this section here called a fine transform.
Example and this is what we're after.
So I'm going to click try it and you can see that this indeed even though it has kind of a strange name
a fine transform actually does what we intend to do.
It shifts the image but how does it do it.
Let's take a look at the parameters of this wobble fine.
We've got a input image.
We've got an output image.
We've got the size of the output image.
All of these look fairly familiar.
Right.
These are all the kind of parameters that we've encountered before.
So no surprises there.
We've also got a border and a border value.
We've also worked with these before.
So no mysteries there.
The only thing that's a little bit tricky to understand is this thing right here.
This transformation matrix.
What is this.
Well let's find out in the sample code.
I'm going to come down here and I can see that the transformation matrix is this thing called M and
I can see that M is being created up here and it's credit using these six values right here.
So what are these.
Well let's take this one hundred him and change it to 80 and then click try it.
And what we see is that the transformed image just moved up.
Let me change that to 40 and hit.
Try it and again it moves up even more.
And if I change this to two hundred and click tried it moves down.
So this 200 him is in fact the vertical shift from the origin and the top left corner changing this
number him shifts the image up or down.
And what that means is that this 50 him is going to be the horizontal shift.
If I increase it to 150 were star image b it's going to be somewhere here.
Right.
That's fairly simple.
Right.
So now that we understand this we can come up here where we've got the formula and we can make a connection
between this formula here and the code in the example what they're calling T X here.
It's nothing other than the horizontal shift.
And what they're calling t y here is nothing other than the vertical shift at this point I'd like to
pose another challenge to you.
What I'd like you to do is use this documentation to implement the shift to the center of mass in the
s code.
I'd like you to shift the image so that it's now going to be centered using the center of mass.
I'm going to give you a few seconds again to pause the video before I show you my solution.
All right.
Ready.
Here's the solution.
The thing to realize is that one of the inputs that we need in order to use this warp fine is yeah.
We need to provide them.
But we also need to provide some sort of size of the image.
So what we can do is we can reuse this new size variable that we've created earlier and we can just
kind of give it a new value when to give it the current size of our image.
Image dot calls for the width and image dot rows for the height.
So now that we've updated our signs we can create a ha transformation matrix.
So I want to credit constant and call it m just like in the documentation and when I create this from
the array.
So when I create a matrix from an array and actually you can copy this entire line here and pasted into
v as code the only modification that you have to make is the horizontal shift and the vertical shift.
So that's gonna be x underscore shift for the horizontal and Y underscore shift for the vertical.
And that's the transformation matrix done.
The next thing that we need is our warp of fine function.
So once again I'm going to copy this to save myself some typing pasted in and then I'm going to modify
it the input of course is our image the destination the output is once again our image.
So we're just going to update our image using we'll fine the transformation matrix is as per above and
we're not using designs.
That's not our name.
I'm going to use new signs.
That's what I've called it the interpolation we're going to leave as it is.
It's just the calculation method.
The border will leave as it is and we can even use this as it is.
But since we created a scalar earlier we can just reuse the one that we used for the color.
Right.
So that was called Black which we created right here.
Right.
And that's it.
That really completes the challenge.
There's only a tiny tiny thing left to do.
You can see that we've pretty much deleted all of the objects from open C.V. at the end.
In the section we've called cleanup and since we're creating this m matrix here we can also delete this
whole thing just like we deleted the hierarchy and the contours and the image after we're done.
All right so let's save this whole thing save our changes.
Come back here and let's test it out.
Let's see if it works.
When I refresh my page him I draw something and click check answer and no errors.
So it seems to be working.
If you really squint you can actually see that when you draw nine here that it's gonna be centered according
to the center of mass in the tiny preview but it's very very small.
So you're gonna have to zoom in in order to see the difference but you should be able to actually spot
a visual difference on the previews that were actually outputting at the end of our page.
All right.
So we're down to the last few steps and the last one that we're going to do in this lesson is when to
normalize the pixel values meaning we should divide our pixel values by two hundred and fifty five.
Why is that.
Well let's take a look at what these pixel values actually look like as they are at the moment.
So I'm going to create a variable here called pixel values and I will set it equal to image dot data.
So when I pull out the data the raw data from our image object from open CV and I want to log these
right someone I'll take a look at them in the console someone along pixel values.
There you go.
Let's see this come in here refresh and I want to draw something inside the canvas and a quick check
answer.
And here is our output.
This corresponds to the pixel values of our tiny 28 by 28 pixel image now you can see there is a bunch
of zeros in the beginning and there is a bunch of zeros at the end.
And that's because we've got the padding around the white bit in the middle which are all of these values
that are around two hundred and fifty five.
What I mean by normalizing our pixel values is I want all of these numbers to be between 0 and 1.
I don't want to have them between zero and two hundred and fifty five and we're just gonna divide by
255 to accomplish this.
Now this is the point where you're going to see how easy this was in Python and how the syntax is a
bit more involved in JavaScript.
What you would think you could do is just take a pixel values divided by two hundred and fifty five.
Right.
That's what we would expect.
But when we actually have to do is use something called the Map method in order to affect all the values
in a javascript array.
So if you want to divide all the values in this array him we're going to have to use this map method
on W3 schools.
You can actually take a look and scroll down a bit and you'll find this example here click on try it
yourself and here what we have is a nice little example.
We have these numbers here and we want to multiply all these numbers by 10.
If I click Run You can see the output right here for the first number has multiplied by 10.
The second number is the third and so on.
Right.
In our case we're going to divide all of these by two hundred and fifty five if click Run.
That's exactly what happens here in the output.
But how does this code work.
What we see is that the map method takes a single input and that single input is actually a function
and the function will divide every element which in this case here is called num by two hundred and
fifty five.
This is how you affect all the elements in a javascript array.
So we're going to use this map method right here so let's implement this.
Let's update our pixel values array with the result from calling our map method.
So let's pixel value stop map.
And between the parentheses I won't have to provide a function.
So what we've got here is we've got the function created here and they're using the name here.
But what I'm going to do is I'm just gonna create the function between the two parentheses.
So when I have a function and I'm going to call every element inside my array just item when I open
some parentheses and then I'm going to return my item divided by two hundred and fifty five point zero
I might add my semicolon here and here and this is it.
This is how I can divide every single value in my ray by two hundred and fifty five.
So below our map method let's log our array after we've made all the changes.
So I'm gonna have a log message here.
I will call it scaled array and we're going to log our pixel values once again.
So now if we've been successful we should see our updated values being shown below our old values so
save your changes come back in here and refresh now draw something in here hit check.
Answer let's take a look at the output.
See what we've got.
Do you notice anything.
So we've got our old values here.
We've got our new values here.
And yeah they're between 0 and 1.
There's zeros and there's ones but there's no values in between these two numbers.
There's not a single decimal value here.
And I can see that in the original array.
We've had some values that were below two hundred and fifty five.
Right.
So there's one here fifty three to a seven.
Now if I looked on here there's no decimal values whatsoever so why is that.
Well it turns out that this line of code home image dump data actually gives us an array of integers.
So we're doing all this math on integers.
And unfortunately if I divide the value 254 by 255 I actually get the value 0.
And that's exactly what's happening in our code.
So what we have to do in order to actually get some decimal numbers here is we have to convert this
whole thing from an array of integers to an array of floating point numbers an array of decimal numbers.
Thankfully javascript mixed is really really easy and all we have to do is insert a line here that reads
pixel values is equal to float 32 array dot from and then the old array here.
Pixel values.
So we're going to update our pixel values array when to change it from an array of integers to an array
of decimal numbers or floating point numbers.
Let's see if this worked.
Let's save our changes refresh our website and try this again.
And here we go.
This is a lot more promising.
Here we can see the actual decimal numbers between 0 and 1.
So now we've successfully normalized our data.
The next thing to do is to create a tensor and make a prediction.
And that's exactly what we're gonna do in the next lesson.
I'll see you there.