Hello my friends.
Welcome back to this implementation.
Are you ready to start that big full loop which will iterate through all.
The rounds in all the 10,000 rounds, meaning the 10,000 customers
to which we're going to. Show the add.
And therefore.
In this loop at each step, well.
We have to implement the.
Three steps here. Well, we already.
Implemented step one, but we have to implement step two and step.
Three at each iteration of the loop.
All right let's do this.
Let's start this for loop starting with four.
Then we need to choose.
A variable name for you know that. Iterated variable.
And this time instead of calling it, you know, as the classic name, we.
Will call. It n, because.
Clearly what we are iterating through are.
The rounds, you know, the rounds from round one.
The first user to. Round 10,000, the last.
User. To whom we. Show yet.
Okay.
So for.
N in and then of course range from.
Zero.
Because you know indexes in Python start from zero.
Up to.
And here you can either put 10,000 but we already put 10,000 in a variable.
So let's put.
N so that you know
then if you want to
try another number of rounds you can just change the value of n here.
Okay.
So for n in range from zero to n then colon.
And there we. Go. Let's start that for loop.
All right.
So I'm going to scroll down a bit you know from here perfect.
And there we. Go. Let's do this. All right.
So what do we need to start with.
Well you know. Right now what. We want.
You know what we ultimately want. You know within.
This particular.
Iteration of number n what we want is to select an that right.
We want to select an add.
And we're going to select it according to these steps.
You know we will. Select the add.
That has the maximum. UCB.
Upper confidence bound.
So the. First thing that. We'll do is.
Start from the first add you know add number one.
But since we're going to work with indexes the index.
Of this add will be zero.
So I'm going to start from.
This add equals zero.
Here I'm introducing.
Of course a new variable add.
Which is initialize. As zero. But then you're going to.
See that we will do a. Second.
For loop which will loop over all.
The different ads one by one.
You know, starting from this one at number one of index zero, then
going to this one at number two of index one and etcetera.
Up to add.
Ten, because indeed, in order to get the maximum upper
confidence bound, we will need to compare the upper confidence bound.
Of each of these ads.
And therefore we'll have to iterate through each of these add to.
Find the maximum upper confidence bound.
And that's why right now I'm initializing this.
New add variable at zero.
To start with, add number one.
And then we'll compute the. UCB of this. Add number one.
And then through the for loop you know we. Will get the other upper.
Confidence bounds and we will get the maximum. One.
You see the idea very simple.
We just have to follow. You know. A certain logic.
Where we know that the ultimate.
Goal is to select the add that has the highest.
Upper confidence bounds.
All right.
So then do you think we have.
To already start that. Second for. Loop.
Well almost.
But we need to do something else.
We need to.
Introduce a new variable which.
Will mean you know that maximum of the upper confidence bound.
Because for each of the add through that second for loop we.
Will compute the upper confidence bound.
But in order to compare them to the maximum upper confidence bound.
Well, a smart thing.
To do would be to introduce a new variable here that will.
Be the maximum upper confidence bound, and which will be compared.
To each of the UCB of the At.
And therefore here I'm introducing a new variable which I'm going to call Max.
Let's say upper. Bound.
All right.
And which I'm initializing to. Zero because.
Indeed at the beginning, well. Let's say that the.
Maximum, you know, the highest upper confidence bound is.
Zero. And then of course. Once we get.
An. Add that has a higher.
Upper confidence bound than.
Zero, we will update this value.
Of max upper bound to this new. Higher.
Upper confidence bound.
All right.
So our logic here then.
Now we can start the for loop. You know the second for.
Loop that will iterate through the.
Different add from 1 to 10.
So there we go. We started second for loop for.
Then a new looping variable you know iterated.
Variable. Which we're going to call this. Time of course.
Well either add but add is already taken.
You know that's already an existing variable.
So let's just choose I for.
I in range from. Zero.
Because indexes in Python start at. Zero up to.
Either ten here or you know.
D because we already introduced that d variable which is equal.
To ten. And if you want to.
Do that.
Same UCB implementation on a different number
of ads, imagine you have, you know, five ads or even 50.
As well.
You simply have to change. That value here.
And nowhere. Else.
All right.
So that's the purpose of introducing these variables here.
So for I in range from zero to d.
And there we go. We can start the second full loop.
And it is now in the second for.
Loop that we will really.
Implement each. Of these steps.
So now now is the time you're going to press pause
on the video to indeed implement that.
Step two.
First I would like you to.
Implement from scratch.
No, you can totally. Do this because you have. Everything here.
I would like you to implement that.
Step two you have also all the variables, so it should be all fine.
And then don't implement step. Three because that.
Will be done for next. Exercise.
And besides it is not that direct. You know, we will.
Have to use. A certain trick in order to implement that.
Step three. All right.
So please implement step two.
Now please press pause on the video and implement step two.
But let me just give you two hints.
You know, because maybe you will encounter issues with something in this part.
As you can see, you will have to compute.
The square root of that value here.
And you have to know that in. Order to use the square.
Root in Python, you have to import a certain.
Library, which is simply called.
The math. Library.
And from which you can use a lot of mathematical tools.
Like the square root.
So actually let's. Do it right now so that it's done.
We can either import. It here.
Or you know, since I.
Prefer to keep the essential libraries here, let's just.
Import. It at the.
Beginning of this implementation.
All right. Right here.
So that it's done, you know, so.
That you can be ready to use that square.
Root function.
And then I.
Will let you of. Course look in the documentation online.
What is the exact function name.
So math there we go.
And then the second hint I wanted to give.
You was the fact that you have to start the step two.
With an if condition.
Right? Because at the.
Beginning no. Add was selected.
And. Therefore this.
You know, this quantity.
Is equal to zero for all the ads.
And if this. Is equal to zero, this. Doesn't make sense.
You know it will. Be equal to plus. Infinity.
So you have to start with this.
If condition to make sure. That the ad.
We're dealing with in the second.
Full loop was already selected.
So that this.
Is different than zero and this. Exists.
And so that then you can indeed compute the confidence interval.
All right. So that was the last hint. And now.
Your turn.
Please press pause in the video.
And please implement this step to.
All right.
Perfect.
So now let's implement the solution together.
Of the step two.
All right.
So as we said we have to actually start here with.
An if condition.
That will check if the add we are.
Dealing with right now you know the add of index.
A has. Already been selected.
It's not the case at the beginning
you know, but over the round it will be the case. You know the.
Add will.
Already have been selected, but at the beginning
for the first rounds we need to check that.
Indeed the Add has been selected.
So in order to. Check this well that's very. Simple.
You know we have.
This variables that gives for each of the add.
How many times they have been selected. So. Far.
You know at round N up to round N.
And therefore we're going to take this now, you know.
And then.
We will check that. The index.
I. Which is the index of the add we we're dealing with right now
in this number of selections list.
Well we're going to check that the.
Element of index I in this number of.
Selections list is indeed larger.
Than. Zero.
Because if it is.
Larger than zero then that means that the Add was.
Indeed already selected at least once. Okay.
And now if you want. But that's totally. Optional.
You can put this condition in parenthesis, right?
And then add. Just a. Colon.
And now now you're going to say to Python what should happen and what must happen
if indeed.
That at I, which.
We're dealing with right now in the second for loop.
Was already selected.
All right.
So now now is the time we can implement all.
This because indeed this.
Is different than zero. This is larger than zero.
So very simply let's just follow this step by step.
First let's.
You know, create a new.
Variable for this which we're going to call average reward.
You know, because this corresponds to an.
Average of the reward because it is the accumulated reward here of this at.
I up to run n divided by the number of times it was.
Selected. So that's nothing else than an average.
And therefore.
Let's create a new variable here which we're going to call average.
Underscore. Reward.
And it is simply.
Going to be equal to well the accumulated reward of that specific adds I up to.
Run n which. Is given exactly.
Thanks to this variable here, of.
Which we're going to have to take the index.
I write that at I, which.
We're dealing with right now. So I'm basing that here.
And I'm taking of course.
In square bracket the. Index.
I because this.
You know.
All of this corresponds exactly to the.
Accumulated.
Reward of that specific add I up to.
Run then okay.
So sum of rewards I.
And then of course we need to divide that by.
The number of times. That. Same ad.
I was selected.
And this is given of course.
By this again right.
The number. Of.
Selections of index I, which corresponds to the number of times the out of index.
I was selected.
And that gives you exactly
the average reward that gives you exactly that value here.
Okay. So very easy so far.
Then next up the next step.
Is to get the.
Confidence interval.
And more specifically we're going to get that value.
So right now we're going to compute.
This value here.
All right.
So let's simply call it delta I you know it's a new variable delta.
Underscore. I equals.
And now there you go.
That's where you needed to take that square root.
Function from the math. Library.
So first we.
Have to call indeed that math. Library.
And from which
well we need to call this function that allows to compute the square root.
And I'm sure you've found very easily online that this function is.
Called execute art.
And then parenthesis.
Right.
This computes the square root of a value.
And the value we have to put inside.
This function. Is of course exactly.
This. You know.
Three divided by two multiplied by the.
Logarithm of the realm and.
Divided. By the number of times that.
I was selected up to. Run n.
Okay, so let's do this.
First we start with this three divided by two.
Then multiplied. By.
Then there you go.
You need to call the.
Log function which is another function of this math. Library.
So actually this math library is used twice.
I'm calling it now to indeed be.
Able to call that log. Function.
And now inside this log function well be careful.
Maybe I should have.
Given you another hint. Here. But be careful.
We can not input n here. Why is that?
That's because actually.
N you know in this range here this first range.
Starts from zero.
So the first value. Of n.
Is zero.
And that's you know because of the indexes in Python.
And you have to know that the logarithm of.
Zero is. Actually minus infinity.
Therefore it would be very.
Dangerous here to only put n in order to.
Protect us from this.
We're just. Going to.
Start. From. One okay.
That's one way to do this. The other way is of.
Course, to, you know, make that.
First for.
Loop going from one to n plus one
so that we can go indeed from round one to N plus one.
But you know, with Python we. Always work with.
The same indexes starting from zero.
So that's why we go with the first option here.
All right.
So all good so far.
And then of course. We have to divide that by.
Well let's look at it.
Again the number of times that.
I was selected. Up to round N.
And that's of course exactly this value.
So I'm copying this again and pasting that here.
And there we go.
Very quickly we got our Delta I, we got exactly this value.
And now. We have. A final value to compute.
Which is of. Course, well you know that long.
Awaited upper. Confidence bound.
That's exactly what we have to compute right now.
And it is going.
To be simply the sum of that average reward plus that delta here.
So let's here add a new line of code.
And then introduce.
A new variable which we're going to call upper.
Underscore bound not max upper bound but upper bound and which will simply.
Be equal to the. Sum of. The average
reward.
There we go.
Plus the delta.
I and perfect.
And now you know, thanks to. This for loop, the.
Second for loop iterating through. All the.
Ads from 0 to 9, but from and number one and number ten.
Well, we have the upper bound, we have the upper.
Bound of. Each of these ads.
And the upper bound
that you know, that specific round n we're dealing with right now inside this.
First for loop.
All right.
So basically.
We are done implementing this. Step two.
We implemented this step two.
And we even got that. Value. Here. You know the upper bound.
But now step. Three.
Is not implemented yet because indeed. Step.
Three consists.
Of selecting the add that.
Has the maximum upper confidence bound.
So now we need to add.
One trick, you know, which is kind of classic in Python.
But in do we need to add. One trick to select the.
Maximum upper confidence bound of these.
And at.
So going back to here, well, we will surely take a little break here.
And actually.
I will directly ask you to. Try to implement that.
Step three you know, between this tutorial and the next one.
Before you. Start the next.
Tutorial, please.
Try to implement. Step three.
It's not that direct. It's not that easy.
But you know you have to use some kind of algorithmic logic.
Okay, but I'll give you of.
Course, some hints unless you don't want the hints and therefore you can.
Directly press pause or quit this. Video now.
But I'm going to give you some hints if you want.
The first hint is that right now you're done with this if condition.
So you can go. Back here.
You know inside this second for. Loop.
And then you're going to have to start.
With of. Course an else. Right.
Which is the condition where you know.
That, that we're.
Dealing with right now has not been. Selected yet.
So what you'll have to do. And that's my last hint.
Will be to use a trick to select the ads.
That have not been selected yet.
Why do you need to select these ads?
Well, the answer is here.
That's exactly because of this.
You know, this denominator which shouldn't.
Be equal to zero.
And since this is exactly the number of times that.
I was selected, well, we need this to be.
Different than zero in order to compute.
This average. Reward.
And therefore in order to compute then. The.
Confidence interval. With the upper.
Confidence bound.
And that's why in the.
UCB. Algorithm it's.
Compulsory to.
Make sure that during the first round all.
The ads are selected. So actually.
During the first ten rounds.
We have to use a trick so that we make sure
to select all the ads so that this way, well.
All the end I end here, you know, for all the different.
Ads will be. Different than zero. Okay.
So basically the exercise.
For next time, you know, the next. Tutorial is to.
Indeed implement that.
Step three to.
Compute the maximum of the upper confidence bound.
Well, at the same time, implementing a.
Trick. To.
Make sure that all the ads are selected.
In the ten first. Rounds.
So very. Challenging, but.
At least try, at least try as hard.
As you can.
And I promise. You that you will. Progress and improve.
Your skills. In machine learning.