Hello, my friends.
All right, let's start the implementation of the upper confidence bounds algorithm.
So we're going to. Make it step by step.
And you're going to implement each.
Of the steps first before we do it together.
And that first step you know I've.
Prepared the slide. Here.
We're going to have a look at it many times.
The first step is that at each round you know for each.
User because.
A round corresponds to a user, we consider two numbers for each add I,
you know, from 1 to 10
this first number and I n, which is the number of times to add.
I was selected up. To round.
And so make sure to understand the indexes here and the variables.
And then or I n which is the sum of rewards.
I have to add number I up to round n okay.
So the first step that I would like you to do.
You know, and I'm going to ask you to press pause on this video.
The first step I'm going to ask you. To do is to.
Make these. Two.
Variables, you know, create.
Two variables for these. Numbers.
The number of times that I was selected up to run
n, and the sum of rewards of the at I up to round that.
So create these two variables.
And then also in the same step one.
I would like you to create other variables.
The first one is the total number of. Users.
To whom we will show one of the ads, and that's 10,000.
So I would like you to put this
10,000 value in a variable that you can call capital N.
Then I would like you to create
another variable for the number of ads we have, meaning ten.
And you can call this. Variable lowercase d.
After this please create a variable.
That will contain the. List of the selected.
Ads over the round.
So you know it will start as an empty list and will become a list
of 10,000 elements corresponding to the 10,000 ads
that were selected to the 10,000 users successively.
Then please create these two variables.
So this first.
One and I n you can call it numbers of selections.
And you have to of course initialize it.
As a list of.
Ten elements only containing zeros.
And then I will show. You a trick to do that easily.
And for the second variable or I n you can call it some of rewards and same.
You have to initialize it as a list of ten elements, but initialize with ten zeros.
And we will populate these two lists over the rounds.
Okay. And finally.
I'd like you to create one last variable, which is the total reward.
And which will simply. Be.
The sum of.
All the rewards received.
At each round.
Because it's important to remember that. The.
Zeros and ones in the.
Data set. Are in fact the rewards.
You know, the single.
Rewards received at each round.
If the user clicks the add.
Then we get a reward of one at a particular round.
And if the user doesn't collect.
Yet, we get a reward of zero.
We get no. Reward, basically.
Okay.
And the total. Reward here that I would like you to create as.
A final variable.
Will be the accumulated reward.
Meaning the sum of. All the rewards collected.
Over the round.
All right.
So let's do this. Please press pause on the video.
And in a second we will implement the solution together.
All right.
Welcome back. Let's do this.
So first let's create.
A new code cell. And let's.
Create each of these variables. One by one.
So at first we said that we.
Wanted to have a variable for the total number of.
Users or the.
Total number of rounds over which we're going to show ads.
To the users.
So there we go.
We want to call it n capital n equals.
And that's 10,000.
All right. 10,000. Yes.
Then we said we wanted to have a.
Variable for the number of ads meaning ten.
And we want to call.
This variable lowercase d equals.
Ten. Perfect.
Then as we said we want to have.
The full list of the ads that are selected.
Over the round.
So you know at first this will be an empty list.
And over the rounds it will get bigger and bigger.
Up to at the.
End it. Will be a list of.
10,000 elements, and the nth element will be the ad selected at run n.
All right.
So we're going to call this variable ad underscore selected.
And this will be initialized as an empty list.
Just like that ad selected.
All right then next one.
Well the next two ones are. These two.
You know and I n number of times the ad I was selected up to run n
and r I and the sum of rewards of the ad I up to round n.
So for the first.
One we will call it numbers of selections.
And since we want to have these numbers of selections, you know, these numbers
of times.
Each ad was selected for all the ads.
Well this will be. Initialized. As a list.
But not an empty list, but a list of ten zeros.
And the trick to initialize this list of ten zeros
efficiently is to just add here times t.
All right, just like that, this will initialize.
This list as. A list of ten zeros.
And then each time we select an ad, for example, ad number three.
Well the third element of this.
List will. Be incremented by one.
All right. So at first it will be zero.
Then let's say ad number three is selected. It will become one.
Then let's say ad number. Five is selected.
We'll replace zero by one.
And then you know each round
it is incremented each time a new ad is selected.
Okay.
And at the end we hopefully want to see one ad.
That is selected way.
More than the others, and.
UCB will figure it out.
Okay then next variable you know this one.
The sum of rewards of the up to round n will same here.
We want to have these sums of rewards for each of the add up to round n.
And therefore we're going to create another list
which we're going to call sums of rewards.
Right. And same.
This will be initialized as a list of ten zero.
So I'm just copying and pasting this.
All right. That's the same.
And of course at first round well each ad has a sum of.
Rewards equal to zero because.
At the beginning no ad is selected and therefore no.
Reward is. Collected.
Then as we said, we want to have a final.
Variable, which is the total.
Reward accumulated. Over. The rounds.
You know, with the different ads we select at each round.
And let's call this. Variable total.
Underscore. Reward.
And of course we have to initialize it as.
Zero because.
In the first round well no AD is selected yet and therefore no reward is collected.
Okay. Good.
So we have all.
The parameters all initialized correctly.
And now what do you think will be the next step?
Well of course the next.
Step will be to start a for loop which will iterate through all.
The different rounds, you know, starting from round. Zero.
Because, you know, in Python indexes start from zero up to round 10,000
and that each round, well.
We will follow these two steps.
You know, we will compute the average reward of up to run n.
Then we will get the confidence interval.
And in step. Three will select the.
Ad that has the maximum upper confidence bound.
You know the higher upper confidence bounds.
All right. So you will see this will be very easy.
We will just follow these steps.
And it will ask you to implement them first.
But no worries I will guide you.
And so now let's take a.
Little break because this for.
Loop will, you know, actually take a few lines of code.
So make sure to have good energy for this.
And then let's smash this together.
Until then enjoy machine learning.