All right, my friends, let's finish this implementation.
Indeed, we're very close to the end.
We have already implemented step one, step two, and step three,
and now we just need to finish this by, you know, updating.
These variables that were. Created and.
Initialized right before this big four loop, iterating
through the rounds, you know, through the users.
All right. So let's start simply with this one.
Add selected, which is the full list of all the ads selected over the rounds.
So let's bear here.
You know here I'm back inside.
This first for loop iterating through. Rounds.
And right now.
How do you think we must update this add selected variable.
Well simply of course we have to add to this list the add that was just selected.
You know right here you know in this.
Second for loop.
All right.
And you know. The trick there is this append.
Function in Python that.
Allows to add. An element to a. List.
And well that's. Exactly what we're going to. Use.
Now we're going to add a dot here.
Then add append for the append function.
And inside this function you have to input the element which you want to add.
To this add selected list. Right.
And that's of.
Course at you know that add variable that was computed here in this second folder.
All right.
So that's how you update this first variable add selected.
Then next step.
Well next step. Is simply to update this. Variable.
Now number.
Of selections which is a list.
Of ten. Values. Corresponding to the ten. Add.
And for each of these values well you have.
To number of times the.
Add was selected up to.
Round n that same round.
And we're dealing with right now in this first for loop.
All right. So let's paste it right here.
And now according to you.
How are. We going to. Update this variable.
Well since this. Add.
You know. Was just selected.
In this new round N well therefore we need to.
Update the. Element of index. Add.
Basically the index of the. Add was just selected.
Inside this numbers of selections list.
We need to.
Update that.
Element by incrementing it. By one.
All right. So here. Not only we need to take. This.
List numbers of selections, but we need to take mostly. The.
Element of index add meaning the index of the.
Add that was just selected.
And then you have two ways to do this.
The first classic way is doing a plus equals.
One which increments this particular number.
You know, the element of index add inside this numbers of selections list by one.
Or you know, if you don't like this notation, you can.
Simply do a qual.
Then you copy this again, you paste it.
Here and you just add. Plus.
One as you want.
That's exactly the. Same depends on how.
You prefer to see it.
Okay then. Let's update this variable.
Sums.
Of rewards, which is the accumulated reward for each of the add.
In the same list of ten elements corresponding to the. Add.
So I'm copying this.
Then I'm facing here.
Then of course, what we need to change inside this.
List is that. Element of index add.
You know that index of.
The add that was just selected.
So there we go.
Let's here add in a pair of square brackets.
Add the index add. That was just selected.
And then same we're going to update it again.
You know we're going to take that.
And then according. To you what do we have to add to this.
Sums of rewards.
And especially to this. Element of index.
Add inside this sums of rewards list.
Well that's exactly the reward that we got by selecting this add.
You know. This add which had. The maximum upper bound.
And where do we have these rewards.
You know for each of the add.
Well that's of course in our data set right.
This data set is simulation telling us for each user.
On which. Add they're. Going to click.
You know we don't know. That in reality.
But this data set is a simulation.
And therefore we have for each user.
And for. Each ad, if the user clicks yes.
Or no on the add.
And since now we know with.
Which user we're dealing with thanks to this.
You know realm, because the round here corresponds.
To the user.
And since we also know which.
Ad was selected.
You know.
Because of this second for loop, well, we can directly access
the reward that was just received at this particular round.
And for this particular ad that was selected.
And the way to do this is simply to take here our data set.
Right, because this.
Is the original data set.
But remember. That we created the.
Pandas dataframe in this. Data set variable.
So we need to take our data set.
And then remember we need.
To add a values in order to access a particular value of this data set.
And then we need to enter inside some pair of square brackets.
Well first.
The index of the.
Row of. The cell we want to access. Which is of course.
N because n corresponds to the user meaning the. Row.
And then we need to enter. The index of.
The column of the cell we're dealing with.
Which is of course the ad that was selected.
Because now we need to get.
The reward of the ad that we selected for the particular user we're dealing.
With right now in this. First for loop.
And that's.
Exactly data set built values.
And. AD. Okay.
And now I'm just going to do something so that.
It's really clear to you that this is the reward we got at this.
Particular round.
N and in order to do this I'm going to take that here.
Cut it. And right.
Between these two lines of code I'm going to create a new variable
which I'm going to. Call reward.
You know, so that I can really.
Highlight that this is the reward.
You know, the reward collected after.
Showing this.
Ad to this user. N this is the reward.
And therefore here I'm just going to add reward.
All right. So that you can. Clearly see that.
Very important concept in reinforcement learning.
You know.
It's all about the reward the reward that is collected at.
Each round and then the. Accumulated reward.
And speaking.
Of the accumulated reward, well, that's exactly.
Our next. Step here, because.
Indeed.
The last variable which we have to update is that variable.
Computing the.
Total reward we get up to. Around n.
And now you will perfectly know how.
To update that total reward variable.
According to you. What do we have to do here?
Well, you know.
We just need to add to this total reward variable
that last reward which we just got at.
This round, that meaning.
This reward of selecting the ad.
And showing it. To the user. N all right.
And that's it my friends.
Now the.
UCB. Algorithm is. Fully.
Implemented.
If this sounds a bit overwhelming, well, I really encourage you.
To try.
To implement that.
Fully again, because really you just have to follow the.
Logic and then everything makes sense.
So I understand that, you know.
This is the first. Time we implement.
Such a. Code because, you know, so far.
We've only worked with libraries.
It was simpler before, but you also.
Need to know how to implement such algorithms from scratch.
And so here it's. Really good that we did it.
But don't worry if it feels a bit overwhelming.
You just. Have to. You know.
Either rewatch the videos or try to.
Re-Implement this from scratch yourself.
And I promise you.
That this. Will sound easy. Peasy. Okay.
So no worries.
Do this.
Again if needed and whenever you're ready, meet me in the next.
Tutorial to plot the.
Final histogram, which.
Will show us exactly which ad was.
Identified. As the best ad.
Of the ad with the highest conversion rate.
By this UCB algorithm.
I can't wait to. Show this to.
You and until then, enjoy machine learning.