Hello my friends.
Welcome back to this implementation of Thompson Sampling.
So at the end of the previous tutorial I asked you to.
Implement by. Yourself.
Well step three now that step. One and step.
Two are well. Implemented.
And so your.
Exercise in step three was to indeed figure out
the trick to select the ad that has the highest.
Random draw among all these random draws taken from the different beta.
Distributions for each of the ads.
So I hope that you tried.
Congratulations that you succeeded.
And even congratulations if you only tried.
What matters is. That you know, you. Practice.
And no worries.
Now we're going to implement this solution together.
So let's do this.
So the first question I have for you is do we have.
To stay in this second for. Loop.
Well yes of course.
Because that trick to.
Keep in memory the maximum of the run draws.
Has. To be done through.
All this.
Second full. Loop, you know, iterating through the. Ads.
And it's only
until we get the final random draw that we can know which one is the maximum.
So of course we stay.
Inside. This for loop.
And well, very simply, since, you know.
We have this. Random beta.
Here, which.
Is indeed the random draw from the beta distribution of the.
Ad I we are. Dealing with right now.
Well, what.
We have to do now is naturally. To. Compare.
This random draw to.
Max random the maximum the random draw.
Which okay so far is initialized to zero.
You know, at the beginning. Of the for loop.
But then of course you guessed that we will update that value.
Of max random.
To the. New random beta.
If the new random beta.
Is. Higher than the maximum, and.
Therefore. You know.
Over the iterations of this second loop, iterating through the at well.
Max random will. Be updated each. Time we.
Get a. Random.
Beta that is higher. Than the.
Previous. Random betas, meaning.
That is higher. Than this max random parameter.
So that's exactly the same trick as before.
You know, with the. UCB algorithm.
When we were updating that maximum of the upper confidence bounds at.
Each. Iteration of that second for loop.
So here it's exactly the same.
And therefore, you know what is the next step.
Now the next step I'm going to scroll down a bit.
The next step now is. To start.
With an if condition.
And if condition which.
Will check if well that.
Random draw.
You know this random.
Beta variable here is higher.
Than this maximum random draw.
And if it is. The case.
Then inside this if condition, we will. Say.
That this maximum random.
Draw has to be updated to.
Become the new random beta.
That was just higher. Okay.
And at the same time.
We will select the add. Of index.
I. Inside the same if condition.
Because anyway, if we get a new higher random beta.
Will this add will be updated as well?
Okay so I'll show you.
Let's first write this if condition. So if.
Random underscore.
Beta is larger.
Than max random.
All right then colon and side.
Well what do we do.
We of course update the value.
Of max random.
You know the maximum of the random draws.
To. Become.
That new.
Random.
Beta, which is indeed higher.
Than the maximum random draw.
Collected so far.
And then as we said.
Well, if that's the case, you know, if the ads I we're dealing.
With right now has a random beta.
That is higher.
Than the max random draw. Well.
We can. Select this add.
To become this index AI. All right.
So let's simulate.
You know the.
First iterations of this loop.
First max random is. Equal to zero.
Then I begins at zero.
So first.
We're dealing indeed.
With the first. Add of index. Zero.
We compute the random draw from the.
Beta distribution of this.
First add of index zero.
Then of course random beta will be larger than max random.
So max random will be. Updated to.
Become the value of that random draw the.
First add. And we will select. That. For that.
Then I will be. Equal to one.
You know the index one mean the index of the second add.
Then we will take the random draw
from the beta distribution of this second add of index one.
If this.
Random draw is.
Higher.
Than max random, which is equal to the random draw.
Of the previous add.
Then we. Will update.
Max random to become this new random draw.
Of this new add. And.
We will select this new add to.
Become the add selected.
And otherwise, if.
You know this new random draw from this new add
is not larger than max random, then we will do nothing and we.
Will just keep the previous.
Add and the previous max random.
And we repeat this, you know,
through the iterations of this second for loop up to the last add.
You know the add of index nine and.
Successively well. Max random will be. Updated.
If needed.
If the random draw is larger than the maximum random draw.
And we will. Select the right add accordingly.
Okay, so very easy trick.
And you know that's. A classic. Trick in Python.
To get. The maximum of.
A list through a for loop.
So it's really good. That you know this trick.
And it's really good that you did it twice.
You know, once with. USB.
To compute the maximum of the upper confidence bounds.
And with Thompson sampling to compute the maximum of the random draws.
So now. You're all set with.
Getting maximum techniques.
And therefore.
Time to move on to the next step.
Now by the way.
Congratulations if you got this right.
I know it's not that easy the first time, but you know by.
Just repeating this.
Over time you will become. So good at this.
And now. What, according to you.
Will be the next step. Here.
Well, I imagine some of you are wondering if we need to do an else here.
And the answer to that. Is.
No, we don't need to do an else.
Because if. This condition is not.
True, well, we'll just keep the last ad that.
Had this maximum random. Draw.
So no need to do an else here.
And in. Fact, well. The second for.
Loop is. Done.
Because it gives us exactly what. We want.
It gives us the ad that has.
The maximum random draw among all. The ads.
From 0 to 9.
And therefore that's why I got out of this second for loop.
And I go back into. This first for loop.
You know, right here.
I'm at this level now.
Right? Okay. So there we. Go.
Now we just need to finish this implementation because.
Basically.
You know, step one, step two and step.
Three. Are done.
I know it's a little.
Bit easier than the UCB.
Algorithm, but you're going to see that in the end.
The. Result might be even better.
I won't. Tell you exactly if it's going to be the case,
but you're going to see we're going.
To have a good. Surprise for the end.
Okay.
So now let's finish this implementation.
Basically we just need.
To, you know, update the different variables we have here.
You know, we need to update exactly these four variables.
Add selected.
Numbers of rewards one numbers. Of reward zero.
And total. Reward.
So I suggest we start by updating.
Add. Selected.
So I'm going to copy this.
Actually this will be simpler.
And then right here you know make.
Sure to be at the right level.
You know at the level of this first for loop.
You know for n in range zero to n.
And now I'm going to paste.
That add selected variable.
And so now over.
To you according. To what do we need to. Do here.
How do we need to update that add selected variable.
Well exactly.
The same as in.
UCB of course since this variable.
Corresponds to.
The full list of the ads that are selected.
Over time, you know it contains.
All the ads are selected over. Time up. To.
The 10,000 rounds, and.
Well, each. Time we.
Select a new ad, we need, of course, to.
Append.
This new ad that was just selected, which has index ad,
of course, into this ad selected list.
And therefore in the append function.
We of course need to input at all.
Right. Perfect.
So that appends the add to this full list of the ad selected which will be.
Remember the input. Of the. Histogram.
Then as we said.
We need to update these two.
Variables numbers of. Rewards.
One and numbers of reward. Zero.
But let's think this through.
How do we need to. Update these two variables.
Well that actually.
Depends on something something. Specific.
It is whether.
When you know, we.
Selected that ad at this particular.
Round for. A particular customer.
It depends on whether the.
Ad was selected, got a reward one or a reward zero.
Because indeed.
If it got a reward one, while this.
Needs to be incremented. By. One, and if.
It got a reward zero, well, this.
Needs to be incremented. By zero, right?
We can actually see that clearly in.
Step one and I1N is the number of times that I.
Got reward.
One up to round n. So we add. A new round.
We get a reward one.
Well this has to be incremented by one.
And this one this is number of times that.
I got reward zero up to. Round n.
So if at our round end.
We go to reward zero, well this needs to.
Be incremented by one.
And therefore.
What we need. To do here since.
We have, you know, two different conditions.
Well we need to.
Of course naturally.
An. If.
Condition once again but a simple one that is if condition that.
We'll just check if well the reward is equal to one.
So so. Far I'm just going to write.
This reward.
And then be.
Careful double equal.
One instead of just one.
Equal because otherwise this would be.
An effective one then colon.
But now you're going to say. Wait, we don't.
Have any. Reward variable.
Well that's because.
Same as in the UCB implementation I just want to.
Highlight what is this reward. Remember.
And therefore just above.
I'm going to specifically.
Say what is this reward.
And that's you know.
That's exactly.
The same as in the UCB implementation.
It is the value. In the. Data set.
Corresponding to. The row.
We are dealing with right now in this first. Full loop. You know, with.
This particular.
Customer and the column of.
The ad that was just selected. Right.
Because the reward is the value we get after selecting this
ad here to show to this specific user we are.
Dealing with right now in this first.
For loop, and therefore the reward here is the value in our data set.
Right.
So I'm taking my data set and then.
The values attribute.
And then in square. Brackets.
I enter the. Row of the user.
We're dealing with right now which is n. So n.
And then the column of.
The ad was just selected which is at right.
That's exactly.
The same as in. The UCB.
But I clearly wanted to highlight what is the reward here, so.
That then we can. Check. In an if condition.
If the reward is. Equal to one.
And if that's the. Case, well what. Are we going to do.
Well as we said.
We are going to increment this. By one.
Because if the reward is.
One, that means that the number of times.
This particular ad got.
Reward, one just got incremented. By one.
Then of course. I'm taking the right index of.
This list.
Of the different numbers of times each.
Ad got reward one.
But the ad.
We're dealing with right now has of course indexed.
Ad, because that's the ad that was just selected.
And therefore.
I just need. To copy this.
And then ad. Equal. Paste that again.
And then ad plus one. All right.
So all good.
This particular value is updated the right way.
And now back in the. Else.
Condition meaning the other condition where.
The reward we got by. Selecting this.
Ad at this. Particular round is.
Equal to zero.
Which can be specified through an else.
Because reward can either be equal to.
1 or 0.
So else is fine here.
And then well what we need to do.
Is of course take that.
You know, I can just copy all this.
And paste that and replace of course numbers of rewards one by.
Zero here.
And same one by. Zero here as well.
Because in this condition we are in the condition.
That the reward we collected is zero.
And therefore.
That variable.
Of the index ad.
Here, which.
Represents number of times this.
Ad here got reward zero needs to be.
Incremented by. One.
All right.
Perfect. And now.
We just have. One final thing to do.
You know exactly. What it. Is. Let's not forget about this.
It is to update that particular variable which gives the total
reward, you know, the total accumulated reward over the rounds.
And since now we are, you know, in a new round, you know, with this.
First for loop here.
Well, of course,
as soon as we get this new reward, we need to update this total reward.
By incrementing it.
By the reward we just got, whether it is.
Zero, in which case
there is no incremental action, or one, in which case we increment it by one.
So let's do this efficiently.
Total reward is equal to total reward plus reward.
All right.
And that way we update our total reward variable.
And that's it my friends.
Now the implementation of Thompson sampling is over.
So congratulations you.
Just implemented your second reinforcement learning model.
And by. The way.
Now the full implementation is done because indeed.
Here. We don't have anything to change.
We can just plot. The histogram because we have the.
Same variable names.
So now it's time for the show time.
In the next.
Tutorial I will show you.
The demo of, you know, Thompson sampling.
We will of course.
Run all. Our.
Cells and mostly we will compare. The two.
Performances between. UCB.
And Thompson sampling.
I remind that. UCB was perfectly able to.
Find that. Best.
Ad you know, with the. Highest click through rate.
In 1000 rounds, but was not able to do it in 500 rounds.
And so now I'm. Super excited to check.
If Thompson sampling will do better than this.
Meaning if it will not only.
Be able to figure out the best ad. In 1000 rounds.
But also if it will be able.
To find it in 500 rounds.
Which UCB couldn't do.
All right, so let's.
Find out. About this. In the next. Tutorial.
I can't wait to. Show you all. This.
And until then, enjoy machine learning.