All right, let's do this.
Let's implement Thompson sampling.
So we already initialize, you know, the main parameters
which are part of the common framework between Thompson sampling and UCB.
So this is n the total number of rounds or the total number of users
to whom we're going to show successively the ads.
And then we have the number of ads which is ten.
Here we're dealing with ten different designs of ads,
which we are experimenting to figure out which one has the highest conversion rate.
And then we have this list which is initialized as an empty list, but
which will be populated over the rounds with the different ads that we select.
So at the end
it will contain all the different ads that were selected at each round.
And now now what we're about to code is specific to Thompson sampling.
So there we go.
We're going to start with the first step.
Step one which says that at each round n
we actually consider two numbers for each ad I and I1N,
which is the number of times two ad I get reward one
up to round n and an I0N
the number of times that I got reward zero up to round them.
So basically what we have to do in this step one is create
two variables here for these two numbers.
And we have to initialize them of course as zero you know zero for both.
Because of course before we begin, you know the first round.
Well these two numbers are equal to zero because the ad didn't
get any reward at all.
So that's what I would like you to do.
And since you know these numbers are for each ad, I.
Well, what we're going to do is we will create a list of ten elements
where each element is indeed this number for each of the ads.
Right. And same for this one.
So now your turn.
Please press pause on the video and create two variables
which will initialize these two lists of numbers.
And you can call these two lists first.
For this one, you can call it numbers of rewards, one
you know with underscores and for this one you can call it numbers of rewards.
Zero okay.
So please press pause and I'll give you the solution in a second.
Okay. Good. Let's do this together.
So as we said we want to introduce two new variables which will be two lists.
First, for the numbers of times each ad gets the reward one.
And we're going to call that numbers of rewards
one. All right.
And as we said we want to initialize it as a list of ten elements.
And all these elements should be initialized to zeros
because at the beginning no AD got any reward.
Right.
And remember the trick to initialize a list of ten zeros.
It is to put a zero here in square brackets,
and then multiply this by the okay.
And then very simply for the other list you know corresponding to these numbers.
Well same.
We're going to call that numbers of rewards
not one but zero and same.
We initialize it as a list of ten zeros.
And over the rounds the elements that these lists will be populated
for this one.
As soon as an ad receives a reward, one in which case we increment
the element corresponding to the add by one and same for this one.
Each time an ad gets a reward zero, we increment the element
corresponding to this ad in the list.
Okay, so very simply.
And that's the two parameters specific to Thompson sampling.
So if you got this two
well congratulations you just implemented step one.
All right.
So now according to you what do we have to do.
Well not yet the big four loop.
Remember that we also want to keep the accumulated
reward that we, well, you know, accumulate over time.
And we're going to introduce the same variable as in UCB.
You know we could have even kept it in the framework.
But remember that variable is total reward.
So here we introduce the same new variable
which of course we initialize still to zero.
And now now we can start the for loop.
You know that main for loop which will iterate
through all the 10,000 rounds in each.
We will show an ad to a user, which will decide to click yes or no on the add.
And so there we go. Let's stop this loop.
So it's exactly the same as in the UCB we still was for.
Then we choose n for the iterated variable because same and represents the rounds.
And then and will go in the range from same zero.
Because indexes in Python start from zero up to n, right.
The total number of rounds, or the total number of customers of users
to whom we show successively the yhat and then colon.
And there we go.
Here we are inside this big first for loop.
All right.
So according to you, what will be the first step here?
You know, there is a second full loop coming up.
It is the full loop that will iterate through the different ads.
But before this we need to introduce a new variable,
which we already did in the UCB implementation.
Remember we need to introduce that variable,
which will be the ad that we select at each front end.
And as in the UCB implementation we will introduce that variable as ad right
ad which will be the index of the ad that will be selected at each round end.
And we initialize this to zero
because you know we're going to have that second for loop that will iterate through
different ads starting from zero up to the last one of index nine.
Okay.
Then let's have a look at the algorithm and let's see
if we need to introduce a new variable.
You know, before we start this second for loop,
I remind that, you know, for UCB we indeed had to introduce a new variable here,
which was max up about, you know, we introduced Max upper bound
and initialized it to zero before we started the second loop.
Okay. So here that's kind of the same.
Let's see if we have to introduce a new variable other than add one
before starting this second for loop.
And well you know, when we have a look at step three here,
we see that indeed we select the ad that has the highest
of these random draws, you know, taken from the beta distributions.
And therefore there is still a maximum value to consider here.
And that's exactly what we will introduce as a new variable
just before starting this second for loop.
That new variable will be exactly that maximum of the random draws.
And we're going to call it max underscore random
okay max random which we will initialize of course to zero.
And now now we can start the second loop.
All right. So let's do this. Try to do it faster than me.
You know exactly how to do it. We start with four.
Then you know the iterated variable will still be I,
which will iterate through the different ads from 0 to 9, because are the indexes.
So for I in the range from indeed zero to
well d, because we had to divide both for the number of ads and then colon.
And there we go.
Now is the time we implement the next step in the Thompson sampling algorithm.
I'm talking of course about step two where for each ad I.
Well, that's exactly what this second loop is about.
You know, it will implement step two for each of the ad.
I from 0 to 9.
And so for each of these ads, we take a random draw
from the data distribution given below of parameters.
And I n plus one where an n is of course number of times
that I got reward one up to run n and the second parameter
and i0 and plus one where NI0N is
of course a number of times I got reward zero up to running.
So that means you can totally implement this step to on your own,
because you have all the parameters, right.
We already introduce these two new variables.
We initialize them the right way.
And so now you can totally implement this step two on your own
before we do it together, I will just give you a hint.
Unless you don't want to listen to it
and look into the documentation online, which is even better.
But I'll give you that hint anyway, the way to get the random draw
from a beta distribution of these parameters
is given by a certain function of well, this random library, which we, by the way,
already imported in our implementation, that random library.
Well, this random library contains a certain function
which can give you exactly what you want here, meaning a random
draw from the data distribution of any parameters.
And that function is called beta var it in one word
betta var iat.
Okay, so that's the function.
That's all I'm going to tell you.
And so now you can really implement this step to on your own.
So please press pause on this video
and will implement this solution together in a few seconds.
All right.
Are you ready?
I hope you got it correct.
Really there is no trap so it should be totally fine.
But let's do this.
So here we are
at the beginning of the second full loop, iterating through the different ads.
And so what we have to do is collect for each of these ads.
This particular number taken as a random draw of this
beta distribution called with these parameters.
So let's do this.
First we're going
to introduce a new variable which will be exactly that random draw.
And we're going to call it random underscore data.
So that's exactly the random draw
taken from the beta distribution of these two parameters.
So random beta.
And then as I told you, the way to get the random draw from a
beta distribution of certain parameters.
Well we first need to call the random library
from which we're going to call the beta variate function.
All right so that's the function.
And inside this function well of course we input these two parameters.
And I'm plus one and I and zero plus one.
And since right now
we're dealing with a specific ad you know the ad I in this second for loop.
So it starts from zero and then goes to 1 to 3 etc.
up to nine.
Well these two parameters that we have to enter are
for this specific ad I meaning for the specific index
I in these two numbers, and I want n and i0 n okay.
And therefore the way to get these two numbers
is by calling our list of numbers of rewards one.
You know, the number of times each ad, and especially our ad,
I we're dealing with right now got the reward one.
So we take this list.
We paste it inside this beta void function as the first parameter.
And then of course we take the index I.
Right.
Because we're dealing right now with this specific ad I in this second for loop.
And then it's going to forget the plus one.
And then same for the second parameter.
That's going to be the number of times this particular ad I we're dealing with
right now in the second for loop has received the reward zero.
So here I'm basing that.
But I don't forget to take the index
I of this ad I and then of course I ad plus one okay.
And that's it.
That's all you had to do for this. Step two.
So you see it's actually a bit simpler than the UCB algorithm.
But now I trust that you're going to use the same right trick to implement
step three.
Because indeed in this step three
we have to select the ad that has the highest of these random draws.
You know, we took ten successive random draws for each of these ad,
but inside the same for loop that is iterating through the ad.
We need to use a trick to keep in memory the highest of these random draws.
And at the end of the full loop, we'll get the final highest of these random draws.
So the trick that you have to use is exactly the same as in UCB.
You're going to use, of course, this variable max random, which we introduced
and initialized to zero right before this second full loop.
And you're going to use it to keep in memory over this second for loop.
Well the maximum of these random draws.
So that's the new exercise for you.
Your new exercise is basically to implement step three with that same trick.
We already learned in the UCB implementation.
And now I'll see you in next tutorial to implement this solution with you.
And until then enjoy machine learning.