Hello and
welcome back to the course on Deep Natural Language Processing.
Today we're looking at the Bag of Words model.
first thing I'd like us to look at is an email.
An email I received just a few days ago.
So here we go.
the email is about a catch up, and, my friend is asking, hello, Carole.
Checking if you're back in Oz or Sans for Australia.
Let me know if you're around and keen to sync on how things are going.
I deffo as a definitely,
could use some of your creative thinking to help with mine.
Cheers, Ava.
And so, what I'd like us to pay attention to.
First of all, of course, you can see that I sent this email to myself, but,
that's just because I wanted to keep my a friend.
actually is because I already replied to the email, and then I wanted to reset it.
And also I wanted to keep my friend, keep his privacy.
But this is a real email.
This is the exact text that I got literally a couple of days ago.
And, the titles would be different, but I just called.
I changed it to catch up. And so,
what is interesting about this, we're going to be looking at how,
we can apply natural language processing to this email
in the next couple of tutorials,
and it will help us work with a real life example.
And then the other thing is that you here, you can see in Google,
the Gmail app for iPhone,
you can see that it's giving me some suggestions.
Very interesting.
It's saying I it's already doing some quick replies that I can use.
It can be yes, I'm around and back or sorry, I'm not very interesting.
So let's keep that in mind and we will come back to this later.
In the meantime, text of the email is here.
What can we do with it.
All right.
So first things we're going to start off simple.
We're going to create a model.
We're going to
look at how we can create a model that will give us an A yes no response.
Because that's one of those questions.
the question is are you back in Australia?
Let me know if you're old and keen to think so. Yes.
No, of course it's better to have a long response.
And that's that's the social norm.
And, it's it's, added the etiquette to, like, converse with people,
not just say yes, no, but even let's try to get a yes no response.
Let's see how we would go about that.
Because that's the first step into NLP.
And then further on we will see how we can expand that even more.
All right.
So we're going to start off with with a vector a vector or a
just like an array, a full of zeros.
Yeah. Let's call it a vector. So these are like that.
So just 0000 is how many zeros.
Well, a lot of zeros.
20,000 elements in total.
20,000. Why is that.
Well it's because of the way that we're building as well.
20,000 is the number of words
that are commonly used by the average native English language speakers.
So here's a, quick search on Google how many words in the English.
So that's the search I took.
I came up with how many words are there in the English language?
171,476 words.
That's how many entries in the Oxford Dictionary, plus some obsolete words,
plus derivative words.
Yeah. And so on. But also,
people also you can see Google's giving us suggestion
that more subtle adult native, test takers
range from 20 to 30 20 to 35,000 words.
Average native test takers of age eight or, you know, 10,000 words, average
native test takers, a four or, you know, 5000 words.
An adult native test takers learn almost whatever
the science is going into so much detail.
but the interesting thing here is that,
how many like, what I wanted to point out, first of all, 20,000.
And you will see why exactly we use this number, not more.
what I wanted to point out is how many words are there in the English language.
Even this in its own is actually, Google is applying natural language processing.
It's it's looking at what we wrote and and then is also,
checking, other similar answers.
How many boards in the English language does that other person,
the average person, know. So that's not the question to ask.
But it came up with that. Then it came up with many other questions.
So you can see that the irony is that even in this search on its own,
we're already falling victim of natural language processing.
even though that wasn't our intention,
that's not what we're going to be talking about.
But it's just funny that it came up anyway.
So 20,000 words and, fun fact,
is that we actually use,
about 3000 words
out of those 171,476 words, we only used 3000 words,
not just in, conversational language, but you can see here,
a vocabulary of just 3000 words provides coverage
for around 95% of common texts,
95% of, common text that I like.
I'm assuming that's including books and stuff like that.
So if you do the math, it's why only use 1.75%
of the total number of words in the English language?
So as you can see, even that 3000 like our 20,000
is more than even the 3000 that covers 95% of the situation.
So we're pretty good.
We're definitely covered if we say that our vocabulary,
all possible words that we can encounter is going to fit into a vector of 20,000.
So every basically what we're saying, this is important.
What we're saying
is that every word in the English language has a position somewhere on this vector.
So for example, this the word f could have this position.
So if you count 123456 the seventh position in our custom
made vector, is that word events always going to be on that position?
That's very crucial for this.
For instance the word badminton let's just say like that
we can construct this vector any way we want.
The word badminton could be on this position.
It's always going to be on this position.
And the word table is going to be on this position.
And this is like how this bag of words model works.
So, just keep in mind that once you like, once we've taken all 20,000 words,
and then we've assigned them a space, that's where they that's
what they will this like space and this vector will be associated with.
They'll be associated with the word.
But if this will be associated with the word badminton, this will
this position will be associated with the word table.
and the other thing is
here you can see I've grayed out the first two and the last one,
first two are going to be reserved for source and iOS.
So stands for start of sentence, iOS stands for end of sentence.
And the last one will be reserved for special words.
And that's for those words that you're wondering about.
I can I can hear your brain churning right now.
What about those other 150,000 words that we didn't take into account?
What if they come up?
Well, if they come up, we're going to just associate them
with this, with this last thing, this last element.
We can just throw them all in there.
Any kind of words that we can recognize in the 20,000,
we cannot throw them into that lost element.
All right.
So let's go back to our email text.
Here it is. Hello, Carol. Checking if you're back in Oz.
Let me know if you are around etc. etc. etc..
Cheers.
V and so let's see
how this can be put into our bag of words.
If you've probably noticed by now
that this is our bag of words that we're constructing here.
So now we're going to throw the text into this bag of words.
How's that going to happen?
I'm just going to throw it in and then I'll just I'll explain how it happens.
So there it is. That's the result.
It that's it.
It all of course depends on how we construct our vector.
But this is our result in the way we construct our vector.
And let's let's look at this. way.
So we've as we've discussed previously, we took the 20,000 words
and we associated each position with, a word.
And now we go through our, text and find
and then like, increase the counter in each position of the associated word.
So hello, let's say, in our vector, it is in position number five
because we only have one. Hello.
In this whole email we're going to put a one here.
Cairo is definitely not an English language word.
So we're going to have to put it into there.
And the reason why there's three here is because we have Cairo.
Then also and V
those are non-English language words not among those 20,000.
They're all going to go here.
Then we've got the comma surprise.
The comma also has a position.
Let's say it was in position number.
So 36789.
So the ninth position is associated with a comma
because we have one comma in our email.
Oh actually we have two commas okay. So this should be a two.
But let's let's not think about that.
Let's let's forget about that comma.
I didn't notice it.
So assuming we have one comma in our email, this is a one checking.
And let's say that this, this, element is associated with our checking.
This is a one because there's only one.
We're checking if it's a two because we have two ifs in our email.
So it's going to be A2U is a two because we have two
used in our email including, the rest of the text.
I don't think there's any more use in there and so on.
So that's basically how we fill this bag of words.
We just put in the, the,
quantity of words for every position's pretty straightforward.
We're just,
filling in this vector, as you can see, is going to be quite a sparse vector.
It is going to be lots of zeros, almost 20,000 zeros.
And some of the words are going to be filled in.
okay.
So what is our goal.
So our goal as we discussed before is to come up with a reply
yes or no to this email, which is now in the form of a vector.
And how are we going to do that?
Well, we're going to do it through training data.
So we're going to look at all of the emails
that I have reply to because this is us training a model
to reply to my emails or in your case, in anybody's
case, it's going to be training the model or to reply to their emails.
We're going to look at training data. We're going to need some training data.
I'm going to fish it out of the inbox or outbox.
so let's say let's look at a couple.
So here we've got hey mate.
Have you read about Hinton's capsule networks and general reply to that. No.
so we're going to use that as a training example.
Next one.
Did you like that recipe I sent you last week?
The result? The answer was yes.
it was a good recipe, I guess.
So there we go. So now we have two. Three.
Hi, Carol. Are you coming to dinner tonight?
Yes, dear.
Carol, would you like to service your car with us again? No.
Are you coming to Australia in December? Yes.
And so on.
So ideally we would have tens or hundreds
of thousands of emails like that and responses like that. Yes.
No responses.
I of course, would be like a lot of groundwork to get that data
because we usually don't just respond yes no to emails.
So we'd have to look at this answer and understand what was the sentiment.
The sentiment was no. What was the overall?
Was it a yes or no? No. Yes or no? It's on.
of course, it's kind of more of a theoretical example.
Nobody's going to do this for their own inbox, but nevertheless the point stands.
So how would we train?
How would we use this training data?
We would use a similar principle and convert each one of those emails
to a vector.
in this and again each vector would be 20,000 elements long.
So yeah, I just threw some numbers in here to to get the point across.
It's not exactly accurate.
But so we have these vectors like lots and lots, lots of vectors.
Lots and lots and lots of responses.
Yes and no.
And yeah.
So now what we're going to do is we're
going to,
apply a model.
Once we have all this data we're going to apply model.
So one of the models we can apply to create our bag of words,
or one of the algorithms we can apply to create our bag of words
model is the logistic regression.
So we apply the logistic regression to our yes
no responses to these to this information that we have.
and then
once we have that model, once we've separated.
So we know we kind of like we've modeled
what goes like what goes into a yes.
Like what?
What is likely to yield a yes, what is like the T of the null
and the, border between them.
Then we can feed our actual, email that we got into this model
and then get a response service, for instance. Yes.
And that's it.
So we use all the training data to create a model.
We feed in our, actual email, which this is important,
which has exactly the same format.
So you can see that every input here, every
every time we were training the data,
the independent variable, the independent variable
vector always had the same length 20,000 and always had the same format.
So we know that this position always corresponds to a certain word.
This position is always a certain word.
This position let's say one, two, three.
Which why was where was it. 1234567.
right.
So this was one. No, this one is the if. Right.
So this corresponds to F or something like that.
So we know that it's, it's the same format.
It's always the same length 20,000.
So we can safely city in this vector into there.
It's got the same number of features.
we get an answer.
So for instance we get yes.
So and then we can like look back.
Oh what did the actual email say. It said hello Carol. Check it.
Oh okay.
So based on my training, I would have,
most likely reply to this with a yes.
Interesting.
The other approach that we can take here on, first of all, let's
put this on our diagram. There's our diagram.
And that's a natural language processing algorithm which is called Bag of Words
sits over there.
the other approach that we could apply here or take here is
we could instead of a logistic regression, we could use a, neural network.
We could because we have a vector. Right.
So we have all these vectors we could feed them into as an input layer,
like over 20,000 neurons into our, neural network.
They would go through.
We want to hidden layer two hidden on those as many hidden layers as we want,
our own decision on how to structure it.
And then bam, we've got an output layer and tells us yes or no.
And so we again, which is all this data
that we have here, all our millions and millions and millions of emails
and responses, would use that to train our neural networks
all through backpropagation and, stochastic gradient descent.
All the weights would be updated and bam, we have an answer.
So not bam, we have an answer.
So we would use these answers here
to train that, I would use the pairs like the vector and the answer vector answer.
So to minimize the error
stochastic gradient descent backpropagation updated weights. Bam.
We have a neural network.
It's all trained up now we feed in our vector here
which represents our new email into the neural network.
And voila, we get our answer.
And in this case, might also be yes, they might yield different result.
But if the model is constructed, well, more or less
it should be coming up with similar, or the same answers most of the time.
And so in this case, we've got a deep natural language
process going on by the input emphasis right there.
We've got a deep natural language processing algorithm.
Right. Because we're using a neural network.
And that, is different.
So in both cases the bag of words model, in one case it's an NLP bag of words.
In other cases a deep NLP bag of words.
but in both cases it is still a bag of words.
And it has its own limitations and it has its own.
yeah,
limitations and issues that are not that great.
And so I'll point out, one of them right now is that the response is very simple.
It's just a yes or no, right.
Like we want something more sophisticated.
We want like a conversation.
Can't really have a conversation, can't really build a chat bot
if you're just going to be saying yes no all the time.
So that's one of the limitations.
We'll talk about some more of them.
in upcoming tutorial.
And we'll also see how to overcome those limitations
and what models await us, in the future.
And, I hope you enjoyed this tutorial.
I really enjoyed going through all of this with you together,
and I can't wait to see you next time.
Until then, enjoy natural language processing.