Welcome to this brand new module. 1
Let's start with outlining the premise and talking about the problem that we're going to be trying to 2
solve in the upcoming lessons. Imagine that you're looking for a job in the field of data science or 3
machine learning. 4
Every company out there is swimming in data. 5
So there is a lot of demand for data scientists and machine learning experts. 6
And that's good news for you because, guess what, you've just found a position advertised in the career 7
section of your favorite company. 8
So naturally you go ahead and you polish your CV and you carefully research the company and you write 9
a cover letter and you send off your application and you start practicing your interview questions with 10
your most patient friend or career advisor. 11
Now, meanwhile at the offices of your favorite company that you just applied for, the job applications 12
from the candidates just start flooding in. 13
There is many, many applications and it's not even easy from the company's perspective to select a good 14
applicant. 15
So you know, even though there's a lot of job opportunities in the field of data science and machine 16
learning these days, it's still super competitive. 17
In fact, the classic job application process of selecting candidates based on their CV's, their cover 18
letters, maybe numerical tests and then how people do in interviews, just isn't really working that well. 19
So what more and more companies are doing is giving their applicants a project to work on. 20
For example, my friend recently applied for a job at Uber and prior to the interview they basically wrote 21
him an email that said "Well, we need you to predict demand for our taxis. 22
Here's some data. Send us your solution to this problem by next week." 23
Bam. 24
And that was it. 25
My friend spent the next week toiling over this problem to produce a solution for these guys and move 26
on to the next step in the application process. 27
So, let's imagine that this is exactly the kind of situation that you find yourself in right now. 28
You receive an email from the company that you've just applied for in response to your job application 29
and they liked your CV and they're giving you this opportunity to show what you're made of during a 30
project. 31
Here's the gist of the email "Our team receives too much spam. 32
We only want legitimate emails in our inbox and we want all spam to be filtered out." 33
"Here's some data." 34
"Send us your solution by next week." 35
Tick tock, the clock is ticking. 36
So what do we do next? 37
Well the first step is always formulating the question. 38
And what this usually means is translating a business problem into a machine learning problem. 39
Now, what I mean by that? 40
Well we have a clear business objective in this case from the company - we need to filter out spam emails. 41
But what does this mean from a machine learning perspective? 42
Well, to filter out the spam emails, we first have to know which emails are spam and which emails are 43
legitimate. 44
In other words, for each incoming email we have to assign a category. 45
If the e-mail is legitimate then we have to classify it as Not Spam and the e-mail is then allowed to 46
go on to the inbox. 47
However, if the email had the characteristics of a spam email then our algorithm must detect this and 48
label this email as spam and classify it as such. 49
Thus our objective is to learn what characterizes a spam email so that we can start to classify all 50
the emails. 51
So if you think about it, this is actually quite a different kind of problem from when we were estimating 52
the real estate prices in the previous module. 53
Previously, we had to estimate a quantity - given a set of characteristics for a property, 54
we were estimating the house price. 55
We had something that was called a regression style problem. 56
This time we have a different problem. 57
We have to categorize things. 58
We have a classification style problem. With regression 59
you're fitting to the data, but with classification you're separating the data. Regression and classification 60
style problems are in fact two of the very, very common kind of problems that you will find yourself 61
solving. 62
Typical examples of regression problems have to do with estimating quantity, like say next year sales 63
or how much time a particular task takes and common examples of classification style problems are things 64
like segmenting your business's customers, detecting cancer, or in our case filtering out spam emails. 65
But you know, the thing is there is an additional niggle with this problem here that we were given. Previously, 66
we were working with numbers, we were doing calculations with these numbers and feeding these numbers 67
into our algorithm. 68
But now we're gonna be dealing with emails instead. 69
We're gonna be working with text data. 70
So how do we train our algorithm when we're given text? 71
Can we just pipe in our emails directly into the algorithm and get a sensible answer? 72
Probably not. 73
Text data just isn't suited to run calculations on in its raw form. 74
So this means we have to find a way to translate our text data into a format that an algorithm can understand 75
and work with. 76
We have to process the emails before handing them off to our algorithm to do the calculations. So you 77
can see there's already a lot of challenges ahead. 78
But we've already taken the first step. 79
We've looked at the business problem at hand and we've translated this problem into a machine learning 80
problem. 81
Here's how we can formulate our objective in a machine learning kind of way. 82
We're gonna take the emails from the company and the first thing we're gonna do is pre-process that 83
text data, then we're going to train a machine learning model that can classify an email as either spam 84
or not spam. 85
And finally, we're going to test and evaluate the performance of our trained model. And with our trained 86
spam classifier at hand, we can filter out all incoming spam emails very, very quickly. And you never know, 87
right, by sending our train model back to the company along with an explanation of our methodology, 88
we should be able to clinch that job offer or at least get invited to the next round for an interview. 89
I'll see you in the next lesson. 90
Take care.