All right.
It's time to get started with our Hacker News project.
Now the first thing we're going to do is we're going to set up our environment.
In my case for this project, I'm going to use sublime text, but again, use whatever text editor you
want.
I also have my terminal or command prompt here and I've created a scrape dot pie file.
Now the very first thing we need to do in order to scrape using python is well, we need to download
Beautiful Soup.
This is what allows us to scrape websites.
So all I'm going to do is say Pip three and I want to install beautiful soup four.
This is the latest version.
I'll click enter.
And there you go.
Awesome.
We have a beautiful soup installed.
The other thing that we need is the requests library.
Now we're going to use the request library a lot in this course if you don't have it on your computer
already.
Again, we just use it using requests.
It essentially allows us to grab HTML files like Hacker News.
So.
What we're going to do is say pip three install requests.
So once we have that, what we're going to do and I'm going to show you why we had to download both,
first thing we're going to do is to actually import request.
Or requests.
And the other thing is the beautiful soup.
Now, the way we use beautiful soup is we say from these for.
We want to import.
Beautiful.
Soup.
I know it's kind of weird because we installed beautiful soup before, but this is how we import it
once we've installed it.
So from bs4 import.
Beautiful soup.
Now.
Why do we need these two things?
Well, beautiful soup actually allows us to use the HTML and grab different data.
So it allows us to use that data that we've gathered to do whatever we want to it to scrape it.
But the requests module is actually what allows us to download initially that HTML.
So let me show you what I mean.
The first thing we're going to do is create a new response variable.
And this response will be the requests and the URL that we want to grab the data from.
So you can think of this as a web browser that we're using without the actual window.
Because this is what Google Chrome is doing underneath the hood.
It's trying to grab the information of that website from a server.
So let's go back here and let's go to Hacker News again.
We're going to copy this link and I'm going to paste it in here.
There you go.
Let's make this a little bit bigger like that.
And what we actually need to do here is to say dot get.
So we want to get the information here.
So we're doing a get request to grab this page so that now if, let's say print response, I save this
and I run by saying Python three scraped up by.
All right, I get a response.
A 200 response, but.
Hmm.
Well, what does that mean?
What we're essentially doing here is if I open up view.
Developer and do developer tools.
I click on network here.
Let's make this a little bit bigger.
There we go.
And I refresh this page.
I get a lot of information, but the main thing that we care about is this the first document news here.
If I click over here, you see that this is what we get.
We get an HTML with all the data.
But we also saw here that the response was 200.
That means everything's okay.
The server responded with 200 and it gave us a document.
This is what we just got.
So in here we can actually say response dot text that we got.
So let me run this again and look at that.
We have our entire HTML file that displays the.
Text in here because we don't care about styling, right?
We don't really care about styling.
Maybe we don't even care about images.
All we care about is the text.
And for our purposes, we care about the actual link as well as the votes that the story has.
So we grab this data and now what we can do is use beautiful soup to clean up this data.
Right now.
This is a lot of information.
We don't really need this, do we?
Or at least we want to remove all the information that we don't need, such as stories that have less
than 100 points.
For example, this one over here with 176 points we definitely care about.
So let's take a break and see how we can use beautiful soup next.