Welcome back.
We have our data, but it's still kind of dirty.
I mean, we haven't really filtered.
I only care about the stories with over 100 votes and we also want a pretty way to display them.
So let's work on that.
I want to create a function and we'll call it Create.
Custom Hacker News and this create Custom Hacker News is going to receive links and votes.
Which we have above over here.
Now in here, what we're going to do is we're going to create a new hacker news list that is empty to
start off with.
And for now, all I want to do is return the hacker news list.
But within this list, I only want to add the text and none of the HTML.
So for example, for the links, we only care about the title.
So how can we do that?
We want to loop through some things.
What we're going to do is say four and we'll use enumerate here.
So we're going to have an index and then an item.
And in here we're going to enumerate remember, we've seen the enumerate before, which gives us an
index as well.
And we're going to enumerate the links and I'll show you why we want to enumerate in a second.
So in here I want to grab the title of each link.
By simply doing this, we're going to say Link.
Of index.
Or Let's change this to ID X, that's a little nicer.
So we're going to grab the.
Index of each link.
And we can use something in beautiful soup called get text.
And this is going to get the text inside of the text.
And just to get us started on this, I'm going to just simply say, how can you use dot append and add
to the list the title?
So let's see what happens when I call this.
I'm going to say Create Custom Hacker News and we're going to pass it the links and the votes.
So let's save run this.
Oh, and it looks like I cannot spell enumerate here, so let's fix that.
Should be a new merit with EA.
Let's try that again.
All right, no errors, but we want to ideally print this.
So let's just say.
Print.
And see what we get.
I'm going to run this.
All right.
Look at that.
We have all of the texts, the titles of our links.
That's great.
Okay.
But these titles are useless without a link.
Right?
And if you remember a link in an HTML.
Is defined with what we call an H ref.
This H ref attribute is the link that we can click to go to the next page or to the page of this title.
So we can also grab that and we can call it h ref.
And this is going to equal.
Links.
Index.
And this time around, we don't want to get the text.
We want to get the attribute right so we can actually say dot get and then grab the h ref and just in
case there are no refs or maybe the link is broken, we can give a default in the second parameter and
I'll just say none.
So that now this hacker news append if we just append to let's say this time the ref and I run this.
Look at that.
We have all the links on the page.
That's awesome.
But how can we combine the title and ref now?
Well, this should be fairly simple, right?
We can use a dictionary.
So in our list we can append a dictionary that has the title, which will be the title that we have
here.
And then also.
Have the link.
That will be the.
Right over here.
So here.
Now we'll have a dictionary with title and link.
So if I save.
And run this again.
Look at that.
We have a dictionary that has a link.
And also a title.
Awesome.
So we grab the information that we need, we grab the link and the text.
None of the other stuff matters to us, but we still have the votes.
So how can we do that?
The votes.
Well, we can create a variable or instead of votes just to distinguish it from this, let's just say
points.
And points.
Is going to be, well, what are we going to do?
We're going to grab the votes.
Index.
Remember, we're looping through this and we're simply going to say, get text.
Now let's see what we get initially when we do get text.
So in here, I'm just going to print.
Points and see what we get.
And I'm going to remove the print from here just so we can focus on the points for now.
So I'm going to save and run.
All right.
We get an error, we get all these points, but it looks like I get an error saying list index out of
range.
And this is a little gotcha that you're going to have to sometimes debug when scraping websites.
But essentially what happened here is there's a story and this is a special one that actually doesn't
have any votes, which is very rare on the front page of Hacker News.
But this one over here doesn't have any votes.
So essentially what it's saying is I never grabbed the scores, the points of this.
So we have technically more links than we have votes.
So by the time we get to the to the last link here, well, the list of votes is one less.
So it's going to say, hey, I'm trying to grab the votes at the very end, but there's no more votes.
This list is shorter than the links list.
So we can just do a quick check here to make sure that when we grab these things that we only care about
stories with votes.
And luckily for us, this is why we want to check if a story is more than 100 points.
The way we're going to do that, we're going to add an if statement right here.
Now, before we add the if statement, we actually want to convert this into a number so we can say,
hey, if a story has more than 100 points, then we want to keep it and append it to our new Hacker
News.
So the first thing we want to do is to simply say.
That we want to convert this into an int.
But we also want to replace.
The points here, the space points into, well, nothing.
So we can use the dot replace method and simply say, hey, any time you see space points, which is
what we see over here.
I want you to replace that with just an empty string.
So this is going to return for us a string that is a number and then convert it into an integer.
So let's test it out.
Let's close the bracket here.
Hit safe and run our script.
And there you go.
We have our numbers.
That's great.
But again, we still have this error.
So what can we do here?