Welcome back.
This was probably an exercise that sounded a lot more complicated than it actually was.
So hopefully most of you were able to solve it if you didn't.
By the time you watch this video, you're going to say to yourself, Oh, wow, that was so simple.
I should have figured it out, so don't worry, we'll get there.
Let's see how we can do that.
Well, if we go to Hacker News, you might notice something.
When you click on more, when you click on more, you actually get a new URL here with well, you see
the query string, which is a question mark here that says P two.
So if I do P equals three, here we go to page three, we go to page four and so on and so forth.
So all we actually need is just this link, and all we need to do is go all the way up to our code.
Let's minimize this for now and simply just duplicate what we've just done.
We?
Grab a second response and this response will be requested yet.
And this link with page two.
From here, we simply grab.
This soup and recreate it as soup to.
And that's going to be from the response to text.
Again.
What we'll do here is we'll duplicate this code.
And again, if you notice that this could be more efficient, maybe we can extract this into a function.
That's good.
You're starting to notice these things.
So here we're just going to do links to subtext to.
Well sure grab from our soup too.
And well, soup to.
And we can now just combine these two things by simply calling, let's call it mega links, which will
combine the links with the second links.
So we're combining the two pages and we'll do the same for mega subtext.
Which will be subtext plus subtext to.
And because we've run our code in nice functions, all we need to do now is replace our parameters,
right?
All we need to do at the bottom here is say.
Mega links and mega subtext if I save this.
This should not work.
Let's have a look.
Oh, and classic me made a bit of a spelling error if we go over here.
It's not subtext.
It's mega subtext.
Let's run this again.
You'll see here that it takes a little bit longer again, because we're going to two pages.
But look at that.
We have our Hacker News using two pages.
You can see that it's clearly longer.
And we have a story here with 951 votes.
I should I should definitely read that.
How to access quality of garments sounds very important.
And there you have it.
You have now just covered the basics of scraping.
As you can see, it's very specific to a website that you want to scrape.
Remember, you want to check to make sure that, A, you're able to scrape the websites by checking
the terms and conditions of that website and robots.txt and try to use the data in, well, not nefarious
ways.
Be nice.
Be ethical.
In the next video, we're going to talk a little bit about how to expand on this and what you might
want to try if you want to get more and more involved with web scraping.
I'll see you in that one.
Bye bye.