You have just learned the basics of web scraping.
It's pretty exciting, isn't it?
A lot of power at your hands, at your fingertips.
But let's say you want to go further.
Learn more.
Maybe become a professional web scraper?
I don't know.
Maybe that's what you want to do.
What are some things that I would recommend?
Well, get more familiar with beautiful soup.
But keep in mind what I said at the beginning of this section.
Ideally you don't even have to wipe scrape.
Ideally you can use an API.
Maybe the website that you want to get data from actually gives you an API.
For example, if I go to Airbnb API.
You can see that Airbnb has announced an official API.
So we can actually perhaps read about what data Airbnb just gives us access to.
And that should be your first stop.
Maybe if you don't have that, then you'll look at beautiful soup.
The next thing you might want to look at is this framework called Scrappy.
Think of it as beautiful soup, but with a lot more features.
A framework is essentially an entire kitchen, an entire tool belt that you can use to scrape websites.
And you use things like scrappy when you want to scrape massive websites, massive amounts of data,
a framework versus a library.
Is that a framework almost teaches a new language in itself.
It gives you all the tools necessary to do everything that you want.
When it comes to scraping data, beautiful soup, on the other hand, is smaller.
It's a library, so it gives you a few nifty tools to do small things like what we've done here.
Scrappy.
The framework well is an entire thing in itself.
You can go through their documentation and tutorial.
It takes a bit of time to get used to, but if you want to do this maybe full time scrapie is something
that you should take a look at.
Finally, what do you do with all this data?
Eventually you might want to do something, maybe store it in a text file, in a CSV file, something
that we've talked about and we'll talk about in the course.
But as you collect more and more data, maybe using something like Scrappy, you might want to start
looking into using what we call databases.
And databases allow us to store massive amounts of data and save it somewhere so that we can access
it whenever we want.
And there's a ton of databases, something that is beyond the goal of this course.
But you may have heard of things like MongoDB, Redis, my SQL.
Post gress.
These are all ways to store data, store massive amount of data and access it.
So you might want to learn about how to use a framework like Scrappy and collect that data and store
it somewhere on a database so that other people on your team can maybe access it.
The possibilities are endless, but if there's one thing that you're going to take away from this is
please be nice, be ethical, and don't do anything illegal because, well, I don't want to get in
trouble.
And I'm sure you don't as well.
I'll see you in the next one.
Bye bye.