Let me ask you this.
If you were a website owner.
Would you want people to scrape your website?
Would you want people to access your data, maybe your Airbnb and you worked really hard to list these
products.
Different information on here.
Is it fair that anybody can come and just grab this information and maybe make a copy of your website?
Well, not really.
Right.
I mean, it depends.
Most of the time, you don't really want people scraping your websites.
Although HTML and the data of a website is available to all of us, it's just how our internet works
so that people can see different web pages.
Most websites don't really want you scraping their data.
Now there's actually a way that websites tell you what you can and can't scrape.
It's called Robots Dot Text.
We simply say at the end of a URL slash.
Robots dot txt.
If I click enter.
Look at that.
Airbnb has a robots.txt.
Robots.txt simply says hey, if you're a robot, if you're a web scraper or a spider.
This is why you're allowed and not allowed to do.
Does it mean that they'll prevent us if we break the rules?
No, they can't really prevent us.
But this is a grey area where you want to be ethical here and only scrape the data that a website allows
you.
So if we look at robots.txt, you see here that Airbnb has created a tax file that tells us what you
can and can't scrape.
For example, we look at the user agent here which says the Googlebot Airbnb is telling, Hey Googlebot,
you're allowed to scrape this information, but we don't allow you to scrape account alumni calendar
and you see that there's a lot of things that's disallowed.
What about Bing search bot?
Well, Bing, you're allowed to do this, but you can't.
Scrape any of this data.
We have Yandex.
What else do we have?
We have Baidu Spider.
And then we also have User Agent Starr.
That means anybody else, anybody else?
You're allowed to scrape this, but none of this information.
All right, let's go to another one.
Let's go to Udemy Dotcom.
Here we see that anybody we don't want you to scrape or crawl through any of this information, but
we do allow you to access this.
API 2.0, of course, lending components.
All right, so that's too bad.
It looks like Udemy doesn't really allow us to do anything other than this API 2.0 thing, which we'll
come back to.
What about Hacker News?
We're going to build a project around Hacker News.
So let's see if Hacker News allows us to scrape it.
All right.
So this list is a little bit smaller.
We see that for all users.
We're disallowed to use these endpoints.
So it's saying as long as you don't use these ones, these URLs, we're good to go.
That is, if we wanted to scrape just the home page.
Which is right here.
That's fine.
If we're trying to, let's say, go to comments.
That's fine too, because this one wasn't in robots text.
Again, if you remember, if we go to robots text.
That's fine.
The only thing that they ask is that we have a crawl delay of 30.
That is, if we're going to crawl their Web site, we should have a pause so that we don't constantly
use programs to hit the website because, well, this is going to overload their servers and maybe slow
down their website.
So what do we just learn?
We learn that there is ethical and unethical ways of scraping websites.
Yes, you can scrape any website you want.
And as a matter of fact, there's chrome plug ins.
If you Google Chrome plug in Web scraper, there's a ton of Web scrapers that you can use that allow
you to scrape websites without even writing a single line of code.
It's a good thing to practice ethical programming, and that means checking a website's terms and condition
before you scrape it, checking the robots.txt to see what you can and can't scrape.
Because usually the data you scrape from a website should not be used for commercial purposes because
you're using somebody else's hard work to, well, perhaps make money.
And this is why when we went to Hacker News and asked for a cruel delay of 30 seconds.
Right.
If you're requesting data from a website to aggressively with your program, that's essentially spamming
because this may break Hacker News.
Now, before we get into web scraping, I want to note one other thing, and that is this idea of an
API.
If you remember when we accessed Udemy Robots.txt, they did tell us that there is some allowed areas
that we can scrape or endpoints that we can use.
You see some websites actually allow you to access data.
For example, Udemy gives us an instructor API.
That is, if you're an instructor on Udemy, you can actually use their API.
You can see it right here so that you can get different data.
Maybe you can get a course review list, maybe a course questions list.
But if I go here and I enter this in my URL, I'll get an invalid missing token.
That is because right now I'm not an instructor, or at least I haven't verified that I'm an instructor.
On Udemy, it's going to say, Hey, this data is only for valid instructors, so I need to pass it
a token to prove that I'm an instructor on Udemy.
And a lot of websites give you what we call API or application programming interface access.
In a form like this called JSON to well, give you the data that you need and some websites only allow
you to do this if you're a valid user.
For example, one of the most popular APIs is this GitHub API where you can access different information
from GitHub.
But in order for us to use it, we have to, well, create a user account and validate so that we are
who we say we are.
But then there's also websites like the Star Wars API that gives us free access.
That is, anybody can use it.
We don't need to verify that we are a user.
We don't have to give a token when making a request.
Instead, we can just request data.
So for example, if I do the swap API and type in people and I hit enter.
Look at that.
I get all this data for free so that this well, I can create an app around the Star Wars, which is
what so happy stands for.
It stands for the Star Wars API and we can grab all this information that we want.
We also have things like the JSON placeholder, which I really like to use when building simple projects
where we can get different information like photos.
You can see here that this is an API that gives us information.
With different photos that we can use.
Maybe we have different posts that we can use.
And again, these are what we call in JSON format, which isn't HTML, but just a cleaned up version
of data so people can use it.
The easiest way to get data from a website is usually using an API.
Sometimes websites allow us to just access it easily, or they might have an API, but it's only restricted
to certain user.
Maybe you have to register with their service.
Maybe you have to pay for their API.
Well, in that case we have this.
But then there's also web scraping, which is maybe a website doesn't give you access to an API.
Maybe you want to just have that information.
For example, Hacker News doesn't offer an API as far as I know.
So instead we might want to just scrape the HTML, all the website and see what we can use.
So how can we scrape a website using Python?
Let's find out in the next video.