Hey there.
I want to add a quick note about what we just saw with Airbnb and how search websites like Bing, Baidu
and Google actually work.
Now you may have noticed that the user agent Google bought is, well, what Google uses to index websites.
What does that mean?
Well, Google the way it works, remember, is a search engine.
What it does is it tries to figure out all the websites on the Web and index them.
That is, create a essentially a database of all these Web pages so that when I Google soup recipes,
Google gives me a website that has soup recipes.
Now.
How does it know what websites have soup recipes?
Well, it uses this Googlebot and this is how Google got started.
What they did was they had a crawler that is a program that crawled websites and read websites, kind
of like we're about to do with web scraping to understand what the website is about.
If the website mentioned the word soup recipes or had different text related to soup recipes, well,
Googlebot is going to say this website is about soup recipes.
So when Andre searches soup recipes, I know that these websites are the ones that we want to show them.
And this is what Google and Bing and Baidu all do.
They have their own bots and they're constantly crawling websites to see what is the most relevant website
to give to a user.
So Airbnb here is saying, Hey Googlebot, when you come crawl my website, don't crawl any of these
things because I don't want you to.
But anything else that's not here, you can crawl.
Now, if you actually look through here, this is not everything on Airbnb because Airbnb wants Google
to crawl the website, right?
If I search for vacation homes in Costa Rica, Airbnb is interested for Google to post an Airbnb link
to the top of the search.
So what they disallow here is mostly user specific thing that Google shouldn't have access to, like
maybe reservations or payments.
But there's a lot of other endpoints that are actually allowed for Googlebot to search.
For example, Googlebot is most likely allowed to look at all the houses available in Costa Rica and
actually rank them so that when I type in Costa Rica house rental, well, Googlebot is going to say,
Hey, Airbnb has a link to Costa Rica houses, we'll give that to this user.
So anything that's not in this list, well, we're allowed to use.
One tricky thing is that we see this allow here.
What this allow means it says, hey, calendar here is actually a disallowed endpoint.
However we allow inside of the calendar.
This is a folder you can access that one only, but if it's not mentioned here.
So if anything in here on this list is not mentioned, that is allowed.
So for example, Airbnb, dot slash that's allowed because that wasn't on the list.
If I maybe click on experiences, this is allowed because this s slash s slash experiences was not on
the list.
So a website either has a robots.txt file because maybe it doesn't want specific things scrolled or
if it doesn't have a robust text file or just leaves it empty, it means, hey, I don't care Google.
You can rank my website, you can crawl my website.
That's good too.
And that's why Airbnb had different information here.
We had specifically for Googlebot, for Bing bot, for all these search engines, but also for everybody
else, maybe programmers like us that want to scrape the website.
There you go.
We secretly learn how Google and other search engines work.
Pretty cool, right?
All right.
Let's get back to the topic at hand.