Welcome back.
So we just received all the data from Hacker News.
At least on the page that we requested and we saw here that we received all this HTML information,
which is great.
This is just a string that we received.
What beautiful soup allows us to do is to do something called Parse.
That is, we can use beautiful soup to convert this from a string to an actual object that we can manipulate
and use.
Using Python to modify this data to what we want.
Let me show you what I mean.
So we can now use beautiful soup.
I'm going to remove the texture and we can say beautiful soup.
And then in here I can give it the string that we want with all the data that is response dot text like
we saw, which is the one on the left here.
And then we parse it, that is, we tell it, hey, this data needs to be modified into a HTML that
we can actually use.
So the way we do that with beautiful soup is we simply say HTML dot parser.
And this tells beautiful soup that, hey, this is HTML and I want you to parse it.
So convert it from a string to something that we can actually use.
And this creates what we call a soup object.
So I'm going to assign it to a soup variable, but I can call it whatever I want.
Now, some of you might be asking what this really is.
And the reason that we have to do this is that beautiful soup also parses another data called XML.
And again, if we go to the documentation website, let's go to beautiful soup documentation and let's
just search for a HTML dot parser.
You see over here that we can have different parsers such as the HTML parser which comes built in,
but there's also the XML parser and other parsers that we can install.
So again, any time you have a question, always refers to the documentation and just control F and
you'll find any extra information you need.
99% of the time you're just using the HTML parser as the default.
But what does that mean?
What did this just do for us?
Well, let's have a look.
If I print here the soup object that we just created and let's go all the way down here and let's clear
this for now and run our python code again.
Scraped up by.
All right.
We see here that we got the soup object.
It's actually a little bit cleaner.
It's no longer a string.
You see here that it's in full HTML.
We have the title, we have the head, we have the body tag.
So we can actually manipulate this as if it's.
Well, a python object.
So for example, I can do soup dot body to grab just the body.
So if I save and I run the script again.
There you go.
We now just received the body.
Let me clear this so we can see it.
I'm going to run it again.
You see that?
Now we have just the body.
I can even do something like dot contents.
And if I run this, let's clear this again.
I get the entire content but you see here in a list form.
Awesome.
Now using this soup object we can do some really neat things.
For example, let's say I want to find.
Using the find all method, all the divs that we have in this soup object.
So if I save, let's clear this and run our script.
How cool is that?
We get all the dev objects in a list for that's contained on that page.
What about all the links?
Maybe we want to get all the tags.
Which are the links of the site.
Well, once again, let's clear this for now and run our script.
And look at that.
We have all the attacks.
That is all the links on the page.
And you can imagine there's different uses that we can do here.
We can say title if I run title here.
We get the title tag of our soup object.
If I do something like a here, which is the a tag and I run the script again.
I'll get the first attack that comes up.
Another one that we can do is something called find.
So instead of find all find finds the well, the first thing.
So again, if I do find a and I click run here, you see that I get the same thing.
It finds the first item.
Now, why is this useful?
Because if we go to our Hacker News website and what I want to do is let's say go to the first link
here, right click it and then do inspect.
This gives us well, this HTML dom object that gives us all the information on the page.
Right.
So looking at this, I know that for sure.
I'm going to want this a tag.
That has the link and the title.
But I'm also probably going to want the votes.
Right.
So if I keep going down here.
To this part.
You see here, that right here is the score.
And I'm probably going to be interested in how many points this is as well.
For example, I can use the ID attribute here, so let's copy this score.
I can even do something like find and give it an ID that equals this value.
If I now let's clear all and run the file.
You see that?
I get that exact same tag with the idea of this score.
So by allowing us to grab these different elements throughout the page, we can selectively pick what
data we want.
Let's take a break here and learn more in the next video.