Beautiful Soup is a Python library used for retrieving elements
from an HTML and XML document using a parser.
This allows you to scrape and extract data from a file or
to modify the file using Python code,
then writing it back to a new HTML file with your modifications.
Like all the other libraries we've discussed thus far,
Beautiful Soup can be found on the pypi.org site with the documentation
out of the maintainer's personal crummy.com site.
This is a third‑party package and does require you to pip
install it before being able to import and take advantage
of all the built‑in functions.
To begin,
you will basically pass a BeautifulSoup constructor
with the same name two arguments,
the code and which parser you want to parse it with.
The HTML can be a local file.
Some developers use this tool to read in a file they are working on,
review and modify it with the Python code,
then produce a new, cleaned up file.
Generally,
when being used as an analyst tool or for some type of bot or scraping,
you need to pull down the HTML from the server using a
request call and parse the data that way.
By default,
Beautiful Soup will work with the built‑in Python
HTML parser that comes standard.
It is recommended that you use the lxml for speed,
especially if you're working off an older version of Python.
The built‑in html.parser is not very good.
Lxml and html5lib are installed with pip separately.
Beautiful Soup is a rich library full of all sorts of functions
broken down and specialized for specific tasks.
I'm going to talk about three of them today,
starting with the find and find_all functions.
The most important use of Beautiful Soup is to parse the file,
find and pull out all the elements you need.
This could be standard tags.
For example, retrieving all the a tags will give you the links of the site,
which can be used for spidering.
If I'm looking for data leakage,
I can search for all the comment bracket patterns,
or in the header, is it disclosing server information?
For purposes of scraping images and data,
the find and find_all allows you to drill down on a page
and gives your parser a starting point.
Find_all returns a list containing all the results,
even if there are 0 or 1 of them.
Find simply returns the value found.
Some of the additional search capabilities include find_parent,
find_next, and find_previous,
which allows you to move into the tags surrounding your starting point.
Beautiful Soup is one of my favorite packages.
It does need to be installed with the pip,
which is currently set to install Beautiful Soup 4. This creates a bs4
package that I pull Beautiful Soup out of to import. The first couple
lines here are pretty straightforward.
I'm retrieving a website on my local web server I
created that contains a catalog of coins.
I'm using the requests get call to handle this, and I am then sending the
text into the Beautiful Soup's constructor and parsing the documents with
the lxml parser. This will give me my tree of Python objects I can then
play with. Looking at the website in question, you see I have a catalog of
coins that provides pictures, names. and prices.
If I'm a collector,
this may be good information I could use to help find
the best value for my collection.
In the source code,
I can see the individual coins are separated by div tags,
which has the class product assigned to it.
Knowing this,
I can break this tree down by individual products and retrieve the
information from inside each of the div tags.
I'm using the built‑in find_all function to pull out the individual
products, but another way around this is to reference back to the regex
section and build a pattern and search for that inside the code. Because
class is registered inside the Beautiful Soup,
I have to add an underscore to the end of it in order to use it.
Once I have my list of all the div groups that contain the coin information,
I can import them one by one and split them.
Looking at the source code again, I see there's a line break at
the end of each line that separates the money from the description
and the description from the picture.
Using my knowledge on strings and list parsing, I am going to
grab the directory and name of each picture,
retrieve the image, then call out the image, product, and amount in the results.
Normally I would drop this into a database at this point, but once finished,
I should see all my images in the file directory and the details in the
results. As you can see here, I have all the detail from all the coins, and
on the left here I have all the images.
Developers use this package to help make wide‑scale
changes across their web application.
It's a lot easier to parse content through automated means instead of
having to update content manually. With the way we develop applications
changing, this will slowly become less and less important. But for now,
you will have this tool available to you for your various needs however
you see fit. Wrapping up this course, we looked at close to a dozen
modules and packages that really help make your job in InfoSec easier.
Basics in sys, os, and psutil get you into the processes and operating system.
They contain functions to retrieve metrics,
run jobs, and automate a ton of the quick and dirty work
needed in things like incident response.
Re, or regex, is a search pattern package.
You will have your hands dirty in files, logs, and other data‑heavy documents.
Being able to pull the information out quickly is a huge benefit.
Cryptography is where we start to get into the heavier security
functions where we're able to hash, encode, and encrypt, along with
cracking into the X.509 certificates, securing files and hiding sensitive data.
YARA was purposely built to make your life easier.
The rules can be imported and ran to help find and fight phishing, help with
incident response, and determine if you've been exposed to the latest
zero‑day compromise. Sockets and Scapy focus on the networking side, while
requests and Beautiful Shop focus on the web.
All four provide you with the help to build a bulk of the
tools you will use in this industry.
Well, that's it for this course. Attached to the exercise files
are additional lists that contain dozens and dozens of other
packages that complement these libraries.
The best advice I can give you is to do your research,
understand all the nuances of the functions provided in play.
The more you code, the better you will understand, and the better you understand,
the more ways of implementing you can come up with and the
better results you will see. That is all for now.
Thank you for watching, and keep an eye out for more Python
and security courses that will expand on the detail of some of
these packages in the near future