We have talked a lot on the networking side of the house.
Let's move into the web and web application pen testing
and some of the available packages,
modules, and functions you can take advantage of. Requests is a
well‑documented package that can be found on the pypi.org site.
There are already a few guides available here on
Pluralsight for requests as well.
One of them gets into the interesting use case that we will
explore on web scraping and requests.
This is more or less developing your own bot to
retrieve information from a website.
There are a ton of proper use cases out there for this type of
service, but keep in mind as we go through this that this starts to
border on pen testing and you'll want to be sure you have permission
of any application you reach out to before running the script
against it. To get started,
requests did not require you to pip install it in
order to import the functionality.
A few of the key functions to talk through.
Now keep in mind, we're talking about web calls into the web
application, but the get and post are the main methods for
retrieving and sending information into a server.
Your browser normally handles this behind the scenes while
interacting with things like web application forms.
But since we're incorporating this functionality into our Python scripts,
it will be on us to manage in the code. In addition to the get
and pull, all methods, including put, patch,
head, delete, and options, are available to you. Status_code is
important to track any issues and that your code is operating as
expected along the way. As we make our get and post requests, it is
important to know if these were successful or not.
Text and content both allow you to access the response
that comes back, which is usually in the form of a source
code for the page you called.
Text returns the information in a nicely formatted Unicode response, whereas
content is returned in bytes with all the new line characters and is harder
to read, but easier to work with in your tools.
Finally, the last function I'll quickly talk about is the header function.
Calling the response headers can expose you to all sorts of useful information.
If not locked down, the header can expose what type of server you're
running on along with the version. Information leakage is one of the
OWASP Top 10 items you'll want to look for with this tool. As
mentioned, requests mimics all the functions your standard web
browser handles for you.
It does require you to pip install the package in order to
import the functions, and in this first demo, we're going
to make a get requests call.
This is a standard retrieve of the website example.com.
With the data returned, your browser would normally read in
things like the status code, header information, and content body
to determine how to display the data.
When I run this demo, you will see I pull in the same
information and instead display the individual parts.
Why would you care about the status code in this way, you may ask?
Well, availability is part of the core security triad.
What if you needed to monitor 20, 30 or even 100 web applications
your company maintains? This information allows you to build a
dashboard that monitors the status code every couple of minutes and
alerts you when the site goes down.
Now going back,
I can show you we also pull the content into nicely formatted Unicode.
Let me comment this out.
So when I run this now again,
I pull back all the different source code for the
same example.com web application.
I also have another option of the print content, which brings it back in byte.
A little bit harder to read, but all the same information.
I also have a second demo to run. This one is going to look at the
web application Juice Shop, which is available on the Heroku app.
Juice Shop is a broken web application that's made for web
application assessors to go through, test their tools,
test their skills, try to find all the vulnerabilities of the application.
In this case, I'm going to do some web scraping.
I'm going to open up a get request again that's going to go to the
directory in the web server where these images are stored.
I put a list together of a few images that I thought would be good to
pull in, and if found, if I return that 200 status code,
I'm going to grab that image and write it to my server.
So what you're going to see when I write this is at the top here,
these images are going to start to appear in the list.
So any image I was able to find on the web server s now printed the
picture and scraped it off the Web application into my site.