Extracting directory structure, crawling, so mapping the application layout and its structure is another
very important task.
Although there are some single page applications, the applications you will test can generally consist
of multiple Web pages.
And by multiple, I mean a lot in these pages can be independent or they can be linked to one another.
There's actually no magic to get the structure unless you're the actual developer of the application
and it is hours and hours of work.
So the best way to extract the structure of the application is to visit every page and click every link
and fill every form, then observe all the URL so you can manually walk through the application and
identify Web pages from authenticated and unauthenticated users perspective.
So this whole process is called crawling or spidering the application.
Now, you might think and you would be right that it's not really possible to do everything manually.
This really is a time consuming process.
So, gosh darn it, wouldn't you just love it if somebody created a tool for crawling, and how about
some scripts for good measure?
Well, I'm glad you asked.
Because now let's go over to Cali and open up your browser and berp, I know if raised like me, it's
rude, but I'm going to ask you to do it again.
Open your browser and burp.
OK, so sorry.
Go to Dashboard and make sure that capturing is active.
Then disabled master interception from this button.
OK, so za'atar.
But in your browser, redirect the traffic to berp by enabling Foxe proxy.
So now Berp will passively intercept all traffic coming from Firefox's.
And what we're going to do here is visit pages and click links.
So so that way, berp will create a structure of the advocation from the clicked you earles by intercepting
passively.
Right, so I'm going to click on some links here on the page.
And then log in.
Open up a few other pages and.
OK, that's enough to show you.
So, OK, look, the history in BIR.
So all the requestor here.
And so the aim here is to see a sort of a broad overview of the application layout.
So.
There are a few more things in this step to perform, but I want you to just have a look here.
I'm sure that you've heard about robots that he asked.
I know almost every website uses this file to allow or disallow directories to be called by bots, so
robots dot text is a file that uses a specification protocol called robot exclusion protocol.
Now, it's not really something that you need to consider, but you might want to have a look at a later.
But, yeah, displaying this file is very handy if you want to see the sensitive pages and directories
easily.
So what am I saying?
Yeah, go to the robots dot text file of Boab.
And here are some directories that are not allowed, so I'm going to visit each of them admin directory.
Documents.
And images and then wait, what's this password's directory?
Now go to berp.
Click Target tab.
Quick site map.
On the left pane, you can see the site structure.
And also, there are filter options above, you know, this is great, I hope you get excited, as I
do so by clicking show all resources like access and images, they're all going to show up on the map
as well.
I think the commercial version of Berp will do this task automatically, but I really do want you to,
you know, get your hands dirty, as I was saying before.
I know it's not the most efficient way, but this is how you learn, so we need something a bit more
intrusive.
Now, I should think so, Kelly.
Linux has a number of tools for this job.
There, Buster is one of them.
So open up the terminal and simply type there, buster.
And again, we will come up.
There, Buster, is the directory brute force here for Web applications.
So now let's provide the target, you, Earl, which is HTP Cohen, so that says one nine two dot one
six eight two zero four that one three zero B Web slash.
And here's a starting point for door buster, and it will automate the tedious tasks of cataloging the
pages within the application.
That sounds good, huh?
So it works by requesting a Web page.
Parsing through it for links and then sending requests to these new links until all the Web pages are
mapped.
So then let's increase the number of threads here to 20.
And then choose a list, Brouse.
Now, there are several lists in der busser directory under Wordlist.
And I'm going to choose the medium directory list.
OK, and QuickStart.
Oh, and another good thing is to identify administrative and test pages, these pages can contain sensitive
information and provide entry points to perform attacks such as a brute force attack.
And it's also possible to see old and backup files in the directory structure.
Don't laugh, I've seen it many times in real world situations.
So if the old version of the application functions and has any vulnerabilities, bingo, you can own
the entire system.
Besides that, the folders and files belong to the application, so there may be meta files and folders
of the server software as well as the application framework.
And I mean, no one files and folders such as my admin and so forth.
So Derbez here also looks for this kind of stuff.
And here, as you can see, it detects Drupal.
My admin and escarole light.
OK, so at this point, I'm going to stop this scan and you can take a report.
Full text report.
Browse for the location is safe and.
Give it the name.
I'm just going to type B rap and generate report.
So now you can go to that folder.
And here is the report we saved.
So now you can analyze it.