Hello everyone in this
video. We are going to see how you
can use nmap crawlers scan.
So in this scan, you will identify if our
crawlers are allowed to
crawl the particular website or the
target if
the crawler through which we are going to scan is
blocked by the website, then we
will not be able to scan that particular website or
send any type of request.
So basically, what is a crawler?
in this can
the website is going to identify the crawler based
on the user agents that we send
whenever we say a particular
request.
So you can scan a Target by giving an
IP address and specify the HTTP -
user agent hyphen tester to
test which crawlers are allowed to crawl the
target. You can always add hyphen V
to see the output in verbose mode.
And below is the example which is given to perform this
type of scan.
So it is the Practical time and let's see how you
can do this type of scan.
As you can see on the screen.
I have written the command, which is nmap.
Port 80 the script that we
are going to use to test the Crawlers if it
is allowed crawling, which is HTTP hyphen
user agent hyphen tester and our Target,
which is test PHP dot Woolen web.com and
hyphen V to see the output in
verbose mode.
As you can see I have run this command already
and I have also got the output. Let's
jump to the output and see.
We can see over here the status for the
browser user agent 200 means these particular
user agents were given a response
code of 200. Okay, which means they were
allowed to crawl onto that particular
Target.
So let us see one by one.
The first one is Mozilla, which means the Mozilla
agent left. Www.
Then you can see lip curl agent PHP python Snoopy
HTTP Lite PHP crawl Zend
HTTP client HTTP client wget is
also allowed and www.mechanics is
also allowed to scan and crawl the
particular website.
So I hope you guys understood.
This video. Thank you.