Hello everyone. So in
this video we are going to see the nmap
robots file scan.
So the syntax of doing this scan is nmap the target
IP address followed by hyphen hyphen
script equals to the NSE script,
which is HTTP - robots.txt.
So you can scan a Target by simply giving IP
address and specify the HTTP -
robots.txt NEC file. This
will give you the directories which are allowed and
disallowed for the Crawlers to
scan onto the target web server.
Sometimes website disclose some
sensitive and important directories like
slash admin slash secrets.
/ confidential and they forget to
restrict them or to password protect
them this way you can identify those
vulnerable sensitive sensitive
directories.
and try to go there if
Any directory listing is present then you will be
able to find out.
sensitive and juicy information
So the example of doing this scan is
very simple and map the target IP address
followed by the script which
is hyphen hyphen HTTP - robots.txt.
So it is the Practical time and let's see how you
can perform this particular scan terminal.
I have given the command which is nmap.
the target server hacktified.in followed by
hyphen hyphen script HTTP -
robots.txt hyphen V
to see the output and verbose mode - p
followed by two port numbers, which is 80 comma
443
So now I'm going to see the robots file
for this particular web server.
As I hit enter you can see the scan has started successfully
and we have also got the output.
in the output if you see
In the robots.txt file, there is only this allowed
entry 1.
Which is slash CGI -
bin.
You can go to that particular directory and see if
that directory is not properly authenticated.
Then you may found out some sensitive files
into that directory if directory listing is
present.
So the output of this particular script is all the
directories which are disallowed which means
should not be allowed by the Crawlers like
Google Bank Apple Twitter
Baidu Etc. They should
not be crawled and we are able to see those disallowed entries.
So let us see the complete robots.txt file
and
what is the command to see the complete
file? So it is Curl. We're going
to use Curl and the target server.
With the endpoint which is robots.txt and I will
hit enter.
So now you can see this is the output of
the robots.txt file of this
particular web server where you can see the entries
for all the Bots and the Crawlers where
it is.
Shown that user agent googlebot.
This allowed nothing which means everything is
allowed to crawl onto this particular
web server.
As you can see there are no rules written but there is
only one entry of disallow. That is
slash. CGI - bin.
You can also try to do robots.txt and many programs
wherein you may identify any
entry which is disallowed but kept open
indirect realistic and you may
tend to identify some of the sensitive directories.
So I hope you guys understood this video. Thank you.