In this demo,
we'll look at the Spiderfoot command line tools and their capabilities.
We will start with the help documentation. Inside Kali Linux, we
will look at the spiderfoot command line help with this command
and the interactive command line tool used to remote access a
running Spiderfoot instance.
We will configure a scan from the command line and manage the scans and the data
those scans collected with the interactive command line tool. We'll also grab
the latest version of Spiderfoot from its GitHub repository. For this demo,
we'll be on the current Kali Linux, so we can skip the installation step and
start using the command line quickly.
There are two command line tools to know, the Spiderfoot
command we used in our first demo.
This tool can be used to start the Spiderfoot instance and
its web user interface with the ‑l option and specifying
the IP address and port number.
We can also use it to start a single scan without starting
a new running instance of Spiderfoot.
The other command line tool to know is spiderfoot‑cli.
This command interacts with the running instance of Spiderfoot.
We have to specify the Spiderfoot server URL. Once
connected to the Spiderfoot server,
we can execute a variety of commands to manage scans,
review scan data, and even start new scans. Before we do that, though,
let's begin by initiating a command line scan. This scan calls the dnsraw
module and gives it the example.com domain as the target. The ‑q and ‑r
help format the output to be more readable.
In this example,
we can see the various DNS records for example.com,
including its SPF and TXT records.
Here's another scan utilizing the accounts module to find
billg's account on a variety of online services.
You might be wondering where the sfp_dnsraw and
sfp_accounts module names came from.
In the first demo, we saw the listing of Spiderfoot
modules in the web user interface.
You can also see that list from the command line with the ‑M
option, along with a short description.
The problem with manually launching modules from the command line is
knowing what data types the modules watch and produce.
You may find that you get no output from a module you think should
work, but it doesn't accept the data type you provided.
It's not always obvious.
I spent a lot of time reading the module code to understand the various data
types individual modules accepted and produced. The ‑T option will show you
all of the data types Spiderfoot understands.
You can also start a scan and utilize more than one module. Here we're using
two modules on the command line, and both produce output. So far, we have
looked at using the Spiderfoot tool in single‑use scans.
Now let's start a Spiderfoot instance with the web user interface.
This is the same command we used before, and while that's running here,
we'll open a new tab. Using the spiderfoot‑cli tool,
we can connect to the running Spiderfoot instance and run some
commands. With the ‑s option, we specify the URL of the local
Spiderfoot instance. Notice that we have successfully connected to the
server. Here you can see the commands available.
We will focus on some key commands and leave the rest for you to explore.
If you use help with the name of the command,
you can see more details and the options available for that command.
The scans command will display the scans that have
completed or are running. With this list of scans, we can
pick one to examine more closely.
We have to use the ID of the scan to refer to it with other commands.
To look at the data collected from this scan, we use the data command
along with the scan ID. There can be a lot of information to sort
through here, so what do we do?
The data command documentation shows us that we can limit
the information to specific data types. What were those
again? There's a command for that.
Let's look for USERNAME. Now we can see a username we
can track down from all that data.
Another approach we can use is to look at the data summary. With the
summary command, we can see the data types collected on the left and Total
and Unique counts of that data type on the right.
This view might help us identify data types of interest.
There's also a find command to search through the scans for
strings or using a regular expression.
Let's start a new scan.
We can see the options available to us.
Let's launch a scan, check its status, and then stop it before it finishes.
We are targeting the hackthissite.org domain, and we're looking for as
much data as we can get. With the scans command, we can see that it is
just starting, but let's stop it now. Notice the message. We have
signaled to the scan to stop.
This may not happen right away, as some modules may be still executing.
Now we can see that the scan has stopped and is listed as aborted.
Finally, we want to walk through how to grab the latest version of
Spiderfoot. Kali may not have the latest version of Spiderfoot,
but we can get it from GitHub. If you search for Spiderfoot in GitHub,
you'll find your way to this GitHub project page.
We need to clone the spiderfoot Git repository, and we need the details to pass
to the git command. We just need to copy the repo URL. The git clone command
will copy over the Git repository and its files.
Now we have a spiderfoot directory with files. We can see
that this is a newer version of Spiderfoot.
Check out the README file for information on
installing any Python required libraries.
Also,
please note that we have been using the Kali version which uses the
commands named spiderfoot and spiderfoot‑cli.
Spiderfoot actually names those commands sf.py and sfcli.py.
Other than naming, they work the same way.
Now let's take a look at what we learned about
Spiderfoot and use it in a red team scenario.