To be effective, Red Teams must have key information about their targets.
Hello, and welcome to Reconnaissance with SpiderFoot.
My name is Keith Watson, and I'm an Information Security Professional.
I hope you've enjoyed my other courses on information
security and penetration testing at Pluralsight.
Here we'll turn our focus to a key tool for Red Teams: SpiderFoot.
Having the right tools to recon the target leads to more
efficient and effective Red Team testing.
Red Teams need to know data about network services they can explore,
technical details of systems they could exploit,
business processes that could be disrupted,
as well as information about key individuals in the organization
to which they could apply social engineering.
This information feeds the planning process,
threat scenarios, threat actor emulation,
and execution activities during the engagement.
Reconnaissance is one of the first steps for a Red
Team testing against a target.
It consists of both information gathering activities and an
analysis of the information collected.
This process is often a cycle,
as more specific information may be needed as the
analysis of the gathered data progresses.
The team may take advantage of open source intelligence resources,
as well as more active technical approaches that
probe a target for data directly, or passive methods,
which utilize information maintained or gathered by others.
As information is collected,
we can categorize it based on its type as either technical
information about the target's digital assets,
administrative information about the target's network domains,
assigned IP address ranges, autonomous system numbering,
and outsourced services,
physical information about the location and characteristics of its offices,
data centers, ancillary facilities and operations,
and co‑located digital assets.
And human‑oriented information for specific employees,
leaders, and the organizational structure.
With that information collected,
an analysis is needed to determine which data elements would be most valuable
in the planning and execution phases of the engagement.
The first consideration should be the threat profile.
What threat actor group is the Red Team emulating?
What is the intent of that specific threat or their why?
Are they gathering sensitive information or just stealing it?
What are their TTPs, their tactics, techniques,
and procedures for how they drive their attack?
What information is needed to emulate a specific threat?
A threat scenario is a story, if you will,
around how the Red Team will focus on the target security
defenses against a specific threat.
During the analysis, the rules of engagement must also be considered,
as not all data elements might be allowed in the execution of the engagement.
From the analysis will come actionable targets on which
the team can direct their efforts.
The Red Team may plan to target specific business assets,
or try to manipulate a set of business processes.
The outcome of the gathering and analysis activities then
feeds into a threat‑modeling phase.
SpiderFoot simplifies the open source intelligence collection process for
Red Team operations by automating data gathering from many online
information sources through an easy‑to‑use interface.
Steve Micallef originally developed SpiderFoot as a learning exercise.
It started as a C# project after he saw a demonstration of an early
domain name reconnaissance tool in the mid‑2000s.
Steve posted an early beta version of SpiderFoot in 2005.
The project sat untouched until 2012 when it was rewritten in Python.
Steve was able to utilize a large number of information sources
and public APIs that were newly available at the time to expand
the usefulness of SpiderFoot.
It starts with a simple search target and utilizes its collection of modules
to access a variety of information sources on the internet.
As new data elements are discovered,
SpiderFoot will trigger other modules for those specific data types.
The result is an organized view of relevant information
collected from the initial search target.
The primary site for SpiderFoot is spiderfoot.net.
It's an open source project with its source available on GitHub.
SpiderFoot has a modular design,
allowing developers and Red Teams to create their own unique modules to
enhance Spiderfoot's information collection capabilities.
Depending on the type of information needed,
SpiderFoot has selectable use cases.
Also, specific modules can be selected for a scan,
or the data types needed can be specified.
Multiple scans can be configured and launched independently.
Each scan's collected data is separate from other scanned data.
SpiderFoot uses internal data passing between modules.
Each module produces one or more data elements that other modules may utilize.
SpiderFoot has both a web user interface and is
controllable through the command line.
SpiderFoot also has passive and active techniques for collecting information.
SpiderFoot is a reconnaissance tool and sits squarely in
the recon space in the cyber kill chain.
The information gathered and analyzed during the reconnaissance phase
also helps through other phases of the kill chain.
The MITRE ATT&CK framework is a knowledge base of
attacker tactics and techniques.
You can see the high‑level tactics listed here.
Now, quick aside.
We care about the ATT&CK framework because in a true Red Team engagement,
we are emulating the tactics and techniques of a specific threat actor.
ATT&CK defines the whole range of reported adversarial methods.
It also lists the known and suspected threat groups,
along with their observed techniques.
When a client asks for emulation of a specific threat actor,
MITRE ATT&CK is the primary source to define the techniques needed.
Okay, back to SpiderFoot.
SpiderFoot is used for reconnaissance,
which is the first ATT&CK framework tactic,
and it covers three relevant techniques.
T1589 Gather Victim Identity Information,
T1593 Search Open Websites/Domains, and T1596 Search Open Technical Databases.
For T1589,
the data collected by SpiderFoot corresponds to the following sub‑techniques,
.001 Credentials, .002 Email Addresses, and .003 Employee Names.
For T1593,
SpiderFoot and its modules collect information that specifically
corresponds to the following sub‑techniques,
.001 Social Media, and .002 Search Engines.
For T1596, SpiderFoot utilizes a variety of databases to gather information.
These include the following sub‑techniques,
.001 DNS/Passive DNS, .002 WHOIS, .003 Digital Certificates,
and .005 Scan Databases.
Let's take a quick look at the overall architecture for SpiderFoot.
Let's say you're a Red Team operator and you need to
conduct reconnaissance against a target.
You're not sure what information you need yet,
but you're interested in gathering as much information as you can.
You have a starting point, though, the company's website.
So you fire up SpiderFoot to help gather more information.
You use the web interface to configure and launch the scan and maybe
use the command line to check on the status of the scan or to launch
a more targeted module‑based scan.
SpiderFoot uses its extensive set of modules to actively pull
and analyze information from the website.
The relevant data is stored in a database.
Unique data elements that are discovered are passed to other
modules that are interested in those elements.
For example, while spidering the website,
the module informs other modules as it finds internal and external URLs,
web content, and web server headers in response codes.
Those findings will trigger other modules that know how to collect
more information using those unique data elements.
Basically,
SpiderFoot starts small and expands its search as information is collected.
SpiderFoot can take an active recon approach and directly
probe the target's network and services.
It can also take a passive approach and use other online information sources
which have previously collected information about the target,
and it will query those sources.
Some information sources limit access to their collections
of data through an authenticated API.
You can request or purchase access from those providers and
configure the API keys in SpiderFoot,
which will expand the amount of information SpiderFoot can utilize.
As you use SpiderFoot,
you will find some very interesting and enticing
information about organizations, people,
and systems on the internet.
Just remember that SpiderFoot was created to protect organizations.
Our use of this information is to further that goal.
Viewing this information is not illegal; acting on that
information without permission can be.
As with any engagement,
be sure you have the permission of the people and organizations involved.