[MUSIC PLAYING]
So this is the Hacker House Hands-On Hacking Open Source
Intelligence Theory module.
So we're now going to run through the various theoretical
elements that are required to understand what we mean
by open source intelligence or open source intelligence
gathering.
This is a largely passive activity,
which is conducted at the start of most assessments,
to identify valid targets or systems
or people of interest that can be used when we actually begin
our more active phase of attacking when we start to send
packets and probes to systems.
So what do we mean by open source intelligence?
Open source intelligence is intelligence
that's obtained from public data sources or open data set
sources.
So it is a largely passive attack
because if you were to say use a search engine like Google,
you search for information in a company,
you're not sending information directly to that company.
You're looking it up in an external resource.
You're looking it up in a third party location.
So it's quite likely that the target company
is unable to see you conducting this kind of research.
And in fact, it looks no different to research
being conducted by academics or scholars
or journalists who may be performing research
into companies or individuals.
So it's a way of gathering intelligence or gathering
information that might be useful to you as an attacker.
So by doing things like scraping content from company websites,
scraping information from personal websites,
using various search engines like Google, Baidu, MSN,
and Yahoo, it's important not to just restrict yourself to one,
as you may find additional content in others.
Doing things like looking at social media networks,
going through people's social media activity,
looking at things like LinkedIn and others,
and also doing things like IP and domain registrar details.
Browsing through various information
that you might find within there that could be of use.
There's also various leaked and hacked data dumps,
information which has been disclosed now
into the public domain, which you
can gather and review through and find information on.
So for each of our modules, you'll
come across a tools of the trade box, which
introduces a number of common tools
which you might want to use when conducting
these kinds of assessment activities.
So things like SHODAN, Maltego, goog-mail, theHarvester,
recon-ng, metagoofil, various whois and DNS utilities,
and APIs, indeed, from various public databases
that you may have signed up to.
So open source intelligence is the process of gathering data
that could be of use to an attacker.
And what kind of information is of use to an attacker?
Well, for instance, email addresses.
If you're searching through lots of internet searching,
and you're doing through Google, do some Google dorking,
you might find email addresses of people
who work at companies.
And that's usually a pretty good target
is to scan through search engines
to try and find as many email addresses or direct contacts
as you may possibly be able to access.
So as an example, there is a copy
of goog-mail being run against IBM just to identify
are there any exposed email addresses.
And all it's doing is conducting a very basic Google search
to identify and expose email addresses at the IBM domain.
And this is using a concept known as Google dorking.
Or these Google dorks are specialized search queries that
are fed into Google, are fed into different search engines
to identify different vulnerable science and data.
As an example, it could be something
as easy as inurl/etc/password and then the common lines that
seen in the Unix password file, the shadow files,
to basically expose information from these various
[? geckoffs, ?] entries in the password file.
So by searching for that entry, you
could potentially expose information of a system that
wasn't intended.
And there's a whole database of these different search queries
known as the Google hacking database, which
you can get from this exploit db.com website here.
So by simply downloading that, you'll
find various search queries that you can use against the target
to inadvertently expose information in search engines.
So as an example here, we've done a quick search
for inurl/etc/password.
And what you can see here is as an example
that the Higher Education Commission in Pakistan
was indeed vulnerable to some kind of file inclusion
or directory traversal vulnerability that
exposed its password file to anyone on the internet.
So you could actually misuse this function
to read files from the Higher Education Commission
in Pakistan, which I believe is now a patched now as we contact
system owners as we find these things.
However, you know, this kind of search query
is very important to understand that actually conducting
this kind of search is both lawful and completely fine.
However, if you were to click on that link
and actually try to open up that password file and access
that information, then you could be
construed as committing a computer offense here in the UK
because you have requested a remote computer do something
without authority.
And so whilst it's OK to identify
these kinds of vulnerabilities through Google dorking,
if you do not have permission from the system owner,
and you were then to click on these links
and actually try to exploit these problems and gain
access to data without authority,
you may be committing a criminal offense.
So it's really important to understand
when you're doing Google dorks, just against random targets,
not to then click on the links and start
exploiting those targets unless you have permission to do so.
Another thing you'll find from good old open source
intelligence is things like document metadata.
So if you scan the internet, and you scan your target company
website, you'll find they may publish information
like maybe they publish quarterly sales
figures or quarterly reports.
Or maybe they share information in PDF or Word document product
brochures, things like that.
And these documents can actually also leak some data
because the documents will contain some known as metadata.
They may contain emails.
They may contain usernames.
It might contain date information
from when it's been published.
And also things like EXIF data and thumbnails.
You'll find maybe images that had pictures taken on cameras.
And then they had the thumbnail stored
within that EXIF information.
And it might not have been stripped from the web page.
And you could actually maybe see the image
before it was edited for post-production website use.
And indeed, things like thumbnails and GPS coordinates
would be found in this kind of EXIF data.
So that's document metadata.
It's the kind of information you'll find just
by downloading documents, parsing
through those documents, and seeing what kind of information
may be within them.
And again, things that would be of use to us
as attackers are often usernames and email
addresses because these will help us
when we come to later trying to breach
into systems and networks.
As people began conducting a lot of different open source
intelligence tasks, running various scripts against search
engines or performing various tasks,
people began creating ways to unify that.
They created frameworks.
And one of the better frameworks is
this recon-ng or reconnaissance ng.
And it's an open source intelligence framework
that follows its input very similar to another popular open
source framework, which we'll look at.
And it's a module-based framework
that allows people to create workspaces and effectively run
little scripts.
And it then stores all the information
in a SQL like database.
And it can enumerate quite a lot of useful information for us.
And it's very similar to other frameworks,
like metasploit for the purposes of keeping the input unified so
that it reduces the amount of learning
that you might need to do.
But it does have a bit of a learning
curve for effective use.
It also requires a lot of manual API configuration and setup
to get the most out of it.
So recon-ng is a fabulous tool for conducting open source
intelligence.
And we'll look at how to use that and some of the ways
in which that works as well.
Here's an example of recon-ng being run.
We can see that it creates a workspace
by simply creating a workspace.
And then we can use different modules.
We then add some information to a table.
And then run it.
And we get the output of the module being run on the screen.
And then that information is stored
into a database where it will help us for reporting purposes
and reviews later.
So recon-ng, there's an example of it being used.
And alternative to recon-ng, or indeed a complement to it,
is Maltego Chlorine edition or Kali edition.
So Maltego is a commercial paid for product,
which is quite heavily used if you were
to go into an intelligence analyst type role,
or you were looking to do open source intelligence as part
of your daily job activities.
You may find that Maltego is a graph-driven tool that
allows you to perform relationship
mapping between different graphs and objects.
So you could list an email address,
and then list the domain that it's
linked to, and then the website that is linked to that domain.
And it allows you to visualize relationships
between different information that you could perhaps get
from open source intelligence.
And it fits a client server type model.
And it does have a free version in Kali Linux.
But because this is a paid for tool,
we try to avoid showing too many commercial products
in our trainings.
And we try to focus solely on kind of open source tooling.
However, if you do open source intelligence as a day-to-day,
and you become an analyst, then you
may wish to use Maltego instead of things
like recon-ng, as it has a maybe better use cases for people
who do this every day.
So Maltego, there is a free version in Kali.
But again, you know, we can look at recon-ng and some others.
Another very useful website is a website offered
by a guy named Troy Hunt.
And Troy Hunt offers a website that
can be used to determine whether or not
your email address has been leaked in any publicly
known data leaks.
And what's also equally useful for us
is that we can use it to identify if one of our targets
has already had their information leaked
in a publicly exposed database.
So what we can do is then check through.
If we use the API, we search through all the emails
we know at our company.
Find out if any of them have been
exposed with password data.
We could then check through those leaked data dumps
to find any of that sensitive data
to see if any of those passwords are in there.
And passwords are a little bit like a hacker's best friend.
You know?
We can copy them.
We can reuse them.
They're a little bit like keys for a network.
So we really want to get as many passwords as we can.
So because haveibeenpwned offers this API interface,
there is some rate limiting on there.
But what we can do is we can actually
query this API pragmatically and identify
if there's any data leaks out there that
might be of use to us when we're targeting a particular company.
And some of the most common data leaks
are things like Myspace, Dropbox, and LinkedIn
because there's been so much information
leaked out there that you'll probably find at least one
of your targets in one of these data sets.
And the idea is, of course, that when
people choose passwords for websites they sometimes
pick the same password elsewhere and reuse it.
So we want to get these passwords from public data
stores if we can.
Another excellent utility is a search engine
called SHODAN, which SHODAN is a search engine for IP
connected devices.
It provides you with historic and on-demand scanning
of IP addresses and has a database of various hosts
and open services historic in nature.
And it is continually scanning the internet
to identify different systems and purposes.
And there's a command line tool, an API.
And also every kind of Black Friday,
they have a discount for it.
And you can use SHODAN for free, but also as well they
do a paid service which is well worth it.
And if you hang around on the Black Friday,
you'll often get Black Friday deals.
And you can get some scanning credits
for using this service for cheaper.
And that's a very, very good tool.
And I would definitely recommend that SHODAN
is one you want to make use of.
It's got various plugins for browser extensions.
And what it will do is it will collect data
from any of these open services, so things like screenshots,
various other things.
And then it will store that information
and make it accessible to you.
So there's a free version, but also a paid version of SHODAN.
So that brings us to the end of our theory for the Hands-On
Hacking Open.
Source Intelligence module.
And what we would do now is if you are in the classroom
environment, we would ask you to conduct searches
on the person next to you using some of those tools.
We're going to do some demonstrations of how
all of these tools work now in a practical demonstration.
But we would get you to conduct some searches.
So if you want to do some searches,
you could use some searches on me.
I'm HackerFantastic.
And you can have a look through me on the internet,
if you'd like.
Or alternatively, if you have a cool
handle or you know somebody else who might do perhaps
have a look at information.
If you don't want to look at information on one
of your classmates or your classmate
feels like you don't want them to,
then it may well be best to just look up
myself or your own information.
And again, what we want you to do in this
practical really is to be able to find a vulnerable host
with a tool like SHODAN, learn some information,
and gather some email addresses and gather
some data that's of value.
So as an attacker, when I'm doing open source intelligence,
I'm searching a lot for email addresses, passwords,
software type, software version information, things
that are going to be beneficial to me
when I begin targeting the company in a more active way
and more active probes.
So I'll now finish the theory and move on to the practical.