[COMPUTER NOISES]
[MUSIC PLAYING]
Welcome to the hands-on hacking open source
intelligence practical module.
So I'm now going to demonstrate some of the techniques
that we've just discussed and show you
how to conduct open source intelligence reconnaissance
on a particular target domain.
So if we go back to some of our tools of the trade,
we'll now start by downloading some of the scripts.
And throughout this course, we'll
often reference some files from a directory
at http://static.hacker.house/stuff
These contain various scripts, exploits,
and utilities, which you might be
using throughout this course.
So I'm first going to download the script goog-mail.py,
and this is a really simplistic Python script,
which has been kicking around for quite some time.
I don't even believe it's credited
with an author in fact.
And this little script will effectively just
go into Google, and it does a search for the domain
name and the common @ character to see whether it can identify
any email addresses by then parsing through any of those
returned responses and seeing whether it can actually
identify email addresses.
So we'll run python goog-mail against uk.ibm.com,
and that will basically just do a search in Google,
find everything that looks a bit like an email address
for ibm.com, and then it will output any of those users
back to us on the screen.
And we can see here, now we've got
a number of email addresses, which
might be relevant for people doing this kind of querying.
If you were targeting IBM, you can
see here-- if I was to do emails, for instance,
and then do edit paste, we could see that there's
various emails in there.
And in fact, we got a total of 14 email addresses
for people who perhaps work at IBM.
And one of the things that you can find about here
is if you actually do this search against an organization,
you'll find that these email addresses are the ones that
commonly targeted by spam and malware,
as they're often automatically targeted
by bots who do that same activity we just did there.
So using Google, we can do things like a search
and identify email addresses, starting with things
like the company name.
And obviously, we can also use the Google hacking database
to actually try and find any kind of known vulnerabilities
that might exist in that application.
So if I just do google-hacking-database.
And we can see here that this is the Expoit DB Google hacking
database.
And we can see, there's various search queries
that we could use that can be used for finding
systems and vulnerabilities.
So as an example, here, if we were to do a query
for inurl:"q=user/password", this would try to find Drupal
instances on the internet.
And there's been a number of vulnerabilities in Drupal.
And we'll explore some of those.
But if we were to simply click inurl query and do this query,
we could find whether or not these were Drupal instances.
And just by doing that search, we
can see, here, there's a number of Drupal servers
that are online that we could perhaps begin looking at.
So these are examples of Google dorks, which can
be used to expose information.
And when you're doing open source intelligence
queries, what you want to expose is
software type, software version information, IP addresses,
domain names, email addresses, any user names, and content
that might be considered sensitive.
And in fact, if you've done open source intelligence thoroughly,
you may find that you can actually breach systems just
from this phase of your activity.
But we'll go into some more probes in a second.
So here, we can see, is another Netgear probe for instance.
So we could actually search in Google for this common string
and find various Netgear devices.
You could then search based on the company domain
name as well.
So using Google dorks, we can find information.
And as an example, if we were to do @uk.ibm.com,
that's that same query that goog-mail
was doing to try and find those email addresses.
And we can see there is one that was listed in that list.
So that's using search engines, using Google dorks,
using references.
And I don't want you to limit yourself to oh,
I'll just look in Google-- do some Google dorks.
You should look in things like Baidu.
Look at things like Yahoo.
And in fact, some of the best open source intelligence
I've ever gathered has, in fact, come
from very disparate sources.
Things like your local reference library--
you may find blueprints of buildings.
You may find contract information.
Going and looking at things like wireless regulators
and Ofcom and all the regulatory bodies.
You may find information in their data sets
that are relevant to the target organization you're probing.
So don't just limit yourself to Google.
Consider it a kind of academic exercise to research everything
you can find in public areas.
And indeed, reference libraries are great places
to look as well.
So just coming back to the output
that we have here on the terminal.
We've got a couple of terminals set up
that we're going to use in the background here.
Now, what I'm going to do is--
obviously if you were to keep running scripts like this
and having lots of text files, it
would get very cluttered and a bit messy.
So what we tend to use is a framework.
And people began using different frameworks or tools
to consolidate their information from open source intelligence
queries.
As they're pulling data from different places,
they try to store it centralized in a unified format,
so that we could get it from it.
So Recon-ng is an excellent framework,
which has been designed in a modular-like way
to take input in a similar way to things like Metasploit.
So that you have less of a transition learning
two new frameworks, you can take what you've learned on one
and use it in the other.
So it follows a similar framework design.
So as we can see, now, when we run this tool,
we get this big wall of red.
And the big wall of red is because you
need to set up various API keys to make the most of this tool.
And if you're going to do a threat
intelligence or analytical type roles,
then you would probably want to configure more of those modules
to really get the most out of this tool.
But for the moment, we'll show you
how it works on a basic level.
So we've got Recon-ng, and here at the bottom,
we can see there's a prompt.
It's giving us some statistical information.
If you either type question mark or help,
you'll get a list of all the different commands
that Recon-ng supports.
And one of the first commands we'll do is workspaces.
And if you simply type the command,
it will give you an example of what it's expecting.
So if we were to do workspaces add,
it then says you need to give it a name.
So if I was to do workspaces add hackerhouse,
you can now see that we've had another wall of red.
And then at the bottom here, the default
has changed to a hackerhouse prompt.
And that means that any of our information that we gather now
will be stored inside of that workspace.
And I'll show you how that works under the hood.
But if you were to then do show modules,
you can then see that we've got various reconnaissance
modules-- various discovery modules--
that we can then use to probe open source data
sets and find information.
So we're going to use the profiler module
recon/profiles-profiles/profiler and see what information we can
learn from that.
If we use that module and type show info,
we'll get a description and the name of the module, which is
OSINT HUMINT Profile Collector.
And what that means is open source intelligence,
human intelligence, profile collector.
And the description says that it will
take each user name from the profiles table
and search a variety of websites for those user names.
And you can basically see, here, that this source
input is a SQLite query.
Don't worry if you don't know SQL or databases.
We're going to cover those in a later module in more detail.
But for the time being, just know
that it's using a SQL query in a database.
And it needs to get information from a table that's
called profiles.
And if you do show profiles, you can
see that there's no information in that table at the moment.
And if we were to do add profiles and then add
a user name--
if you want to search your own username,
or indeed search mine, or one of your friends--
you can simply do add profiles and then give the user name.
But what we need to do is-- if we
try to do that by itself, you'll see it say columns and values
length mismatch.
And that's because we need to add
in the correct number of columns,
using a tilde separator.
One, two, three, four.
And for this table, it needs for.
And I'll show you why.
If you do show profiles, you can see
that adding profiles, the user name, and the number of tools,
we have the user name, hackerfantastic.
And then we have these four empty columns.
And this end column here-- the fifth column--
which is actually filled in by the framework itself.
You can see it says user defined.
So we need to ensure that when we're adding information
into this table that we add the right number of columns
as well.
And I'll show you how we can explore that
in more detail in a moment.
So once we've got that set up, we have the inputs
into our source.
We can see that the source is default.
So it's going to take every user name from the profiles table
and query it in these various websites.
And so by simply running this, what will happen is it will go
to all of these different websites and identify whether
this user has an account on those websites,
see whether that username is valid for those websites,
and pulls out information-- you can see--
about this individual.
And if we were to now do show profiles with our 17
new entries, we can see all of the various accounts
that this user might have on things like facebook.com,
video, Slashdot, Xbox.
And indeed, just by querying for these common user names
that people might use, you can learn a lot about a person--
right down to things like tastes in pornography.
Or recreational drug use can even
be found on these kinds of websites.
So it's very important to understand that you can use
open source intelligence to gather intel-- gather data--
on a target and learn things about them.
So having a list of profiles on a website is kind of useful.
But we'll have another look now at how this actually
works under the hood.
And if I was to do show modules now.
And we'll have a look at some other modules in a second.
But so if I go to my other terminal here,
what you can see is if I list the ~/.recon-ng directory,
you can actually see that this Recon-ng framework has a folder
called workspaces.
And inside that workspaces folder
is the hackerhouse directory.
So every time you create a workspace,
you actually just create a directory
on your desk that then contains this data.db file.
And that data.db file is, in fact,
a SQLite database that takes the information from the scripts
and is used as input into the scripts.
And that's where it's centralizing
all of the information in that framework.
And you can actually just use something like SQLite Browser
to open that file and visually represent it on the screen
so that you can explore its contents and structure more
easily.
So for an example here, we can see if we were to look
at the profiles table, we can see how many columns were
needed and what those inputs were-- user name, resource,
URL--
and the purposes of them.
Likewise, we can also use this tool to explore the tables
and see the information that's contained within.
So we can see, here, you have profiles
and all of the different accounts that we found.
And subsequently, you can then export that and use it
for report purposes or similar.
So just as an example, maybe, let's use one more.
We could do something like a query of a Google hacking
database for instance.
We could download some Google hacking databases.
We could look for information in public data leaks.
We could try and do things like whois points of contact.
So we'll have a quick look and see if we
find phone numbers in whois.
You could do things like look at YouTube postings.
How many Twitter users and things they may have.
And we'll have a quick look and see
if we can get a search to work with something like the PGP
search.
Now, often this module doesn't fully respond in time.
So we'll just see if this will actually work.
So this one says show info.
And then we can do select distinct domain from domains.
If I do show domains, I will do add domains.
And I'll add my hackerhouse.com just as an example.
And if we were to run that, what should
happen is it will go to the PGP query--
PGP key servers-- and try to query
it to see if any of those users have shared a public PGP key.
It is a little temperamental, this module.
And it doesn't always bring back data.
But that's just an example of Recon-ng
and how we can make use of Recon-ng
and its various modules.
There are also various reporting modules
that you can use to get your information out of there
and generate HTML-formatted reports.
So you can see the output of any of these tools
that you might actually query.
So that's an example of using Recon-ng, querying information
and using it for things like Google dorks.
There are various other tools as well that you could
use-- things like theharvester.
And the harvester will do things like-- if I
was to do -d uk.ibm.com -b PGP for instance,
theharvester can do various other data sources
and gathering as well.
So it can gather information from Google, Bing, PGP,
Twitter, et cetera and try to see
if it can find any known email addresses
or information that it might be able to get
from that kind of open source intelligence.
And in fact, if we use theharvester
to query for PGP keys and the key server,
we can actually see that it's pulled back a rather large list
of email addresses.
In fact, many more email addresses
than we got from simply Google dorking of users
who have shared a PGP public key on the key server.
So I'll now do emails again.
And I'll add my emails in, knowing
that we had 14 previously.
And just by doing a search in the PGP records,
we can now see we've got close to 150 email addresses
of people that work are IBM.
That's still quite a small number
of people working at IBM.
But it's certainly a very useful collection of email addresses
that we might use if we were targeting IBM.
So we would use these in things like spear phishing attacks
or other.
So in total, we've got 148 email addresses.
So that's kind of useful.
And what we can see from this is-- actually,
a very common scheme arises, where
people use their first name, underscore surname.
And also first name, full stop, surname.
So we can actually start to see the schema of how
people construct their email addresses.
And we could actually do a grep and just see
how many of those use an underscore.
And about 42 are using an underscore.
And then if we were to do a grep--
most of them have of a dot or similar.
So we could try trying to construct--
if we knew the name of somebody who worked at IBM--
we could even try to guess what their email address is
by giving it first name, last name, and a full stop
or an underscore.
So that's kind of a useful way of gathering intelligence.
So theharvester and Recon-ng do this Google dorking for you
and allow you to gather information.
Another tool that you could have used for gathering metadata
by parsing through search results
and downloading all the documents is Metagoofil.
So if we were to do metagoofil -d uk.ibm.com -t,
and we'll download just the documents
PDF and word documents.
And we'll store those to a file called
ibmfiles and results.html.
And all this will do is it will do a Google
search for any kind of common URLs
that it might find that have documents inside of them.
And then it will download those documents,
try to parse through it, see if it
can find any useful data-- like users,
software, packets, and email information.
So Metagoofil is a tool you can use
for automatically extracting that metadata from any
of those documents.
So Metagoofil is a useful tool to complement things
like Recon-ng and theharvester.
And I'm going to show you Maltego now.
So Cali comes with a free version of Maltego.
It's a commercial product, which does
have some limited free usage.
And I'm just going to show it to you as an example of how
to make use of this tool.
Because it does visually represent information
that it pulls from open source queries in a graph.
So here you've got various options.
And if I was to do OK, File, New.
So I've launched Maltego.
You'll need to create an account to use this software.
And if I was to, then, simply start a new graph,
I could then make use of some of these machines,
to automate some of the processes
that I might want to do.
So let's say Company Stalker.
And if I was to say ibm.com because I like big blue,
and I like some of their machines.
So I'm going to have a little bit of a company
stalk against them.
And you can see, here, now, it warns you
that because this is a free, community edition,
your results will be limited.
So simply running that transform, now, you can see,
it's put a little icon for ibm.com, which is a domain name
and given you some details there.
And it's now running these transforms.
And this transform-- so it says server side transform.
So they actually get run by third party servers.
So you would be sending information to a third party.
That's something to be aware of if you have,
perhaps, non-disclosure agreements
or there's some limitations on the testing you are doing.
So instead of ibm.com, in fact, what we'll do
is we'll use a well-known cyber security company.
So if we were to just simply put in nccgroup.com,
and we can have a look at this cyber security company
and see if any information comes back from various Google
dorks and other data.
So in fact, you can see here, on this query,
we've now got a list of email addresses
of people that work on there.
And it will allow us to select those.
And we can see it's now got a graph--
a visual representation of how that information should
be displayed to the user.
And if you go up here, you can see
it's mapped the relationship between this domain
and these users.
And it's limited it to 12 results.
It hasn't given us more than that because it's a free trial
software.
So that's an example of Maltego.
And then simply right clicking on here
would give us additional options--
things that we could search through--
and transforms we could run on top of.
And here, you can see, we could find out what technologies
and various others.
So Maltego is a graphical tool for conducting
this kind of open source intelligence queries.
Again, you can get the same kind of functionality
from things like Recon-ng.
And I do like to focus on open source tools
as opposed to commercialized tools
to try to get the most out of the sessions.
So what we've got here is--
OK-- open source intelligence.
We've done the Recon-ng.
We've done some Metagoofil.
Other tools we could use are things like whois.
We could query a domain name and look up
in things like the ripe information.
So let's have a look for whois hacker.house.
We can see, it will come back, and it
will give us some information about who owns that domain.
What information is stored in the domain's whois record?
And also, you could take the IP address
and have a lookup at the IP address as well
and see what information is listed in the IP databases.
And here, you can see that hackerhouse.hacker and some IP
addresses and addresses and phone numbers
are listed in there.
So finding things like phone numbers
can be useful to then begin war dialing
against the organization.
It's kind of an archaic practice now,
but it sometimes still bears fruit.
And indeed, being able to find individuals who are,
perhaps, prominent within the technology area
of that company-- who work with ICT and similar systems.
So querying things like public databases
as well is a really good way of gathering information.
So we've done some Google dorking.
We've looked at some information.
We're now going to look at data leaks and various information
we can gather from them.
Again, there are modules to do this
in other parts of the website, but I'll just show it.
There are modules through this in Recon-ng.
But I will show you-- if I go to the website,
haveibeenpwned.com, what we will see, here,
is you can actually enter in an email address.
So let's just take one from our IBM list.
If we go cat emails.txt, let's just take John.
A very nondescript john@uk.ibm.com--
and we can actually learn, has this information
been leaked previously?
And in fact, we can see that in May 2016,
email addresses and passwords from LinkedIn were exposed
and that this user's password may have been exposed
in that database as well.
Same with all the online spammer bots.
There was some other information that may have been exposed.
So what we would want to do, now,
is maybe pragmatically query to find any of these users who
might be in a data breach.
And then search that data breach to see
if there's any information of use in there.
So if I was to do, for instance, for email in cat.
If you don't know any programming,
then this is a very basic for loop.
And what we can do is do for emails cat do echo $email
.done.
So I'll explain this loop now.
You've got for email in cat emails.txt.
And what that does is it will take every entry-- every line
in the emails.txt file and add it into the variable called
email.
And then for every line in that text file,
it will do the following command until it is completed.
So every line in that text file, it will do this command.
And all the command I've said is to print out that email address
to the screen.
So we can see, it will print out the email address
to the screen.
And what I'm going to do now is I'm
going to change that command.
And I'm going to get it to run curl, which will send a web
request to a server for me.
And I will supply it with haveibeenpwned.c
om/api/v2/breachedaccounts/ and then the email address.
And the API is rate limited to stop people
doing nefarious things like this.
So I'll, perhaps, give it three seconds between each query.
And what you can see, now, is by querying each of those email
addresses, the server will actually return and tell us
whether or not that email address has
been stored in a data breach.
So if I was to do echo $email, we
could see, for each of the email addresses,
whether or not information has been exposed about that user
and what data set it's in.
You would be able to identify the user
name, the email address, and potentially passwords
for some of these users.
And as I pointed out before, that LinkedIn data set
is usually quite a good one to browse through.
Because you'll find a lot of users have reused passwords.
They may have reused the password from LinkedIn,
be unaware that their account was compromised,
and you might be able to reuse that password somewhere.
And in fact, I've had organizations,
which have been breached, because they've
had things like press@company.
And then they've had their LinkedIn password,
and it's been something like pressoffice.
And I've then been able to access the company by logging
in as press with a password of pressoffice.
So things like that, you'll often
find that by searching through these public data sets,
you might find accounts or dormant activity
at a company that was, perhaps, created on things like LinkedIn
for some reason.
Or the email address was reused.
So you can search through this information.
And that will be of use to you should
you want to then begin active attacks against the company.
So finding things like password data is also very useful.
Now another tool you could use for gathering information
on contacts is a tool called LinkedInt.
And this is a tool that was published online
by some guys and subsequently margin modified
a little bit by a chap so that it could actually
search through LinkedIn profiles and allow
you to do various queries.
And you would need to, basically, set this up
with a LinkedIn account.
You need to give it a LinkedIn username and password.
You can download this tool from GitHub.
And by simply searching through this LinkedInt tool,
it says here you can enter your search keywords.
So I will do a search keyword for International Business
Machines, perhaps.
Use quotes, so I will do quotes for people at International
Business Machines or IBM.
And enter the output, which will be IBM, filtered by company.
And leave blank-- and then the email domain suffix--
what could then be done is it can actually
try to generate email addresses based
on the people's first and last name and output
it into a spreadsheet for you.
So you can actually use this LinkedInt tool
to search through LinkedIn in, search through all the people
who publicize that they work at, perhaps, IBM.
And then get a big list of employees
who are published on LinkedIn and then generate
those email addresses.
And you'll find that by doing this,
you'll get significantly more email addresses and users
to target than you would do if you just solely
relied on information from Google or Baidu or other search
engines.
So searching through social media platform,
searching through companies like LinkedIn--
searching through all the employees
who are sharing information on there,
will give you quite a lot of data.
And in fact, for IBM, just by doing that one set of queries,
we found 973 potential employees that we now
have the name, job title, and email address for that
can be then browsed through at our leisure.
And in fact, it does output this in a kind
of spreadsheet kind of way.
And if I was to just simply cat it,
you can see here, as an example--
we've got a number of Mike and his name and his job title.
And then also this email address, which was generated.
And you can use tools to actually try to predict--
once you know that predictive naming pattern,
you can generate these email addresses.
And find all of these various individuals
and try to contact them if you were sending
them malware or similar.
So that's a use of more open source intelligence gathering--
how to gather more employee information how to learn more
about a target company.
Likewise, digging into haveibeenpwned,
find the leaked data sets, take any of those user names
and passwords.
And that can also be very useful to you.
And one last tool, which we're going to look at,
is the Shodon tool.
So if I was to go down to here and type in Shodon.io,
so this is the search engine of internet of things.
And as I said, it's a paid-for service
that you can, essentially, get access to.
I recommend getting it on the Black Friday specials
that they often do, as they often
give you really good rates for their service.
And this utility is effectively a search engine
for different kinds of computers.
You could search for Apache, find
all of the Apache instances that it's found over the time.
And we're going to see, it's got 25 million Apache instances.
For each one of these, you could click on it,
and it will show you the home page of that server.
If you were to then click on Details,
you'd get a overview of a map that shows you
what kind of weather the IP address is
believed to be located.
And things like services and any information
that it gathered-- you know MySQL information, web service
information.
So as an example, let's just use Shodon to find something
like gasoline.
If we were to do a search for gasoline, what we would find
is that, here, we can find a number of computers
that are connected to the internet that
are used for dispensing gasoline from a gasoline pump.
These are petrol pumps, effectively.
So here we can see that in Saudi Arabia--
if we were to click on here, we can actually see,
here is a petrol pump located in Saudi Arabia.
We can see the various services that it offers.
And indeed, we can actually see how much gasoline and diesel
is in the current tank.
And we can see it's on King Salman
Road, Riyadh in Saudi Arabia.
So you can learn information using Shodon to query
through various search engines.
And you can look at things like industrial systems--
things that really shouldn't be connected to the internet.
Here's another example of a petrol station connected
in LaRose.
And you can see, here, is a port that you can actually query--
again the diesel and gasoline and the
amounts available in the tanks.
Another useful thing with Shodon is
that there's a command line tool as well, which you can use.
So you can actually just type Shodon.
And once you register for Shodon and get an API key,
you can actually do some of these queries
on the command line.
So if I was to do search port 1,001 for instance,
we could actually then do a search
and try to find all of the open service
ports that match this 1,001.
And we would then get that in a scriptable import.
So we can actually then parse through it.
And you can see, here, we could output
that in various different file formats.
And we can see all these different pumps
and things that are connected for these ubiquity network
devices--
these modems that are giving information about petrol data.
So various bits of information that you can learn there.
So Shodon is kind of like a search engine for computers.
We can search through it, and we can
find different kinds of computers
that we're interested in.
So we could search for, indeed, our target.
So let's say-- we could go into showdown
and do a search for ibm.com.
And we could find all of the systems
that Shodon knows about that might
be associated with ibm.com.
And this also means that we're not sending any information
to our target.
We're not sending any packets.
We're not sending any data.
So it's purely passive.
And there's very little defense that an organization
can do against this, other than itself
being aware of what information it shares on the internet
and shutting down, perhaps, in inadvertent exposure.
As an example, perhaps, if the CIA were routinely
searching through their contractors GitHubs,
maybe they would have noticed when somebody
had leaked internal tools onto their public GitHub
several years ago.
But subsequently, these kinds of searches--
these kinds of constant looking at your perimeter,
looking at the information that's out there in public data
sets, making sure you're aware of what your adversary is aware
of--
because people often leak information that they may not
be fully aware of.
So I believe that that brings us to the end of our open source
intelligence modules practical.
The exercise for you to do for this module
is that I want you to basically conduct searches either
on myself.
If you're in a classroom environment, perhaps the person
next to you, or indeed, if you have a cool name
that you use on the internet like ultralaser or drdoom, then
maybe you should search through public records
and see what you can find with tools like Recon-ng.
Try to use a tool like theharvester.
Remember if you do do some Google dorking, not
to access any of those links.
When you see a vulnerable server,
just be like, OK, that's a vulnerable server.
But remember not to access that vulnerable server.
Because you may breach some legislation.
So it's OK to identify the vulnerabilities using it.
So try that.
Try to use Google dorking.
I want you to try to find one vulnerable system
on the internet.
Try to find just some information,
perhaps, you shouldn't have access to-- a password file
or something like that.
And then that's basically the exercise
that I want you to do-- is to follow through,
try some of those tools out that we've just used.
OK, so that concludes the practical module,
and I look forward to teaching you the next one.