Well you've already seen the amount of data that Suricata can generate.
That's not all.
It can also extract files as they flow through your network.
Let's now take a look at how this is done.
While inspecting streams in your network,
Suricata can also identify file information and extract files
for storage and later processing or analysis.
File extraction works on top of the protocol parsers,
which means that file extraction is dependent on a
protocol parser being implemented.
For example, you can extract files in the HTTP protocol,
since there is an HTTP protocol parser.
The parsers in turn work on top of the stream reassembly engine,
which suffice it to say,
is getting deeper into how Suricata tracks network traffic.
However,
there are some configuration changes that need to be made to
ensure that file extraction works as expected,
and we'll cover that in this video and the following demonstration.
The primary file extraction and storage options are under the
file‑store section in the configuration file.
Which files are actually stored is controlled by rules.
So, if you're considering file extraction,
you'll need to create some custom rules to help.
Before we get much further into file extraction,
it's worth noting that Suricata also generates file info events.
You can look for these event types at EveBox under Events ‑> FileInfo,
and then select the record to view more information,
such as the content type, size and different hash values.
You can also view these events directly in the EVE.JSON file.
There are some caveats though, and this is where we need to
discuss making changes to the configuration file.
The filemagic, or just magic,
tells us the identified file type using the libmagic library.
The content type will not always be recognizable.
Perhaps it's compressed, obfuscated or encrypted.
File extensions may not always match either.
They are, after all, arbitrary to the content they represent.
So, getting a file hash can be a good IOC to research a file on,
but, there is a problem.
The state of the file info event is truncated.
So, what does this actually mean?
Well, Suricata only looks so far into the response stream,
and with files, this can often exceed the default value that Suricata will look.
This results in a truncated state, which results in an inaccurate file hash.
Only part of the content was read after all.
To resolve this, we need to make a few changes to the configuration file.
First, we'll enable file‑store.
This tells Suricata to store files.
Then, we'll modify the response‑body‑limit to 0 under the libhtp section.
This tells Suricata to read to the end of the file, regardless of length.
There may be a performance impact,
so you'll want to test out file extraction and the impact it has
before fully rolling it out into your network.
Now that the configuration changes are made,
we need to create rules, so that Suricata knows what files you want to extract.
File info events will still be generated and are not dependent on
file‑store settings or the rules we are about to create.
The first keyword we need is filestore.
This tells Suricata, while parsing the rule,
that it will utilize file storage capabilities.
The first rule is an example of storing all files found in the HTTP protocol.
The next keyword is fileext, or file extension.
This is used with the filestore keyword,
and tells Suricata just to match on file extension.
In the second rule example,
Suricata would extract any files ending with a pdf
extension over the HTTP protocol.
The final example uses the filemagic keyword,
which in turn utilizes the libmagic result from the file content to
match against the value you use in your rule. Knowing what value to
use with your libmagic keyword requires that you know the possible
results from the file types you want to extract from libmagic, and
you can experiment with this by using the file utility from most
Linux terminals.
Let's recap.
If a file is read to completion,
it will have a closed state in the corresponding file info event.
The file hash and other information will now be accurate,
and if there is a file extraction rule,
there will be a file located at var/log/suricata/filestore.
We'll now get hands on in our training VM and configure
Suricata to perform file extraction,
write a rule to extract PE files and discuss where these files are stored.