Logstash, if you're working with the ELK Stack, could be your best friend.
As I've mentioned previously,
it's what stashes and parses the log information that we're ingesting.
It is its own tool inside of the suite,
and the configurations can get pretty robust.
Let's take a look at this before we get into our use case of using
Logstash to enrich the Zeek logs for Arkime.
Logstash has the ability to perform several actions on your data.
It can ingest multiple input formats and sources.
It can parse and transform the data as it's being ingested or on the fly.
It also has a selection of outputs that allow you to be flexible
with where you're stashing or sending your data.
It also has the ability to use extensions so that we can get or build
Logstash plugins to help us add to the capabilities.
So what are we doing with Logstash, exactly?
Well,
right now we only have a basic configuration of the
tools set up to ingest our Zeek logs.
The configuration is here.
It's too much to display on the screen,
but this is pretty much the default setup to use with Logstash.
If you have the ELK Stack currently running in your environment,
you can pull up the logstash.yml file and just validate that there's not
too many settings to tweak in this one specifically.
We're just defining our path variables in it for our instance.
Defaults use the localhost address and the standard ports on this.
We'll look deeper into more advanced configurations
when we hop into the lab in a minute.
But we do have a sample config for the Metricbeat and
Apache logs already set up here.
This configuration has inputs, filters, and outputs.
We saw it a little bit in the previous demo when we added our
input filters and outputs for the Zeek logs.
This is a little bit more advanced using the grok filter,
which helps us parse the logs better, and we also have our output going
directly to Elasticsearch. When we hop into the demo we'll see a lot more of
these pipeline configurations that are much more advanced.
This one is pretty simple, and as you saw in the demo, with our Zeek input,
we're using the JSON filter to help us with the data.
Let's explore how we can parse and enrich the data for use with
Arkime. And we need to figure out what exactly we're going to be
using for this coming demo as our platform.
More to follow on that.