The first thing that we want to do in this lab is to ensure that we are able
to ingest our Zeek logs with ELK and view them in Kibana.
This is necessary so that afterwards we can jump into the configuration for
parsing and modifying the logs for mapping with Arkime.
Let's hop into the lab and get it done.
All right, and here we are in the lab,
and what I have pulled up first is just my Kibana dashboard,
so this is just a basic installation of ELK.
So I have Elasticsearch and Logstash, Kibana running on my 30.8 platform,
and then we're going to install Filebeat on our Zeek platform and
get it configured so that we can ship logs from that over to
Logstash and view them in this Kibana instance.
And so as you can see,
we have just Metricbeat for this system itself sending
data to the ELK Stack right now.
Metricbeat is just helping us with a lot of the
underlying OS information and stuff like that.
If we go over to this Logstash index pattern that I
already pre‑created in Kibana, there's nothing here,
so we don't have any pipelines set up for Logstash coming from Zeek,
which is what we want.
So the first thing that we need to do is jump over to our Zeek instance,
and what we need to do first is go ahead and install Filebeat.
Now, depending on what Linux platform you're using, or Linux distro you're using,
the setup process for any of these Elasticsearch or ELK Stack products or
tools could be different than what I'm showing you here.
I already preloaded the repository in here so it just can
automatically pull the latest version for me.
Now that it's installed, in order to go ahead and configure it,
we're going to just go to the YAML file,
and it automatically installs in the etc/filebeat folder.
And as you can see,
I have a couple files here and we're going to go for
the filebeat.yml file right here.
And it might help if I actually want sudo for that.
There we go.
All right, so this is the Filebeat configuration.
And when we're talking about this,
there's a couple things that we need to do in order to ensure that
we are able to ship our log off to Logstash.
So the first one is these inputs.
We have a couple different options.
We can do logs, we can do files, and so on and so forth.
And so I'm just going to go ahead and enable both of them.
Now this log one inside of Filebeat is soon to be deprecated because
it's going to just start tracking the file from now on.
In the path right here,
what we need to do next is go ahead and tell us what path
those log files are going to be on.
So these are our Zeek logs that we want Filebeat to be
tracking and sent over to Logstash for us.
So this is the directory that my logs are currently sitting out for Zeek,
and then we're just going to grab all of the logs because I
don't want to differentiate between them.
I just want everything shipped over that Zeek provides us right now.
And the file stream, this one up here,
is the other one that I was talking about that we
can utilize for these purposes,
and we're going to go ahead and enable that one as well
and change the path for here as well.
All right, now that that's done,
there's one more thing inside of Filebeat that we actually need to change,
and that's going to be where to send the data.
Since this is just a basic installation,
I'm not going to be changing too many other things here,
but what we need to do is go down over to the output
section and configure the Logstash output.
As you can see, Elasticsearch is already preconfigured as an output,
and since we're not sending straight to Elasticsearch we're
just going to go ahead and comment those out.
And then you see we already have a predefined
Logstash output section that we can use,
and so we're going to uncomment these and then define
where we're going to send the data.
So in this host section I'm going to go ahead and type in the IP
address of the virtual machine that were shipping the logs to
with the ELK Stack already installed, and in my setup I have it as port 5045.
All right, that's it.
Now we can go ahead and just write that, save it to file, and exit.
And then after we're done with the initial configuration for that,
we need to be able to set up Filebeat.
So the command to do that's going to be for you to go
over to the filebeat/bin directory.
And, again, this directory may vary depending on how you installed Filebeat.
And the command for that is going to be just setup ‑e.
And I am on it today with forgetting to do sudo.
And since we're not sending over to Elasticsearch,
we're sending over to Logstash instead,
we are going to just ignore these warnings for it.
So now Filebeat's all set up, and let me fix my misspelling.
All right, now Filebeat is enabled, so it'll start when the system starts.
Let's go ahead and start the actual Filebeat service.
And it looks like everything started just fine, and we're running, all right.
So now the next step is to hop over to our other box that
we have the ELK Stack installed on,
and we need to configure the pipeline for Logstash to receive the data.
So there's not too much configuration there.
So what we need to do is go ahead and configure just the pipeline.
And in the Logstash directory we have a couple of these different areas.
We have the pipelines.yml file, which is really not doing much for us.
It's just directing us to the conf.d folder with
the rest of the pipelines in it,
and we have the logstash.yml file which is just the
basic configuration for Logstash itself.
So what I'm going to do is, I'm going to hop into the conf.d folder,
and right now I only have that pipeline for Metricbeats that we will
see in the slides later on and we'll explain those.
So what I want to do is go ahead and nano a new file,
call it zeek_input.conf,
and then let's copy/paste our pipeline configuration in here.
Now this is a very basic configuration.
All we're doing is accepting the beats input,
we're using the JSON filter so we can make sure that it parses those
JSON messages and extracts the information from them,
and then we're telling it to output to Elasticsearch on this local box.
So I'm going to save that, exit it,
and then I'm going to go ahead and restart Logstash.
And since that configuration file that we just made for this
pipeline is already in the conf.d folder,
Logstash already knows to start using that and accepting it as an input,
so anything in that conf.d folder, it's going to use.
And now that that's done,
we can go ahead and refresh the dashboard and wait for the data to populate.
And all I'm doing is checking on my indices and looking at
the index patterns to make sure that we're receiving the
right fields and everything.
And hopping back over to my Discover dashboard in Kibana,
as you see, we have the Logstash index pattern selected here,
and we have some logs from Zeek just started coming in.
And if you look at them, we can see that Filebeat is the agent that sending them,
we can see that the host is globozeek,
and we can start diving into some of the information in here.
We can filter on the originating hosts or the ports and so on and so forth,
just like any other SYN.
And so now that we have the basic log shipping from
Zeek over to the ELK Stack completed, we can view our logs in here.
In the next couple clips we'll start talking about how we can enrich those logs,
how we can filter and parse them a little bit better and map
the information over to our Arkime's schema.