Ingesting and analyzing audit logs are not the only things we can do with Kafka.
In fact,
we're going to have some extra fun by detecting network
port scan attacks and triggering alerts in a similar
fashion as the audit.log scenario.
I'm going to get started by creating a file that will
store our network logs using touch.
The file must be placed in the same logs directory
so Kafka Connect can access it.
The network logs will have the following format.
I'm actually going to insert one in the network.log file so
Kafka Connect can publish one event right away.
The network logs will be created using a traffic sniffer like tcpdump,
but other network analyzers such as Wireshark,
may work just fine.
The format is rather standard and includes the time when the event occurred,
the source IP of the network package, its target or destination,
and some other details we can cover a bit later.
Back to the README file, we can deploy a new connector on Kafka Streams.
The pattern is rather similar as in the previous demo.
We're using a FileStreamSource connector that we're calling network‑logs.
This connector will check the network.log file for new appended rows,
and for each and every one of them,
it will create a new event that will be published in the network‑logs topic.
Now, this time, we're going to use another component called Kafka Streams.
Kafka Streams offers a similar set of functionalities as ksqlDB,
but it's using Java code instead.
You're more than free to check the source code inside
the network‑logs‑analyzer folder, but the logic sounds something like this.
A port scan attack is defined by a rapid succession of
connections on different ports from a single source.
So what the network‑logs‑analyzer does is checks if
in a time interval of 10 seconds,
a single source has tried to connect to multiple ports on the target.
If such a scenario occurs, the network‑logs‑analyzer will trigger an alert.
Now, looking at this Docker command,
you should take two factors into consideration.
First,
if the directory where you placed the docker‑compose file is called differently,
then the network name should be adjusted.
The same thing applies to the volume model.
We need to mount the secrets folder so that the absolute path
might also match what's relevant for you.
Just like ksqlDB, this application will run in the background,
so we won't be bothered by its logs.
With everything related to Kafka in place,
we can start analyzing the traffic and performing the port scan attack.
To analyze the network traffic, we're going to use tcpdump,
but we need a target first.
I'm actually going to attack the audit‑logger application.
To determine the target IP, we can use docker inspect command.
Typically, port scan attacks are performed against other machines,
but today I'm going to keep things simple and attack a Docker container.
The IP is 172.28.0.4, so I need to note it somewhere,
because we're going to use it right away.
To analyze the traffic, we're going to use the tcpdump tool.
Here,
we are telling you to only analyze the traffic that occurs on the en0 interface,
which is my network interface, and with the target host IP address 172.28.0.4,
which is the Docker container.
Since we want all these network logs to end up in Kafka,
we're piping them on the network.log file.
In parallel, we also need to perform the port scan attack.
This attack can be performed using a tool called Nmap.
If you're not familiar with Nmap or port scan attacks,
I definitely recommend checking the Nmap.org website for more information.
Again, the target can be any machine on your network,
but for now I'm going to stick with the Docker container.
However,
I don't recommend using Nmap for launching port scan
attacks against systems you do not own,
as it may attract severe negative consequences.
Civil lawsuits may be brought by the owner of the scan system,
so it's best to only target systems that you own.
It seems the scan was pretty quick,
so let's head back over to tcpdump and stop the sniffing
process as well by pressing Ctrl+C keys.
Excellent!
We can see that 2008 packets have been captured.
So this means our network‑logs analyzers should have triggered another.
To check if this happened, we're going to use ksqlDB again,
since it's the easiest way to inspect the content of a topic.
The topic name is alerts.port‑scan‑attack.
It seems something has indeed been published in this topic,
but ksqlDB is having some troubles to figure out which type of data it is.
To solve this issue,
we need to define a data structure just like in the previous demo.
Only then, we can see the result using a select statement.
The downside to this approach is that we can only see events that have
been published after the select statement started.
So we need to perform another port scan attack.
First, I'm going to initialize traffic sniffing using tcpdump,
and then launch another port scan attack using Nmap.
The best part is that a port scan attack only takes a couple of seconds,
so we don't have to wait a long time until we get the results.
Again, 2006 packets have been captured.
So we should definitely have an alert produced.
Switching back to ksqlDB terminal, and indeed,
we can see that Nmap has launched the attacks from two different ports.