Even though we spend only a half hour together, we've done a lot of things.
We started by defining what is Apache Kafka and what
capabilities a distributed streaming platform has.
However,
only talking about Kafka doesn't give us many benefits so we
also need to know where we can get it from,
and most importantly,
what extra features we can add to our system on top of
the core Apache Kafka components.
Then we've seen what makes it so special by analyzing its core features.
Finally,
we briefly covered NIST cybersecurity framework and
its protective technology function.
Theory is cool, but practice is even better.
So we started building interesting protective solutions using Apache Kafka.
First, we deployed a tree broker Kafka cluster using Docker.
Since Kafka brokers cannot run on their own,
we've also added a three‑node zookeeper cluster to the list of components.
Then using a web service called Audit Logger,
we've audited all the HTTP calls that are creating a resource and
stored all the entries in an audit log file.
Next,
we've used Kafka Connect to transfer those audit entries
from the log file to Apache Kafka.
Kafka Connect is a very powerful tool when it comes to
transferring data in and out of a Kafka cluster.
With the audit entry stored in the Kafka cluster,
we needed a way to analyze them and trigger alert when a user spams the
web service by creating too many resources at once.
For that,
we've used ksqlDB and it's SQL‑like language capable of creating data streams.
Finally,
we had some extra fun by launching a port scan attack using nmap and
dumping all the network logs in a file using tcpdump.
Then,
we've used Kafka Connect to transfer those logs to the Kafka
cluster so that we can analyze them with another Kafka
ecosystem component called Kafka Streams.
The goal of this component is to analyze the network logs and
trigger alerts when it detects a port scan attack.