During this course, I will use Visual Studio Code,
but as long as you have a basic text editor and a terminal,
you can easily follow along.
Just as I mentioned in the previous clip,
we will use Docker to deploy the Kafka components.
And to be more precise, we'll use a docker‑compose file.
A docker‑compose file is a descriptor of all the components we want to deploy.
Here,
we have no more and no less than 11 components that are going to be deployed,
three zookeepers, three brokers, a schema‑registry,
Kafka Connect, ksqldb‑server and cli, and finally,
a custom application called audit‑logger.
We'll go through each application and explain what it does.
But first, I would like to talk a bit about security setup.
In this example,
we'll be using Mutual TLS to secure the connection between components.
But, since all of them are using a JVM to run, we need a bit of a special setup.
Let's take, for example, the Kafka broker.
As you may notice, there are a lot of properties that need to be added.
In fact, if you are interested in setting up a secure Kafka environment,
I recommend one of my other courses, Securing a Kafka Cluster.
In it,
I explain how to properly set up different security
mechanisms such as encryption, authentication,
and authorization with a Kafka cluster.
Back to our demo.
The key aspect you need to notice is the volumes mount.
Using a Docker volume,
I'm mounting my local secrets folder on the Docker
container at the path /etc/kafka/secrets.
Then, all secrets from this path are referenced in the properties.
We'll soon see exactly what those secrets are,
but first, we need to do another preparatory step.
We need to add some entries in the hosts file.
On UNIX systems, the hosts file is placed in the etc directory.
On Windows, the path is C:\Windows\System32\drivers\etc\hosts.
For every broker and zookeeper instance, we need to add a loopback address.
This setup is required because we are deploying multiple instances
over brokers and zookeepers on the same machine.
With this in place, we can finally check what's inside the secrets folder.
In the secret folder, there are two files,
create_secrets script and a password.txt.
The password.txt file contains the password we're
going to use to protect our secret, which is going to be notsecret.
The create_secrets script can be used to create all
secrets required by our setup.
To run the script, we can simply execute it from the TERMINAL.
We will be prompted a couple of times if we trust this certificate,
and we simply have to write yes, and then press Enter.
So what is this script actually creating?
First, the script is creating a self‑signed Certificate Authority.
Using the CA,
the script then signs a certificate for each component I mentioned before.
Now,
Java applications are using two cryptographic
containers called KeyStores and TrustStores.
Basically,
a KeyStore contains the certificate that will be used to
identify a server and a public/private key pair.
The key pair is used to encrypt the communication
between the server and the client.
The TrustStore, however, only contains the CA certificate.
After everything is complete, we should have two JKS files per component,
a KeyStore, and TrustStore.
So, we can start deploying the components.
Since the docker‑compose file describes what components we
want to deploy and how to deploy them, we only need to run a single command.
But first, let me switch back to the main directory.
Here, we need to run the following command docker compose up ‑d.
And that's it.
Now, depending on how fast your computer is,
it might take a couple of minutes to deploy everything,
so I suggest waiting for at least 1 or 2 minutes before proceeding further.
To check everything is up and running, we can use the docker ps command.
We need to check that all containers are up.
I can already see three zookeepers, the schema‑registry, and the audit‑logger.
And then, if I scroll up a bit, the three brokers, Kafka Connect.
And if I scroll up even more, ksqldb and ksqldb‑cli.
In total, we have 11 components, and all of them are up and running.
Excellent.
Now, we can jump into the first feature we're going to build,
ingesting audit logs into Apache Kafka.
Before ingesting logs, we actually need to create them,
that's the role of the audit‑logger.
In fact,
the audit‑logger is nothing more than an HHTP server that
creates an audit log when a resource is created.
To execute the HTTP request, we're going to use curl.
Here, we are creating a resource of type vm called server‑1.
In fact, let's create another one, this time with a different name.
To authenticate the user,
we're using a simple basic authentication header that
contains a base64 encoded username and password.
Excellent.
Both of these requests returned a 201 Created response,
but where exactly can we see the logs?
The logs can be found in the logs folder that now
contains a file called audit.log.
The structure of the audit.log is pretty simple,
we have the date and time for when the resource was created,
the username alongside the IP of the person that created it,
the resource type, the action the user performed,
and finally, the name of the resource.
Now that we have the logs in place,
it is time to ingest them in the Kafka cluster.
The way we are going to do this is by using Kafka Connect.
Kafka Connect is specifically designed for sending
data to and from a Kafka cluster.
However,
Kafka Connect is not the only tool that can transfer logs to a Kafka cluster.
Other solutions such as Logstash, Beats,
Apache Flume, Fluentd, and many others may work just fine,
so check with your company if any of these tools are already used.
Kafka Connect leverages the concept of a connector.
Basically, a connector is a task that defines where data is coming from.
In this scenario, we're using a FileStreamSource connector called audit‑logs.
The audit‑logs connector will take the data from the audit.log
file and publish it to the audit‑logs topic.
The file is mounted inside the Docker container using a Docker volume,
so Kafka Connect has access to the log file we just saw.
Now, Kafka uses topics to keep different data in buckets.
Just like a database is using tables to separate different kinds of data,
Kafka uses topics to separate events from one another.
To actually see the contents of a topic,
we need to use ksqldb, which is composed of two components,
a CLI and a server.
Right now, we're using Docker to open a CLI that connects to the server.
Ksqldb allows us to query and manipulate data in Kafka
by using an SQL‑like query language.
For example, we can print all the contents of a topic using the PRINT statement.
And there they are.
We can see that the two audit‑log entries we produced
earlier are now present in the Kafka cluster.