You probably already have an idea what Apache Kafka is,
but why should you use it?
What makes it so special?
Probably the most important feature of Apache Kafka is its high throughput.
Kafka can handle millions of events per second.
You heard me right.
There are companies that have throughputs of trillions events per day,
which is insane.
This makes Kafka an excellent tool for ingesting and
manipulating a ton of data in real time.
However,
the only way to handle this amount of data is by being extremely scalable.
A Kafka cluster can scale up even to 1,000 brokers without any issues.
Another cool feature of Kafka is that we can have permanent storage.
If we compare it to other message brokers that can only
store messages for a limited amount of time,
Kafka can store them indefinitely.
Finally, Kafka is designed to run in a highly available configuration,
meaning that if one or more instances go down,
the cluster may still work without interruptions.
Now, since this course is security‑oriented,
I would like to talk a little bit about the NIST Cybersecurity Framework.
NIST is a set of standards, guidelines,
and best practices for managing cybersecurity risks.
It's a voluntary program,
but it is extremely powerful and will definitely
improve your cybersecurity program.
As you may notice, it consists of multiple functions,
Identify, Protect, Detect, Respond, and Recover.
If you are interested in finding out more about the
NIST Cybersecurity Framework, I recommend that you check out Pluralsight,
as there are multiple courses that detail every aspect of it.
Today, we'll explore Kafka from the Protect perspective.
The Protect function has several categories,
Identity Management and Access Control, Awareness and Training,
Data Security, Information Protection Processes & Procedures,
Maintenance, and Protective Technology.
Out of these six categories, Kafka fits best under Protective Technology.
To be more precise,
we'll explore how Kafka can be used to implement the
first Protective Technology subcategory.
Audit and/or log records are determined,
documented, implemented, and reviewed in accordance with policy.
To fulfill this subcategory,
we'll go through a series of demos and start to set
up a Kafka cluster using Docker.
If you're planning on following along,
you should keep in mind that Docker is required.
Then, we'll start using different components from the ecosystem.
First, we'll ingest audit logs into our Kafka cluster using Kafka Connect.
Once the audit logs are ingested,
we can analyze them using another component called ksqlDB.
Finally,
we'll see another tool in action by analyzing network logs and
checking for port scan attacks using Kafka Streams.