The amount of data produced is growing exponentially each year.
In an era where almost everything happens digitally,
we need really powerful protection mechanisms to guard our
resources and protect them from intruders.
Hi.
My name is Bogdan Sucaciu, and welcome to my course,
Protective Technology with Apache Kafka.
You've probably heard the term Apache Kafka,
but you don't really know what it is or what it does.
Well, it's a long story, so let's start from the beginning.
Apache Kafka, or shortly said Kafka,
was originally developed in 2010 at LinkedIn and was
subsequently open sourced in 2011.
In 2012,
the project graduated from the Apache incubator and became
a top‑level Apache project allowing the users of the
software to use it for any purpose,
distribute it or modify it without concern for any royalties.
Apache Kafka is an open source distributed event streaming platform used by
thousands of companies for high performance data pipelines,
streaming analytics, data integration, and mission critical applications.
Wow, there are a lot of buzz words.
So let's take a step back and see what a distributed
streaming platform actually is.
Well, first of all,
it can be used as a messaging system by using the publish/subscribe pattern.
In this pattern, there are two types of applications, producers and consumers.
Producers create and publish events to the messaging system while
consumers subscribe to those events and consume them.
Popular alternative messaging systems are ActiveMQ and RabbitMQ.
Kafka can also be used for storing data in a distributed way.
It supports clustering so data will be uniformly spread across systems.
It can also be replicated in case one system fails,
thus ensuring data is not lost.
The last process a distributed streaming platform can perform
is taking benefit of the streaming model by processing events
as they occur in the system.
Companies should react as fast as possible to customers needs,
and by using streaming, all incoming events are processed in almost real time.
Apache Kafka is an ecosystem represented by a sum of different components.
The centerpiece of the ecosystem is the Kafka broker.
The Kafka broker is the most crucial component since it
represents the backbone of the entire ecosystem.
We then have the producer API which is the most basic unit
for producing data to the Kafka broker.
In order to retrieve data after it has been published,
we will need the Consumer API.
Next, we have the admin client we can perform different operational tasks.
Creating producers and consumers can result in a lot of boilerplate code.
To reduce the necessary effort to create these applications,
two other APIs have been created, each with its own specific purposes.
Kafka Streams API has been designed for building streaming
applications within the Apache Kafka ecosystem.
Kafka Connect is designed to transfer data in and out of the Apache Kafka
ecosystem to and from some common data systems like databases,
key value stores, search indexes, and file systems.
Along with these components, we need one more element to make it work.
This element is called Apache Zookeeper.
Apache Zookeeper is not part of the Kafka ecosystem,
but for now, it is required by Kafka in order to handle different kinds of tasks.