Now that we've seen what Apache Kafka is comprised of, where can we get it?
The short answer is kafka.apache.org. There you can find
all the binaries and libraries required to run Apache Kafka
on your enterprise; however, Kafka, by default,
is not enterprise ready.
While working for an enterprise,
we often encounter some very specific requirements that must be
fulfilled in order to ensure the system is production ready.
Requirements such as data governance, integration capabilities
with various systems, testing,
high availability, and, the most important one, security.
To tackle all these enterprise requirements, we need to extend Apache Kafka.
If you want to solve a specific problem using Kafka,
you will probably encounter two types of solutions.
First,
we have open‑source solutions where members of the Apache Kafka
community publicized a way they solved different problems.
On the other hand, in the last couple of years,
we have seen multiple companies tackle some common enterprise
requirements and sell them using a commercial license.
One of the most complete lists of solutions currently
available can be found through this link. There,
you will find open‑source and commercial options, but be
warned, it doesn't cover all of them.
The adoption of Apache Kafka has massively increased throughout the
last couple of years, so, there is a high chance that someone else
has published a solution or a problem you are encountering, but it
may not be on the list.
To find a solution, in this case, just do a quick Google search. You
will be amazed at how many solutions are out there.
Two of the solutions we're going to use today are
part of the Confluent Platform.
The platform contains many components, including
commercial and community‑driven offerings.
First, we have the Schema Registry. Schema Registry is used to tackle
some problems encountered in the data governance process and, more
specifically, handling messages schemas and enforcing a specific type
of data for each Kafka topic.
Then, we have ksqlDB.
This tool can be utilized to create streaming
applications using an SQL‑like language.
Both of these solutions are non‑commercial components,
meaning, we are free to access,
modify, or even redistribute the source code. However,
we are living in a cloud‑driven world, and chances are that you are
already using a cloud provider to power your system.
Taking this into consideration, I want you to also be aware that there are
numerous cloud providers that offer Kafka as a service.
There is Confluent Cloud, AWS MSK, Azure Event Hubs,
IBM Event Streams,
Aiven, Heroku, CloudKarafka, Instaclustr, and so many others that
now provide Kafka as a service through a cloud subscription.
Definitely check first with your cloud provider to see if it offers
a managed solution for Apache Kafka because it will definitely pay
off in the long term.