Welcome back.
So we learned that as a data engineer you collect data you ingest data and then you store it in a data
lake often a Hadoop cluster.
Now this idea of a data lake can also be used with something like a W S S three object storage.
So those are the two most popular ones.
And then once they're in your data lake you can usually run batch jobs on this data to clean it up or
modify it or wrangle it in some way so Hadoop allowed us to do that with map produce.
We can use Apache Spark.
You can just manually write scripts on Amazon S3 or you can just even do best jobs with com databases
like askew all databases.
But this idea of stream processing is now becoming more and more popular instead of processing data
using just batch jobs let's say every hour or every day.
There's now a movement towards this idea of real time stream processing.
That means when data is received processing it right away instead of waiting a bit until we can do batches
at a time this allows us to have faster reactions to data.
Now this idea of stream processing is pretty complicated but one of the main tools that's being used
right now for stream processing is something called Kafka Kafka.
If we look on their Web site is a distributed streaming platform.
It allows us to read and write streams of data like messaging system.
It allows us to process and it allows us to store data as well.
But as you can see what Kafka does it allows us to receive messages and pass it on to different places
so why you'll often see is we use Apache Kafka to receive all these messages logs and data from all
these sources we collect them in a central location and then Kafka can pass it on to different locations.
For example we can pass off whatever Kafka receives to a Hadoop cluster to store that data and then
perhaps if we want use Apache Spark to process that data into a data warehouse but now with the idea
of real time processing Kafka allows us to now pass things on to well itself or we can use something
like spark streaming or flank which we've talked about a patchy flank that is used more for stream processing
or Apache Storm or even cornices by Amazon which actually can also be placed like Kafka here to handle
all these messages from all these locations.
Now at the end of the day all of this might have been confusing.
You may be overwhelmed by all of these tools that are available.
But remember what I said at the beginning of this section a data engineer is somebody that you'll interact
with as a data scientist.
So it's good to know what they do what their roles are what their responsibilities are and you'll work
closely with them for you to have the data available to perform analysis on it or build machine learning
models.
But I hope that this diagram now makes sense.
A data engineer creates this landscape for you where they ingest data through Kafka store information
using something like Hadoop or Amazon S3.
They process that data using something like Apache Spark or Canis and then they even store data in data
warehouses.
So tools like redshift or Bitcoin can be used by people inside the business to learn something useful
from the data they build the structure for you to flourish.
So be nice to them and if you're ever interested in any of these technologies they're changing very
fast but they're definitely worth a look and they're not as intimidating to learn as you might think.
I'll see in the next one by.