OK.
OK so what does a data engineer actually do.
Let's start off with three big wars that we need to understand before we really understand what a data
engineer does.
Yes I know I keep circling around data engineer and I keep getting you excited but I promise we're getting
really really close.
We just need to discuss these three wars that you may or may not have heard of.
One is data mining and data mining simply means pre processing and extracting some knowledge from the
data so we use some sort of data to extract knowledge from that's what data mining is Big Data.
Well it means that we have a lot of data a lot of variables so much data in fact that you can't run
on your laptop and there's just too much data for your laptop or computer to hold usually when we talk
about big data.
It's data that's so big that you need to have it running on cloud computing or multiple computers such
as a W.S. Azure and Google cloud because they have a lot of computers and a lot of storage to store
this data.
So that's what big data has Big Data is data that usually can't have just on one computer.
This usually happens because what data sets get so big.
We have petabytes and petabytes of data now so just having data in a database like my ask you all or
post grass or any type of database becomes really difficult when that database is just one single machine
now.
New technologies were invented to solve this problem of big data like Hadoop and no Eskew.
Which we'll talk about now a Data Pipeline is essentially a pipeline that a data engineer built to essentially
use the fact that we had this big amount of data we need to extract information from this data using
data mining.
So we need to bring or build a pipeline that allows us to flow from that unknown large amount of data
to a pipeline that extracts data to a more useful form.
So a data engineer essentially does this creates a data pipeline where all the information that different
devices like IO T devices mobile applications web apps cameras cars and pretty much anything that collects
data and stores information or logs data into servers or to the cloud a data engineer essentially accumulates
all this information into nicely packed databases and stores engines so that different parts of the
company can create visualizations they can monitor the performance of their product they can get business
insights and make business decision from this data and even use this data on their apps for example
for user profiles before a data scientist or a machine learning expert or even a business intelligence
or data analyst gets hired for a big company.
The thing that they need to do before all of that can be done is to hire a data engineer they build
the pipeline that allows us to work as data scientists.
Remember this diagram and how we've been focusing on this part well.
A data engineer allows us to do this data collection part.
They bring in all this information organize it in a way for us to do our data modelling.
They really help us with this data collection part that usually as a machine learning engineer or data
scientist you don't have to concern yourself too much with but you know what my favorite demonstration
of what a data engineer is is in the next video.
I hope you're excited because things are gonna start to make more and more sense.
I'll see you on that one by.