Hi, Adrian, and welcome back.
Now, in today's video, we are going to discuss about the basics of a data sync service.
Now, at a high level overview, a data sync is an online data transfer service that simplifies and
also automates moving of data between the storage systems and services.
So this can be seen in the diagram where you have a location one and location two.
And if you want to move some data between these set of locations, then data sync can be a reliable
service to use.
Now, one of the benefits of using data sync is that it provides the end to end security.
So this includes encryption, integrated validation as well.
Now, at this stage when we discuss about location one and location two, it is still not clear what
exactly this refers to.
So let's go ahead and understand this in more detail related to the supported locations.
Now, when we discuss about the overall supported endpoints, note that data sync can copy the data
to and from the NFS servers, the SMB servers, the HDFS, the object storage systems, the three buckets
you have, FS, FS X, these no con devices, the Google cloud storage buckets as well as Azure files.
Now when we discuss about data saying there are two primary set of architectures, one is moving the
data between the services and then moving the data between a bus and on premise or maybe a bus and different
cloud environment like Azure, etc..
So when we discuss about transferring the data between the on premise and a bus, note that there is
a requirement of data in Cajun.
So here you have a on premise storage, it can be NFS, PDFs, etc. So if you want to move the data
to a side for an example to FS three etc., you will need a data sync engine.
Very important part to remember.
However, when we discuss about transferring of data between storage services, then in such case no
data sync agent is required.
Great.
Now, before we proceed further, let's have a quick demo to understand things in more detail.
Now, currently I am in the Data Sync service and I already have a task configured.
Now, if you look into the task here, the source is an S3 bucket named Demo Cloud four and I bucket
and net destination is FS.
So there is a PFS that is running where the data needs to be copied to.
Now, if you look into this task here, let's look into the history.
In history you have two execution IDs, one with error and one with success.
So if you look into the success over here, it gives you details related to how many files were transferred,
the total data transfer, the amount of compression and others, and this helps in the overall walkthrough.
So let's do one thing for today's demo.
I'll rerun this task again to see how exactly the data gets synced.
Now, at this stage we know that the S3 bucket from which the data is saying is demo, cloud and bucket.
Let's quickly open this up and I'll be adding certain additional files as part of this bucket.
So I have added files zero one and file zero two.
Let's go ahead and quickly upload this grid.
So at this stage there are two additional files over here.
Now, from the EC2 instance where I have the face mounted, we have a specific directory called this
data sync.
So this is the folder where the data sync will be uploading the files to as part of the overall task.
Now, if you look into this specific folder, there is only one file which is HTML, which was copied
as part of the earlier execution.
So let's go ahead and rerun our data sync.
So what I'll do, I'll start it again.
So it has started the execution.
Let's quickly wait.
Let's go back to the task.
The status is currently running.
And after 2 to 3 minutes, our status is now available.
So if you'll quickly look into the history, our new execution task is now success.
Now, from the ClA, if you run the LS command again, you'll see that we have two new files that are
currently synced.
Great.
So I hope with this at a high level overview, you understood what the data sync services trying to
achieve.
Now, one more thing that I wanted to show you is that as part of data sync, you can set a schedule
where after every, say, 2 hours, the data gets sync again so you can go ahead and specify a schedule.
Let's quickly look into it.
So if you look into the frequency you have hourly, daily, weekly, and you can also have a custom.
So say every hour the data gets synced between two set of locations that you specify.
So this is also something that you can set.
Now, before we conclude, let's look into a simple use case of transferring the data between the Google
Cloud storage to Amazon S3.
So if you have such a use case, what are the steps that will be involved?
First step is that you have to deploy a data sync agent in your Google Cloud environment.
Second step is that the agent reads your Google cloud storage bucket, so the permission needs to be
present.
Third step is that the object from your Google cloud storage bucket move securely through TLS into the
cloud by using the public endpoint.
And the fourth step is that the Data sync service writes the data to your S3 bucket.
So if you look into this specific diagram as well, it will give you an overview where you have a data
sync agent which reads the objects, and then through the HTTPS connection, through the data Sync service,
the data gets moved to your S3 bucket.
So that's a very high level overview about the data sync service.
And with this will conclude today's video.