Hi, everyone, and welcome back.
Now, in today's video, we are going to discuss about the Amazon FS for Luster in more detail.
Now at a high level overview, we already know that the Amazon face for Luster is a fully managed file
system that is based on Luster.
Now, if you look into this diagram, it basically signifies the oral workflow.
So what do you do?
You generally create the luster file system.
Then you can link or you can integrate it with a service, which is primarily the Amazon S3 where the
sync can happen and then you associate or you mount this specific file system to multiple set of resources.
It can be easy two instances X and X on premise, etc. And then once the file system is mounted, then
you can go ahead and run your machine learning HPC tasks.
So this is the overall workflow.
So as part of Luster, there are certain use cases that you'll have to remember is that it is primarily
used for machine learning use cases.
The HPC use cases for big data analytics and even for VFX.
So as part of the overall workflow, let's assume that your FS four luster is integrated with S3 bucket.
So the first primary step would be that the data that is stored in S3 is loaded to your FS four luster
for processing.
So there would be some data that needs to be processed, which is in the S3 bucket.
So you load that data to Luster file system.
Luster file system in turn is mounted on a set of EC two instances.
So once the files are mounted in Luster, the processing on those files can take place in the set of
EC two instances and the final process output can be returned back to the S three for the overall retention.
Now, one specific part that you need to remember is that the FS four luster is natively integrated
with data repository like Amazon S3 So if you look into this specific screenshot, you have the data
repository integration, it is of type three, and then you specify the S3 bucket.
So this is the specific bucket from which the data will be added to the Luster file system.
Now let me also quickly show you so I already have the Luster file system that is available now in this
specific file system of if you look into the data repository, you have the Amazon S3 bucket, you have
the import path, you have the export part.
So you can associate a specific S3 bucket with Luster.
In fact, whenever you create a Luster file system, let's go a bit down.
You have a specific option related to the data repository where you can import the data and also export
the data to S3.
So if you'll see, you will have to specify the three bucket.
And again, there are certain import as well as export settings.
So in the import settings you can specify there are three primary options here.
So you have to choose which updates on the data repository.
In this case, three should be propagated to the file system.
So basically whenever we make some updates to the S3 bucket based on what kind of updates should be
propagated to the file system, primarily because it will be linked.
Similarly, you also have the export settings where you can choose which updates that are part of the
file system should be propagated back to the data repository, which is S3.
All right.
So I hope at this stage you understood what the data repository means.
Now in here you need to understand the basic concept of lazy loading of data.
So at a high level overview, the lazy loading is a practice of delaying load or initialization of resources
or objects until they are actually needed.
So this can be better understood with this diagram where let's assume that you have a bucket and it
contains petabytes of data and you have the FS for Lustre file system.
Now when you integrate the FS with the S3, do you need all of the petabytes of data in S3 to be part
of the lustre?
The answer is no in most of the cases, and this is the reason why what you do, you only load the necessary
data.
And based on that, this necessary data will be processed by the EC two instances.
So this not only improves your overall performance, but it also saves a lot of resources.
Now, when we discuss about the overall deployment options, there are two primary options that you
will see.
One is the scratch and second one is the persistent.
So if you will see even in the diagram in the deployment and storage type, you have the persistent
and you have the scratch.
And depending upon the option that you choose, the overall storage capacity as well as the throughput
will also change.
So the first deployment option that will explore is the scratch.
So at a high level overview, the scratch file systems are designed primarily for temporary storage
and short term data processing.
The benefit here is that it provides high burst throughput.
However, since it is designed for temporary use cases, the data is not replicated and will not persist
if a file server fails.
The second deployment option type is.
Persistent and this is recommended for long term storage and workloads.
The benefit here is that the file servers are highly available and the data is automatically replicated
within the same easy in which the file system is located.
Now let's also explore that.
So if you go a bit down into the configuration of lustre, you have the persistent SSD, you have the
persistent HDD.
Again in persistent, you have both the HDD and SSD and you have the scratch SSD.
Something to note here is that the throughput changes.
For example, in scratch SSD, you have 200.
In persistent SSD, you have multiple set of options that are available.
Now before we conclude, there are certain important pointers that you need to keep a note of.
First one is that you can configure the FS for lustre to keep content synchronized in both the direction
the file system as well as the linked S3 buckets.
Second option is that alternatively, you can also have the option to import and export batches of new
and changed data between the file system and S3 for the overall fine grained control over the data synchronization.
So both of these options are available depending upon the use case.
Now, before we conclude, I also wanted to show you at a high level overview on how the file system
is connected to a specific resource like EC2 instance.
So the more accurate word would be mounting.
Now, at this stage we have one cluster file system and this Leicester file system, as we know, one
of the data repository here is the AP lab sample set bucket.
So now we want to associate this file system to the EC two instance that we have.
So what you need to do, you have to run the mount a specific command and within here you'll have to
specify the file system DNS.
So for example.
If you go back, you get the DNS name.
Let's copy the dense name.
I'll copy this, let's copy this here.
And then you'll also have to replace the mount name.
Let's copy the mount name.
I'll add it here and then you can specify which directory will this file system be mounted.
For example, let me just create a new directory.
I'll just copy labs fix on the root.
All right.
So what you can do, you can specify the directory, which is CPU Labs F6, and you can run the command,
and that's about it.
If you run mount, you should see that our file system is mounted on CPU labs F6 directory, and if
you explore the contents of CPU labs FS folder here, you should see the files that are part of the
S3 bucket.
All right.
So I hope this has given you much more clarity on how file systems that you create as part of PHS are
associated or mounted as part of the EC.
Two instances.
So with this will conclude today's video.