Hi, everyone, and welcome back.
Now, in today's video, we are going to explore the storage options specifically related to the open
search service.
Now, one important part to remember is that the open source service uses the dedicated master node
to increase the cluster stability.
So you see that there are three master nodes and you have certain amount of data nodes.
So at a high level overview, a master node performs a cluster management task, but it does not hold
the data or respond to the data upload request.
So when we discuss about the nodes where the data can be stored, it is primarily the data nodes that
you see.
However, the master node takes care of the overall cluster management.
Now, when we discuss about the overall storage options, you have the master nodes, so you can have
a certain amount of master nodes.
This is something that as we know it takes care of the overall cluster management.
On top of that, you have the hot data nodes over here.
So these are the nodes where you have a EBS volume or the instance or volumes where the data can be
stored.
And then you also have the ultra warm data nodes, where the data is generally stored at the three level
here.
And depending upon the use case, you can use the combination of them and it can give you a significant
price to performance benefits.
So let's do one thing first.
Let's go ahead and discuss about the hot data nodes.
Then we'll be discussing about the ultra warm data nodes as well.
So at a high level overview, the standard data nodes uses the hot storage, which takes in the form
of the EBS volumes or the instance store.
One of the benefits here is that the hot storage provides the fastest possible performance for indexing
and searching of new data.
So this is the first primary option.
Now, as we know that this provides the fastest possible performance, but it also comes with a price
because EBS volumes, when you have large amount of data, it can be a little expensive.
So in the organization where price can be an important factor, you also have the ultra warm data nodes
that are available now at a high level overview.
The ultra tier acts as a caching layer on top of the data in S3.
So in this case the ULTRA moves the data from S3 onto the ultra warm nodes on demand, which in turn
speeds up the access for subsequent queries.
And within here you can add or remove the ultra warm nodes to increase or decrease the amount of cash
against your data in S3 to optimize the overall cost.
So let's do one thing before we proceed further.
Let's look into where do we see all of these options now within the open search service console, whenever
you create a domain, let's quickly explore.
So within here you can select data nodes.
As we know in data nodes, the primary storage type is EBS and instance store volumes depending upon
the instance.
So the storage type is EBS volumes.
On top of that, if we go a little down, you have a warm and cold.
Data storage will be discussing more in detail about the cold data storage.
But at this stage we are looking for the warm warrior.
So when you enable the warm data nodes, let's enable it.
Here you can specify the number of warm data nodes and you also have instance types.
Now again, the instance types are very limited over here and depending upon the overall size, you
can use one among them.
Now, as we are discussing that the ultra warm tier acts as a caching layer on top of the data, which
is in S3.
So in such cases there are certain limitations.
First one is that the data in ultra warm is immutable.
That means you cannot modify the data in case if you want to modify the data, you can bring back the
data to the hot tier, modify it if needed.
The third primary option and the cheapest one is the cold storage.
Now, one of the benefits of cold storage is that it allows us to store any amount of infra currently
accessed or historical data and analyze it on demand.
And within here the storage is backed by Amazon S3 so everything is stored at the S3 level and when
you need to query the cold data, you can selectively attach it to the existing ultra warm nodes.
So when you discuss about the cold storage, there are certain pointers to remember.
First one is that the data is not directly variable and you must attach it before the data can be analyzed.
This is very important part to remember.
So let's discuss about the important part related to the overall pricing.
When we discuss about the hot tiers, the pricing is based on what you have provision.
For example, let's say you have provisioned a storage of one terabyte.
So irrespective of how much data you have stored, you will be charged for one terabyte.
It does not really matter if you're using one MDB or one TB.
Now when we discuss about the ultra warm node that.
Charges are at a per hour level, just like the other note, because again, you have to deploy a specific
node.
But when we discuss from storage point of view, you only pay for the storage that is actually stored
in S3.
So if you have stored ten terabytes of data, you pay for that.
And when we discuss about cold storage, it does not incur a compute cost.
And like ultra mode, you only build for the amount of data stored in S3.
However, in cold storage, as we know, if you want to access the data, it must first be loaded post
which you can run the query against it.
So if we look into it at an architectural perspective from a high level overview, you have a certain
combination of the data node which provides the fast performance.
Then you have certain set of ultra warm nodes and then you have the cold storage and then you also have
the master nodes.
So that's the very high level overview.
And with this will conclude today's video.