-: So let's have a look at S3 inventory.
So the idea is that using an S3 inventory operation,
you can list all the objects
as well as their corresponding metadata in your S3 buckets,
which is a better way than using an S3 list API operation
to list all your objects
and gets all the associated metadata.
So some usage example for S3 inventory include
creating an audit and report on the replication
and the encryption status of all your objects.
So using this, for example,
you can identify which objects are not encrypted.
You can get the number of objects in an S3 bucket
or you can identify the total storage
of all the previous object versions of your bucket
because S3 inventory can list all object versions.
So the output files is CSV, ORC and Apache Parquet.
And this inventory can be generated
on a daily or weekly basis.
All the data can be queried then using the famous tools
you have in AWS such as Amazon Athena, Redshift, Presto,
Hive, and Spark, including that Presto, Hive and Spark
are not within AWS, it can be from outside
and you can generate a filtered report using S3 Select
and use that report into the S3 batch capability.
The use cases for inventory is going to be business,
compliance and regulatory needs.
So let's go in there hands-on to see how it works.
So let's set up as inventory and for this,
I'm going to create a bucket and the bucket name's
going to be s3-inventory-staphane-demo,
something like this.
And then I will go ahead and just create this bucket.
So my bucket is now created and I can view the details.
And now I need to set up inventory
on another bucket of mine.
Then I picked one of my buckets.
So demo-stephane-s3-bucket-2020, which has some data in it.
So that's all that matters to me.
And I'm going to click on management and then scroll down.
And at the very bottom I have inventory configurations.
So I can create an inventory configuration right here.
I'll call it demoinventory.
You can set up a prefix
if you want to apply to a SQL prefix, but we won't.
And this will include a current version only,
but you can include all object versions if you wanted to.
And then the report details should go in this account,
and then you need to specify a destination for it.
So we can go into my s3-inventory-demo-stephane,
which I just created from before.
And then there is going to be a following statement
that will be added to the destination bucket policy
to allow Amazon S3 to place data in that bucket.
So here is the bucket policy that will be applied.
Then what do we want the frequency to be?
So daily, and the first report will be delivered
within 48 hours or weekly as well as in the output format,
so CSV or ORC, or Parquet
and CSV is the format you need to choose
if you plan to use S3 batch operations
or if you want to analyze the S3 inventory using Excel.
And if you use RSV and Parquet, then you can, for example,
analyze it's using a tool like Athena.
So this configuration is going to be enabled, okay?
And we'll select CSV whether or not
we want server-side encryption so I can disable it for now.
And if you want additional fields for the reports,
you can have size, less modified, storage class, ETag,
multipart upload, replication status, encryption,
bucket status, then whether or not there intelligent tiering
and the all objects luck configurations.
So we're good to go.
So this bucket has not been found in the eu-west-1 region.
So I need to choose to recreate that bucket,
but in the same region that I'm in right now.
So let's go ahead and do this.
So I'll take this S3 asset.
So this inventory buckets.
And it's always good to have errors.
So this way I always keep them on video.
So we'll delete it.
And as you can see, it shows that inventory
has to be done from within the same region.
So I will recreate this, but this time in eu-west-1.
And then create my buckets.
And this is not going to work right now
because the old bucket is being deleted.
So let me pause for a second or I will just rename it demo-2
and this should work just fine.
My bucket is now created,
and I will just add demo-2 in here
and we should be good to go.
Let's create this.
And now my inventory configuration is created.
So what I need you to do now is to wait a little bit
for this inventory to be active,
and this could take up to 48 hours.
So we'll get back to you in 48 hours.
Okay, so I've left the inventory on for a few days now.
And if you go into the demoinventory bucket,
you can see that a folder has been created for every day.
So if I take any of these days, as we can see,
we get a manifest.checksum and a manifest.json.
And this contains metadata around the data itself.
So if we look at the manifest.json file and we download it,
that file manifest.json
is going to show us the source bucket,
the destination bucket, the version of the manifest file,
the file format, which is CSV and the file schema.
So all the columns that are belonging into my CSV file,
as well as the files that corresponded to this manifest.
As you can see, one file is in it.
This is this one right here, demo, demoinventory data.
So demoinventory/data, and then this key right here.
So let's go ahead and find that file right now.
So back in my folder, I go back to the demoinventory
and then scroll down, find the data.
And in here, I have found all the files that correspond
to every single day.
So July 4th is this one.
So I will take this file,
but they should all be the same because my buckets content
has not changed and I'm going
to save this file and extract it.
And so if we have a look at the content of the CSV file,
we can find the first column corresponds to the bucket name.
The second column corresponds to the files in it.
The third column is going to correspond
to a version ID of the files.
Then we get some true and false information
around encryption, I guess, some file size dates and so on,
as well as the last column, which is going to be sorry,
the type of encryption right here that we have, okay?
So this file contains really all the metadata around
all our S3 files, which is very helpful
if you wanna get a list
or if you're gonna get an encryption status and so on.
And when you're done with the hands-on,
please make sure to go to management
and then to delete or disable this inventory configuration,
because otherwise it will keep on running every single day.
So that's it for this lecture.
I hope you liked it.
And I will see you in the next lecture.