-: Now let's talk about CloudWatch Logs.
So when you want to store your logs in AWS,
the best place is CloudWatch Logs.
So the idea is that you're going
to group these logs into log groups,
and this is a name that you want to give it,
but usually it's representing an application.
And within each log group you have log streams.
And they represents instances within the application
or different log file names
or different containers, and so on,
then you define log expiration policy.
For example, you may never want the log to expire,
or do you want it to be deleted after 30 days and so on,
because you are paying for storage on CloudWatch Logs,
then from Cloudwatch Logs,
you can export the logs through multiple places,
such as Amazon S3, Kinesis Data streams,
Kinesis Data Firehose, Lambda, and ElasticSearch.
Now what types of logs can go into CloudWatch logs.
Well we can send the logs using the SDK
or the CloudWatch Logs Agent
or the CloudWatch Unified Agent.
Now the CloudWatch Unified Agent send logs to Cloudwatch
into the CloudWatch Logs Agent is now sort of deprecated.
You have Elastic Beanstalk, which is used to collect logs
from the application directly into CloudWatch,
ECS will send the logs directly
from the containers into CloudWatch.
Lambda will send logs from the functions themselves,
VPC Flow Logs will send logs
specific to your VPC metadata network traffic,
API gateway will send all the requests
made to the API gateway into CloudWatch logs,
CloudTrail, you can send the logs there
based on the filter
and Route53 will log all the DNS queries
made to its service.
So another thing you can do that's very important
is to define metric filter and insights.
So the idea is that you have your Cloudwatch Logs
and you can use your filter expressions, for example,
to find all the specific IP within, within logs.
So find the log lines where that IP appears
or find every log line in your logs
that contains the word errors.
And then thanks to this metric filter,
you can start counting these occurrences and then these
becomes a metric, okay?
And this metric can be linked into a CloudWatch alarm.
Then the other feature that's really cool to discuss
is CloudWatch Logs Insights.
So the idea is that the CloudWatch Logs Insights,
You can query logs and add these queries
into CloudWatch Dashboards directly.
And some common queries are added directly by AWS,
and this is a very easy language to use.
So first let's talk about the S3 export.
So you send from CloudWatch into Amazon S3,
and this can take up to 12 hours
to become available for export.
And the API call is CreateExportTask,
but this is going to be done in its own time.
And so it's not near real time or real time.
Instead, if you want to stream logs from CloudWatch logs,
you need to use subscriptions,
now subscriptions are what?
Well subscriptions are a filter
that you apply on top of your Cloudwatch logs,
and then you can send it to a destination.
So it could be a Lambda function.
For example, you define a custom or one that is used by AWS.
If you want to send data into Amazon Elasticsearch directly,
or it could be Kinesis data Firehose,
if you want to send it for example,
to Amazon S3 in near real time.
So this is a faster alternative
to the one I just showed you
from before using the exports from CloudWatch to S3,
or you could use Kineses Data Streams, for example,
to send the data into Kinesis Data Firehose
Kinesis Data Analytics, Amazon EC2 or Lambda, and so on.
Finally, with Cloudwatch Logs,
you can do some log aggregation across accounts
and across regions.
So you may have like multiple accounts with,
for example, a region that has a subscription filter.
That's sent into Kinesis Data Streams in a common accounts,
same for account B.
So region two, same architecture and so on.
And so we can centralize all these logs together into
Kinesis Data Streams, for example, and then Data Firehose.
And then for example, Amazon S3.
Okay.
So that's it for the overview of how Cloudwatch Logs work.
I hope you liked it, and I will see you in the next lecture.