‫Now let's talk about CloudWatch Logs.
‫So when you want to store your logs in AWS,
‫the best place is CloudWatch Logs.
‫So the idea is that you're going
‫to group these logs into log groups,
‫and this is a name that you want to give it,
‫but usually it's representing an application.
‫And within each log group you have log streams.
‫And they represents instances within the application
‫or different log file names
‫or different containers, and so on,
‫then you define log expiration policy.
‫For example, you may never want the log to expire,
‫or do you want it to be deleted after 30 days and so on,
‫because you are paying for storage on CloudWatch Logs,
‫then from Cloudwatch Logs,
‫you can export the logs through multiple places,
‫such as Amazon S3, Kinesis Data streams,
‫Kinesis Data Firehose, Lambda, and OpenSearch.
‫Now what types of logs can go into CloudWatch logs.
‫Well we can send the logs using the SDK
‫or the CloudWatch Logs Agent
‫or the CloudWatch Unified Agent.
‫Now the CloudWatch Unified Agent send logs to Cloudwatch
‫into the CloudWatch Logs Agent is now sort of deprecated.
‫You have Elastic Beanstalk, which is used to collect logs
‫from the application directly into CloudWatch,
‫ECS will send the logs directly
‫from the containers into CloudWatch.
‫Lambda will send logs from the functions themselves,
‫VPC Flow Logs will send logs
‫specific to your VPC metadata network traffic,
‫API gateway will send all the requests
‫made to the API gateway into CloudWatch logs,
‫CloudTrail, you can send the logs there
‫based on the filter
‫and Route53 will log all the DNS queries
‫made to its service.
‫So another thing you can do that's very important
‫is to define metric filter and insights.
‫So the idea is that you have your Cloudwatch Logs
‫and you can use your filter expressions, for example,
‫to find all the specific IP within, within logs.
‫So find the log lines where that IP appears
‫or find every log line in your logs
‫that contains the word errors.
‫And then thanks to this metric filter,
‫you can start counting these occurrences and then these
‫becomes a metric, okay?
‫And this metric can be linked into a CloudWatch alarm.
‫Then the other feature that's really cool to discuss
‫is CloudWatch Logs Insights.
‫So the idea is that the CloudWatch Logs Insights,
‫You can query logs and add these queries
‫into CloudWatch Dashboards directly.
‫And some common queries are added directly by AWS,
‫and this is a very easy language to use.
‫So first let's talk about the S3 export.
‫So you send from CloudWatch into Amazon S3,
‫and this can take up to 12 hours
‫to become available for export.
‫And the API call is CreateExportTask,
‫but this is going to be done in its own time.
‫And so it's not near real time or real time.
‫Instead, if you want to stream logs from CloudWatch logs,
‫you need to use subscriptions,
‫now subscriptions are what?
‫Well subscriptions are a filter
‫that you apply on top of your Cloudwatch logs,
‫and then you can send it to a destination.
‫So it could be a Lambda function.
‫For example, you define a custom or one that is used by AWS.
‫If you want to send data into Amazon OpenSearch directly,
‫or it could be Kinesis data Firehose,
‫if you want to send it for example,
‫to Amazon S3 in near real time.
‫So this is a faster alternative
‫to the one I just showed you
‫from before using the exports from CloudWatch to S3,
‫or you could use Kineses Data Streams, for example,
‫to send the data into Kinesis Data Firehose
‫Kinesis Data Analytics, Amazon EC2 or Lambda, and so on.
‫Finally, with Cloudwatch Logs,
‫you can do some log aggregation across accounts
‫and across regions.
‫So you may have like multiple accounts with,
‫for example, a region that has a subscription filter.
‫That's sent into Kinesis Data Streams in a common accounts,
‫same for account B.
‫So region two, same architecture and so on.
‫And so we can centralize all these logs together into
‫Kinesis Data Streams, for example, and then Data Firehose.
‫And then for example, Amazon S3.
‫Okay.
‫So that's it for the overview of how Cloudwatch Logs work.
‫I hope you liked it, and I will see you in the next lecture.