Monitoring and Auditing: CloudWatch, CloudTrail and Config

Over the previous parts we built an application with many pieces. Building it is the easy half. The hard half comes at three in the morning: is it down? Since when? And who changed that setting yesterday?

This last section is about running what you've built, and it starts with three questions, each with its own service:

And a fourth service reacts to all of them: EventBridge.

CloudWatch metrics #

A metric is a number measured over time, such as CPU usage or the count of errors. Most AWS services send their metrics by themselves, grouped in namespaces and distinguished by dimensions, such as the server ID.

Dashboards put the charts you care about on one page, and they can gather metrics across Regions and accounts.

CloudWatch Logs #

Everything your application prints can land here, organized into log groups that usually represent one application, and within them log streams. Lambda, ECS, API Gateway, VPC Flow Logs and others send logs directly, and on EC2 or on-premises servers you use the same agent.

Logs are kept forever by default, and you pay for that. Set a retention period on each group, from one day to ten years.

Logs Insights searches through them with a simple query language. This one gets the latest errors:

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 20

To send logs elsewhere you have two routes:

CloudWatch alarms #

A chart nobody's looking at is no use. An alarm watches a metric and does something when it crosses a limit.

An alarm is in one of three states: OK, ALARM or INSUFFICIENT_DATA. Some useful additions:

• EC2 actions: stop, reboot, terminate or recover. Recovery moves the server to new hardware if the underlying host fails, keeping its IP addresses and settings.
• Composite alarms: one alarm for "high CPU and many errors", which reduces false alerts.
• Testing: set-alarm-state puts an alarm in any state, so you can check the alert path without breaking anything. We'll try it in the hands-on.

Ready-made insights #

EventBridge: reacting to events #

Almost everything that happens in AWS is an event: a server changed state, a bucket was created, or an alarm went off. EventBridge receives these events on a bus, and rules send matching ones to a target.

Features worth knowing: archive and replay to send old events again, the schema registry that describes the shape of events, and resource policies so that another account can send to your bus.

For most targets, such as Lambda, SQS and SNS, access is granted by a resource policy on the target. For others, such as starting an ECS task, EventBridge uses an IAM role.

CloudTrail: who did what #

Every call to AWS, from the console, the CLI or a service, gets recorded by CloudTrail: who, what, when, and from where. It's on by default, and it's the first place to look when something changes mysteriously or disappears.

To keep them longer than 90 days, create a trail that writes to S3, across all Regions and even the whole organization, and analyze it with Athena. Note that CloudTrail Lake closed to new customers in 2026, so a trail is the way to go.

A powerful pattern combines it with EventBridge: anyone deleting a table, or a sign-in with the root user, immediately triggers an alert. CloudTrail records it, EventBridge matches it, and SNS tells you.

AWS Config: your settings over time #

CloudTrail tells you who called what. Config tells you what a resource looks like: it records the configuration of your resources and every change to it, so you can ask what a security group looked like last Tuesday.

Config rules then judge these settings:

• Rules: managed ones ready to use, such as "no public S3 buckets", or custom ones.
• Prevention: it doesn't stop anything. It marks a resource as non-compliant, and that's it.
• Remediation: a rule can run a Systems Manager Automation document to fix the problem automatically.
• Notifications: through EventBridge or SNS.
• Scope and cost: it works per Region, and you pay per recorded item and per rule evaluation.

X-Ray: following one request #

A request in a modern application can pass through an API, three functions and a database. Metrics tell you something is slow, and AWS X-Ray tells you where. It collects traces of each request as it passes through your services, including its calls to databases and other APIs, and shows you which step is slow or failing.

Other views #

Which one, when? #

Let's take one load balancer:

Hands-on: an alarm that emails you #

We'll watch a made-up metric for failed orders. From CloudShell, replace the email address with yours:

EMAIL=[email protected]

TOPIC_ARN=$(aws sns create-topic --name series-alerts \
  --query TopicArn --output text)

aws sns subscribe --topic-arn $TOPIC_ARN \
  --protocol email --notification-endpoint $EMAIL

Open your email and press Confirm subscription. Then create an alarm that goes off if more than five orders fail in a minute:

aws cloudwatch put-metric-alarm --alarm-name series-failed-orders \
  --namespace Series/Shop --metric-name FailedOrders \
  --statistic Sum --period 60 --evaluation-periods 1 \
  --threshold 5 --comparison-operator GreaterThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions $TOPIC_ARN

Now send the application's reading, as if 12 orders had just failed:

aws cloudwatch put-metric-data --namespace Series/Shop \
  --metric-name FailedOrders --value 12

Wait two or three minutes and check the alarm's state. An email should arrive at the same time:

aws cloudwatch describe-alarms --alarm-names series-failed-orders \
  --query "MetricAlarms[0].[StateValue,StateReason]" --output text

You don't have to break anything to test an alert path. Try it when the alarm has gone back to OK, and another email arrives:

aws cloudwatch set-alarm-state --alarm-name series-failed-orders \
  --state-value ALARM --state-reason "Testing the alert"

Finally, play detective. Who created this alarm? CloudTrail knows, though events can take a few minutes to appear:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutMetricAlarm \
  --max-results 1 \
  --query "Events[0].[EventName,Username,EventTime]" --output text

aws cloudwatch delete-alarms --alarm-names series-failed-orders
aws sns delete-topic --topic-arn $TOPIC_ARN

Summary #

• CloudWatch for metrics, logs and alarms, and memory needs the agent.
• Set a retention period on your logs, because they're kept forever by default.
• EventBridge routes events to whoever handles them, and runs things on a schedule.
• CloudTrail records who did what, and Config records the settings and judges them.
• Test your alert path before you need it.

Test yourself #

Before you move on, try these questions to check you've got this part. Each one is a practical scenario, and the explanation appears once you answer.

You now know what's happening in one account. But companies have dozens of accounts and hundreds of employees. In part twenty-six we manage them with Organizations and Identity Center.