Last month, a production database hit 95% storage utilization and the CloudWatch alarm fired exactly as expected — I could see it sitting in ALARM state right there in the console. But nobody on the team got an email. No SMS. No Slack message. The alarm was screaming into the void. We caught the issue by luck, not by process. I’ve since debugged this exact scenario more times than I’d like to admit, and the root causes are almost always the same handful of misconfigurations. In this post, I’ll walk you through every one of them and show you how to fix each.

The Problem

Your CloudWatch alarm transitions to ALARM state, but the expected SNS notification never arrives. You check the alarm in the console and see it’s clearly in alarm. You check your email — nothing. You check your SMS — nothing. The alarm is working, but the notification pipeline is broken somewhere between CloudWatch and your inbox.

Failure Type Description
No alarm action configured Alarm has no SNS topic ARN in its action list
SNS subscription not confirmed Email subscription stuck in PendingConfirmation
Wrong alarm state action Action configured for OK instead of ALARM
IAM permission denied CloudWatch lacks permission to publish to SNS
Insufficient data handling Alarm stuck in INSUFFICIENT_DATA, never reaches ALARM
Evaluation period too long Alarm takes too many periods to trigger, masking real issues
SNS topic deleted or wrong region Topic ARN points to a non-existent or cross-region topic

Why Does This Happen?

  • No alarm action is configured — This is the most common cause. You created the alarm and set the threshold, but forgot to attach an SNS topic ARN to the alarm’s AlarmActions list. The alarm transitions to ALARM state correctly, but there’s no action to execute, so nothing happens.

  • SNS subscription is not confirmed — When you subscribe an email endpoint to an SNS topic, AWS sends a confirmation email. Until you click the confirmation link, the subscription sits in PendingConfirmation and no messages are delivered. This email often lands in spam folders or gets ignored.

  • Alarm action is set on the wrong state transition — CloudWatch alarms support three action types: AlarmActions (fires when state goes to ALARM), OKActions (fires when state returns to OK), and InsufficientDataActions. If you accidentally configured the SNS topic under OKActions instead of AlarmActions, you’ll get notified when the problem resolves but not when it starts.

  • IAM permissions are missing or insufficient — If the SNS topic has an access policy that restricts who can publish to it, CloudWatch may be blocked from sending the notification. This is especially common with encrypted SNS topics where CloudWatch needs kms:GenerateDataKey and kms:Decrypt permissions on the KMS key.

  • Alarm is stuck in INSUFFICIENT_DATA — A new alarm starts in INSUFFICIENT_DATA state. If the metric you’re monitoring isn’t reporting data (for example, a stopped EC2 instance doesn’t emit CPUUtilization), the alarm never transitions to ALARM and no notification fires. You see the alarm in the console and assume it’s monitoring, but it’s actually idle.

  • Evaluation period and datapoints configuration is too aggressive — If you configured the alarm to require 5 out of 5 datapoints breaching the threshold over a 5-minute period, a brief spike that only lasts 3 minutes won’t trigger it. The alarm may flicker but never fully transition to ALARM state.

  • SNS topic was deleted or is in a different region — The alarm action points to an SNS topic ARN that no longer exists, or the topic is in us-west-2 while the alarm is in us-east-1. CloudWatch silently fails to publish when the topic ARN is invalid.

The Fix

Work through these steps in order. Most issues are caught in the first three steps.

Step 1: Verify the Alarm Has an Action Configured

Check whether the alarm has an SNS topic attached to its AlarmActions:

aws cloudwatch describe-alarms \
  --alarm-names "MyDatabaseStorageAlarm" \
  --region us-east-1 \
  --query 'MetricAlarms[0].{State:StateValue,AlarmActions:AlarmActions,OKActions:OKActions,InsufficientDataActions:InsufficientDataActions}' \
  --output json

If AlarmActions is an empty list [], that’s your problem. The alarm fires but has nowhere to send the notification. Add the SNS topic:

aws cloudwatch put-metric-alarm \
  --alarm-name "MyDatabaseStorageAlarm" \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:MyAlertsTopic \
  --evaluation-periods 3 \
  --comparison-operator GreaterThanThreshold \
  --threshold 90 \
  --metric-name DatabaseConnections \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300

Note: put-metric-alarm is an upsert — it updates the existing alarm if one with that name already exists.

Step 2: Verify the SNS Topic Exists and Has Subscriptions

Confirm the topic is real and has active subscribers:

# Check the topic exists
aws sns get-topic-attributes \
  --topic-arn arn:aws:sns:us-east-1:123456789012:MyAlertsTopic \
  --region us-east-1

# List subscriptions on the topic
aws sns list-subscriptions-by-topic \
  --topic-arn arn:aws:sns:us-east-1:123456789012:MyAlertsTopic \
  --region us-east-1 \
  --output table

If get-topic-attributes returns a NotFound error, the topic was deleted. Recreate it:

aws sns create-topic \
  --name MyAlertsTopic \
  --region us-east-1

Then update the alarm to point to the new topic ARN (the ARN will be the same if you use the same name and account).

Step 3: Check SNS Subscription Status

Look at each subscription’s SubscriptionArn in the output from Step 2:

  • If it shows PendingConfirmation instead of a full ARN, the email subscriber never clicked the confirmation link.
  • If the subscription is simply missing, no one is subscribed.

To re-subscribe an email endpoint and trigger a new confirmation email:

aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:MyAlertsTopic \
  --protocol email \
  --notification-endpoint oncall-team@company.com \
  --region us-east-1

Check your inbox (and spam folder) for the AWS SNS confirmation email and click the link. Until you do, no notifications will be delivered.

For SMS:

aws sns subscribe \
  --topic-arn arn:aws:sns:us-east-1:123456789012:MyAlertsTopic \
  --protocol sms \
  --notification-endpoint +12025551234 \
  --region us-east-1

SMS subscriptions don’t require confirmation — they’re active immediately.

Step 4: Verify the SNS Topic Access Policy Allows CloudWatch

If the SNS topic has a restrictive access policy, CloudWatch may not be able to publish to it. Check the topic’s policy:

aws sns get-topic-attributes \
  --topic-arn arn:aws:sns:us-east-1:123456789012:MyAlertsTopic \
  --query 'Attributes.Policy' \
  --output text \
  --region us-east-1 | python3 -m json.tool

The policy must allow the cloudwatch.amazonaws.com service principal to call SNS:Publish. If it doesn’t, set the correct policy:

aws sns set-topic-attributes \
  --topic-arn arn:aws:sns:us-east-1:123456789012:MyAlertsTopic \
  --attribute-name Policy \
  --attribute-value '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AllowCloudWatchPublish",
        "Effect": "Allow",
        "Principal": {
          "Service": "cloudwatch.amazonaws.com"
        },
        "Action": "SNS:Publish",
        "Resource": "arn:aws:sns:us-east-1:123456789012:MyAlertsTopic",
        "Condition": {
          "ArnLike": {
            "aws:SourceArn": "arn:aws:cloudwatch:us-east-1:123456789012:alarm:*"
          }
        }
      }
    ]
  }' \
  --region us-east-1

If the topic is encrypted with a KMS key, the CloudWatch service also needs KMS permissions. Add this to the KMS key policy:

aws kms get-key-policy \
  --key-id alias/my-sns-key \
  --policy-name default \
  --output text \
  --region us-east-1 | python3 -m json.tool

The key policy must include a statement allowing cloudwatch.amazonaws.com to call kms:GenerateDataKey and kms:Decrypt.

Step 5: Check the Alarm State and Evaluation Configuration

Verify the alarm isn’t stuck in INSUFFICIENT_DATA:

aws cloudwatch describe-alarms \
  --alarm-names "MyDatabaseStorageAlarm" \
  --region us-east-1 \
  --query 'MetricAlarms[0].{State:StateValue,Threshold:Threshold,Period:Period,EvalPeriods:EvaluationPeriods,DatapointsToAlarm:DatapointsToAlarm,TreatMissingData:TreatMissingData}' \
  --output table

If the state is INSUFFICIENT_DATA, the metric isn’t reporting. Common causes:

  • The EC2 instance is stopped (no CPUUtilization data).
  • Detailed monitoring is not enabled (metrics report every 5 minutes, not 1 minute).
  • The metric name or namespace is misspelled.

If you want the alarm to treat missing data as breaching (so it fires even when data stops), update the TreatMissingData setting:

aws cloudwatch put-metric-alarm \
  --alarm-name "MyDatabaseStorageAlarm" \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:us-east-1:123456789012:MyAlertsTopic \
  --evaluation-periods 3 \
  --datapoints-to-alarm 2 \
  --comparison-operator GreaterThanThreshold \
  --threshold 90 \
  --metric-name DatabaseConnections \
  --namespace AWS/RDS \
  --statistic Average \
  --period 300

Setting --datapoints-to-alarm 2 with --evaluation-periods 3 means the alarm fires if 2 out of 3 consecutive periods breach the threshold, making it more forgiving of brief data gaps.

Step 6: Test the Full Notification Pipeline

Send a test message directly to the SNS topic to confirm the subscription and delivery are working:

aws sns publish \
  --topic-arn arn:aws:sns:us-east-1:123456789012:MyAlertsTopic \
  --message "Test notification from CloudWatch alarm troubleshooting" \
  --subject "Test Alert" \
  --region us-east-1

If you receive this test message, the SNS side is working and the problem is between CloudWatch and SNS (go back to Steps 1 and 4). If you don’t receive it, the problem is on the SNS subscription side (go back to Steps 2 and 3).

Then force the alarm into ALARM state to test the full end-to-end pipeline:

aws cloudwatch set-alarm-state \
  --alarm-name "MyDatabaseStorageAlarm" \
  --state-value ALARM \
  --state-reason "Testing notification pipeline" \
  --region us-east-1

This manually transitions the alarm to ALARM and triggers any configured AlarmActions. If you receive the notification, the full pipeline is restored.

Is This Safe?

Yes. All of these commands are read-safe or additive. Describing alarms and listing subscriptions are read-only. Subscribing an email endpoint triggers a confirmation email but doesn’t affect existing subscriptions. Setting the alarm state manually is temporary — the alarm will re-evaluate on the next period and return to its actual state based on metric data. The only change that requires caution is modifying the SNS topic access policy — always verify the policy JSON before applying it.

Key Takeaway

When a CloudWatch alarm is in ALARM state but no notification arrives, the problem is almost never the alarm itself. It’s the wiring between the alarm and your inbox: a missing alarm action, an unconfirmed SNS subscription, a restrictive topic policy, or a deleted topic. Use describe-alarms to check the action, list-subscriptions-by-topic to check the subscriber, and sns publish to test delivery. Fix the gap, and the notifications flow.

Have questions or ran into a different issue? Connect with me on LinkedIn or X.