I configured Secrets Manager to automatically rotate a database password every 30 days. For two months, everything worked perfectly. Then one rotation failed. Then the next one. CloudWatch logs showed cryptic Lambda errors, and the secret stayed stale. The biggest challenge? Figuring out whether the issue was permissions, networking, or the rotation function itself. In this post, I’ll walk through exactly what causes this and how to fix it.

The Problem

Secrets Manager automatic rotation fails. The secret shows “rotation failed” status in the console. The Lambda function that rotates the secret isn’t being invoked successfully, or it’s failing midway through the rotation process. The database password or API key is no longer rotating as scheduled.

Error Message	Cause
Lambda function invocation failed	Lambda execution role missing secretsmanager:GetSecretValue or similar
Unable to reach Secrets Manager API	Lambda in VPC can’t reach secretsmanager.REGION.amazonaws.com
AccessDenied when calling testSecret step	Database credentials incorrect or password policy rejection
Timed out during rotation	Lambda timeout too short or Lambda in private subnet with no NAT gateway

Why Does This Happen?

• Lambda in VPC can’t reach Secrets Manager API — A Lambda deployed in a VPC doesn’t have internet access by default. If it can’t reach secretsmanager.REGION.amazonaws.com, the rotation fails silently. It needs either a NAT gateway or a VPC endpoint.

• Lambda execution role missing permissions — The role needs secretsmanager:GetSecretValue, secretsmanager:DescribeSecret, secretsmanager:PutSecretValue, secretsmanager:UpdateSecretVersionStage, and kms:Decrypt (if encrypted).

• New password fails database policy — The rotation function generates a password that doesn’t meet the database’s password complexity requirements, and testSecret step fails.

• Lambda rotation function has a logic bug — The rotation function must complete four steps: createSecret, setSecret, testSecret, finishSecret. If any step fails, the entire rotation fails and the secret is left in RotationInProgress state.

• Resource policy on secret blocks Lambda — The secret’s resource policy (if one exists) doesn’t allow the Lambda execution role to update it.

The Fix

First, check the rotation status:

# Describe the secret and check rotation configuration
aws secretsmanager describe-secret --secret-id my-db-password \
  --query 'RotationEnabled,RotationRules,LastRotatedDate' \
  --output table

Check the Lambda logs to understand what failed:

# Retrieve Lambda logs from CloudWatch
aws logs tail /aws/lambda/SecretsManagerRotation-my-db-password --follow

Look for errors in the output. Common failures appear as:

[ERROR] AccessDeniedException: User is not authorized to perform: secretsmanager:GetSecretValue
[ERROR] Unable to connect to Secrets Manager endpoint
[ERROR] Unable to set secret value: Password does not meet policy requirements

Fix 1: Add Missing VPC Endpoint

If the error is “Unable to reach Secrets Manager API,” the Lambda needs VPC endpoint access:

# Create a VPC endpoint for Secrets Manager
aws ec2 create-vpc-endpoint \
  --service-name com.amazonaws.us-east-1.secretsmanager \
  --vpc-id vpc-12345678 \
  --subnet-ids subnet-abcd1234 subnet-efgh5678 \
  --security-group-ids sg-12345678

Fix 2: Update Lambda Execution Role Permissions

# Get the current role
ROLE_NAME=$(aws lambda get-function --function-name SecretsManagerRotation-my-db-password \
  --query 'Configuration.Role' --output text | awk -F'/' '{print $NF}')

# Create an inline policy with required permissions
aws iam put-role-policy --role-name "$ROLE_NAME" --policy-name SecretsManagerRotation \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": [
          "secretsmanager:DescribeSecret",
          "secretsmanager:GetSecretValue",
          "secretsmanager:PutSecretValue",
          "secretsmanager:UpdateSecretVersionStage"
        ],
        "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:my-db-password-*"
      },
      {
        "Effect": "Allow",
        "Action": [
          "kms:Decrypt",
          "kms:GenerateDataKey"
        ],
        "Resource": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
      }
    ]
  }'

Fix 3: Manually Test the Rotation

# Trigger rotation immediately to see real-time error messages
aws secretsmanager rotate-secret --secret-id my-db-password \
  --rotate-immediately

# Wait a few seconds, then check status
aws secretsmanager describe-secret --secret-id my-db-password \
  --query 'RotationEnabled,LastRotationAttempted' --output table

Fix 4: Increase Lambda Timeout

If the rotation times out mid-process, increase the Lambda timeout:

# Update Lambda timeout to 5 minutes (300 seconds)
aws lambda update-function-configuration --function-name SecretsManagerRotation-my-db-password \
  --timeout 300

Fix 5: Review and Fix Database Password Policy

If the testSecret step fails because the password doesn’t meet database policy, review the rotation function. For RDS MySQL, the password generator should use:

# Example: Generate compliant password
python3 << 'EOF'
import secrets
import string

# RDS MySQL: 8-41 chars, alphanumeric + special chars (avoid /, \, ", ')
chars = string.ascii_letters + string.digits + "!@#$%^&*"
password = ''.join(secrets.choice(chars) for _ in range(32))
print(password)
EOF

Is This Safe?

Yes. Automatic rotation is a security best practice. These fixes align with AWS security recommendations for secret rotation Lambda functions.

Key Takeaway

Secrets Manager rotation failures usually stem from three issues: VPC/network connectivity, missing IAM permissions, or database policy mismatches. Always check CloudWatch logs first, then verify the Lambda has VPC endpoint access and correct permissions.

---

Have questions or ran into a different security issue? Connect with me on LinkedIn or X.