If you’ve spent 20 minutes waiting for an EC2 instance to launch only to find that none of your software is installed, your configuration wasn’t applied, and there’s zero error message, you’ve hit the silent-failure trap of EC2 UserData. I’ve debugged this countless times in production deployments. In this post, I’ll show you how to diagnose and fix UserData scripts that refuse to run.

The Problem

You launch an EC2 instance with a UserData script, the instance enters “running” state, but your application code, web server, or database client that should have been installed during launch is completely absent. You SSH in and nothing is there. Here’s the maddening part: no error in the console, no failed status, just silence.

Error Type Description
Silent Failure Instance is running but UserData script never executed or silently failed
No Output EC2 console shows instance running; no indication UserData ran
Missing Software Expected packages/configs installed via UserData are absent

This blocks deployments because you’re left guessing whether the script ran, crashed, or was never executed at all.

Why Does This Happen?

  • Missing #!/bin/bash shebang on line 1 — Without the shebang, the OS doesn’t know the script is a bash script and simply tries to execute it as a binary. Silent failure.
  • Script errors silently — no error checking — The script runs but hits an error (missing package, syntax error, permission denied) and the error is logged but ignored by cloud-init because there’s no set -e flag.
  • cloud-init already ran on a restarted instance — UserData runs once by default when the instance first launches. If you stop and restart the instance, UserData does not run again unless you explicitly reset cloud-init.
  • Wrong encoding when passing via console — If you copy-paste into the EC2 console UserData field without proper encoding or base64 wrapping, special characters may be corrupted.

The Fix

The first step is to check the logs. Never assume UserData ran without verifying.

# SSH to the instance and check cloud-init logs
tail -f /var/log/cloud-init-output.log

This shows the full execution output of UserData. Look for error messages, missing packages, or permission errors. For more detail:

# Check the main cloud-init log
tail -f /var/log/cloud-init.log

If nothing is there, it means UserData never ran. Verify it was actually passed to the instance:

# Check if UserData was uploaded to the instance
aws ec2 describe-instance-attribute \
  --instance-id i-0abc123def456ghij \
  --attribute userData \
  --region us-east-1 \
  --output text

If this returns empty, the script wasn’t sent. If it returns base64, decode it:

# Decode the UserData to verify what was actually sent
aws ec2 describe-instance-attribute \
  --instance-id i-0abc123def456ghij \
  --attribute userData \
  --region us-east-1 \
  --query 'UserData.Value' \
  --output text | base64 -d

Compare the decoded output with your original script. Look for missing shebang, syntax errors, or truncation.

How to Fix Common Issues

Issue: Missing #!/bin/bash shebang

Your script must start with:

#!/bin/bash
set -e  # Exit on first error
set -x  # Print each command before execution

# Rest of your script
apt-get update
apt-get install -y nginx
systemctl start nginx

The set -e and set -x flags are critical — they ensure errors don’t silently propagate and you can see exactly which line failed.

Issue: Instance launched before, UserData won’t re-run

If you stopped and restarted an instance, UserData won’t execute again. To force re-execution:

# SSH to the instance
ssh -i key.pem ec2-user@<public-ip>

# Reset cloud-init (THIS WILL RE-RUN ALL USERDATA)
sudo cloud-init clean --all

# Reboot to trigger re-run
sudo reboot

Only do this if your UserData is idempotent (safe to run multiple times). Otherwise, you risk installing packages twice or applying configurations twice.

Issue: Script syntax error or missing dependency

Use the console output log to identify the exact line that failed. Common culprits:

# Wrong — will fail silently if nginx package doesn't exist
apt-get install nginx

# Correct — will error loudly and stop
apt-get install -y nginx || exit 1

Issue: Wrong encoding in EC2 console

Always use plain text, not copy-paste with special characters. Or use the AWS CLI:

aws ec2 run-instances \
  --image-id ami-0abc123def \
  --instance-type t3.micro \
  --user-data file:///path/to/userdata.sh \
  --region us-east-1

Using file:// ensures proper encoding.

How to Run This

  1. SSH into the instance: ssh -i key.pem ec2-user@<public-ip>
  2. Check /var/log/cloud-init-output.log for the script execution output and any error messages.
  3. Run aws ec2 describe-instance-attribute --instance-id i-xxx --attribute userData --output text | base64 -d to verify the script was sent correctly.
  4. If the script has errors, fix them, then create a new instance with the corrected UserData.
  5. If you need to re-run UserData on an existing instance (only if idempotent), run sudo cloud-init clean --all && sudo reboot.

Is This Safe?

Reading logs is completely safe and non-destructive. Re-running UserData with cloud-init clean causes an instance reboot, which means brief downtime. Only do this if your UserData script is idempotent and won’t break if run twice.

Key Takeaway

EC2 UserData failures are silent by default. Always check /var/log/cloud-init-output.log after launch to verify execution. Include #!/bin/bash and set -e in every script, use --user-data file:// in the CLI to ensure proper encoding, and test your UserData locally before deploying to production.

Have questions or debugging a stubborn UserData issue? Connect with me on LinkedIn or X.