An order fulfillment workflow ran fine in every test and then started silently stalling in production — executions would sit in RUNNING for exactly an hour before failing with States.Timeout, with no error from the Lambda function it called. The function itself completed in two seconds every time. The state machine’s TimeoutSeconds had been left at its default for a state that used a .waitForTaskToken integration, and the callback token was never being returned by a downstream system that had quietly stopped calling back weeks earlier. Step Functions was reporting the truth — it just took an hour to say it.

Step Functions failures usually point at a specific state and a specific error, but reading past the top-level “Execution Failed” banner is where the actual cause lives.

The Problem

Executions fail, hang, or behave differently than the state machine definition implies:

Symptom	What It Means
States.Timeout on a specific state	TimeoutSeconds too short for the task, or a .waitForTaskToken callback never arrives
States.TaskFailed with a Lambda error	The Lambda function itself threw, but Step Functions surfaces it as a task failure
Retry never happens despite a Retry block	Error type in the Retry field doesn’t match the actual error name thrown
Execution completes but skips expected branches	Choice state condition doesn’t match the actual input shape (type mismatch, missing key)
Map state processes fewer items than expected	MaxConcurrency or ItemsPath misconfigured, or item-level failures tolerated silently by ToleratedFailurePercentage

Every one of these looks like “the workflow is broken,” but the state machine is doing exactly what its definition says — the definition just doesn’t match what you assumed.

Why Does This Happen?

• TimeoutSeconds too short, or missing on a callback state: States default to a very long implicit timeout unless set, but if you did set one, it applies to the entire task, not just the compute — including any wait for an external callback token that never arrives.
• Retry error matching is exact-string: The ErrorEquals field must match the actual error name thrown (e.g., Lambda.ServiceException, States.TaskFailed, or a custom error name) — a typo or overly narrow match means the retry block silently never triggers, and the execution fails on the first error.
• Catch blocks not covering States.ALL: If a Catch only lists specific error types and an unanticipated error occurs, it propagates unhandled and fails the execution instead of routing to your fallback state.
• ResultPath and InputPath reshaping data unexpectedly: Step Functions’ JSON path processing between states is a common silent bug source — a task’s output can overwrite the entire state input if ResultPath isn’t set, breaking a later Choice state that expected the original input to still be present.
• Map state concurrency and tolerance defaults: MaxConcurrency: 0 means unlimited, which can overwhelm a downstream Lambda’s concurrency limits and cause throttling that looks like “the Map state doesn’t process everything.” Combined with a nonzero ToleratedFailurePercentage, individual item failures can be swallowed without visibility.
• IAM role missing permission for the specific service integration: Step Functions’ execution role needs distinct permissions for .sync and .waitForTaskToken integration patterns beyond the base service action — missing ones fail immediately with an access denied that’s easy to mistake for a task-level bug.

The Fix

Step 1: Get the Execution History, Not Just the Top-Level Status

aws stepfunctions get-execution-history \
  --execution-arn arn:aws:states:us-east-1:111122223333:execution:orders-workflow:abc123 \
  --query "events[?type=='ExecutionFailed' || type=='TaskFailed']" \
  --output json

This shows the exact state, the exact error name, and the exact cause message — not just “execution failed.”

Step 2: Check the Failing State’s Definition

aws stepfunctions describe-state-machine \
  --state-machine-arn arn:aws:states:us-east-1:111122223333:stateMachine:orders-workflow \
  --query "definition" \
  --output text | python3 -m json.tool | grep -A 5 "FailingStateName"

Compare TimeoutSeconds, Retry, and Catch against what actually happened in the execution history.

Step 3: Verify Retry Error Names Match Exactly

"Retry": [
  {
    "ErrorEquals": ["Lambda.ServiceException", "Lambda.TooManyRequestsException"],
    "IntervalSeconds": 2,
    "MaxAttempts": 3,
    "BackoffRate": 2.0
  }
]

Cross-check against the actual error name from Step 1’s output — if the execution shows States.TaskFailed but your retry lists only Lambda.ServiceException, add States.ALL as a catch-all fallback retry entry, or the specific error string that actually occurred.

Step 4: Add a Catch-All to Route Unexpected Errors Somewhere Visible

"Catch": [
  {
    "ErrorEquals": ["States.ALL"],
    "ResultPath": "$.error",
    "Next": "NotifyFailure"
  }
]

This guarantees unexpected errors land in a known failure path instead of just failing the execution with no downstream signal.

Step 5: Trace JSON Path Transformations Between States

aws stepfunctions get-execution-history \
  --execution-arn arn:aws:states:us-east-1:111122223333:execution:orders-workflow:abc123 \
  --query "events[?type=='TaskStateExited'].stateExitedEventDetails.output" \
  --output json

Compare the output of each state to the input the next state received. If a Choice state is skipping an expected branch, the input shape it’s evaluating is usually not what you assumed — check for a missing ResultPath that overwrote the original input.

Step 6: Check Map State Concurrency Against Downstream Limits

aws lambda get-function-concurrency \
  --function-name process-order-item \
  --query "ReservedConcurrentExecutions"

Set MaxConcurrency on the Map state at or below the downstream function’s available concurrency:

"MaxConcurrency": 10,
"ToleratedFailurePercentage": 0

Setting ToleratedFailurePercentage to 0 during debugging ensures any item failure surfaces instead of being silently absorbed.

Step 7: Confirm IAM Permissions for the Integration Pattern

For .sync integrations (e.g., running an ECS task and waiting for completion), the execution role needs event rule permissions in addition to the base action:

aws iam get-role-policy \
  --role-name orders-workflow-execution-role \
  --policy-name step-functions-integration-policy

Should include events:PutTargets, events:PutRule, and events:DescribeRule for .sync patterns, and states:SendTaskSuccess / states:SendTaskFailure permissions granted to whatever service is meant to call back for .waitForTaskToken patterns.

Is This Safe?

Yes — get-execution-history and describe-state-machine are read-only. Updating the state machine definition to fix Retry, Catch, or TimeoutSeconds values only affects executions started after the update; in-flight executions keep running under the definition version they started with, so a fix doesn’t retroactively rescue a currently stuck execution. Adjusting MaxConcurrency takes effect on the next Map state execution.

Key Takeaway

Step Functions failures are almost always readable in plain English from get-execution-history — the tool tells you exactly which state failed and why. Most issues trace back to a Retry block that doesn’t match the actual error name, a Catch that doesn’t cover States.ALL, or JSON path transformations silently reshaping data between states. Read the execution history before assuming anything about the failure — it’s rarely a mystery once you look.

---

Have questions or ran into a different Step Functions issue? Connect with me on LinkedIn or X.