A company has a data processing pipeline that includes several dozen steps. The data processing pipeline needs to send alerts in real time when a…

The most direct and efficient solution is to use Amazon EventBridge. AWS Step Functions natively emits events to EventBridge whenever an execution's status changes (e.g., SUCCEEDED, FAILED, RUNNING). An EventBridge rule can be configured to specifically filter for these status change events from the target state machine. This rule can then invoke a target, such as an Amazon SNS topic, to send notifications in real time. This approach provides a decoupled, event-driven mechanism for monitoring the entire pipeline's orchestration layer without modifying the pipeline's core logic.

1. AWS Step Functions Developer Guide: The guide explicitly details this pattern. It states

"Amazon EventBridge (previously called Amazon CloudWatch Events) can initiate rules when the status of an execution changes... Step Functions sends events to EventBridge for all state transition and execution status changes."

Source: AWS Step Functions Developer Guide

"Monitoring Step Functions Using EventBridge".

2. Amazon EventBridge User Guide: This guide confirms that Step Functions is a native event source for EventBridge and provides the event pattern for matching execution status. The example for a "Step Functions Execution Status Change" event includes a detail-type and a status field (SUCCEEDED

FAILED

etc.) that can be used in a rule.

Source: Amazon EventBridge User Guide

"Events from AWS services"

section on AWS Step Functions events.

3. AWS CloudTrail User Guide: This document clarifies CloudTrail's purpose: "CloudTrail provides a record of actions taken by a user

role

or an AWS service... CloudTrail is best-suited for audit and compliance use cases." This differentiates it from EventBridge's real-time event-driven purpose.

Source: AWS CloudTrail User Guide

"What Is AWS CloudTrail?".

4. AWS Well-Architected Framework

Serverless Applications Lens: This document promotes using services like EventBridge for inter-service communication and eventing. "Use an event-driven architecture to decouple components... Amazon EventBridge provides a serverless event bus that simplifies building event-driven architectures." This supports the design pattern in option D.

Source: AWS Well-Architected Framework

Serverless Applications Lens (latest version)

"REL 11: How do you design your serverless application to be resilient?".