A company uses AWS Glue Apache Spark jobs to handle extract, transform, and load (ETL) workloads. The company has enabled logging and monitoring for all AWS Glue jobs. One of the AWS Glue jobs begins to fail. A data engineer investigates the error and wants to examine metrics for all individual stages within the job. How can the data engineer access the stage metrics?

The Apache Spark web UI that AWS Glue exposes for every job run shows the DAG, stages, and task-level counters (records processed, bytes read/written, shuffle metrics, etc.). These detailed stage metrics are not forwarded to CloudWatch or CloudTrail, and Run Insights surfaces only summarized troubleshooting information. Therefore the data engineer must open the Spark UI for the failed job run to inspect stage-by-stage performance and errors.

B. CloudWatch publishes only high-level Glue job metrics (Succeeded, Failed, DPUUsed) and does not include per-stage Spark counters.
C. CloudTrail records control-plane API calls, not data-plane Spark execution metrics.
D. Run Insights highlights anomalous steps but does not expose the full set of Spark stage statistics.

1. AWS Glue Developer Guide

“Monitoring AWS Glue Using the Apache Spark UI”

¶2-4 (docs.aws.amazon.com/glue/latest/dg/monitor-spark-ui.html).

2. AWS Glue Developer Guide

“Glue job metrics published to Amazon CloudWatch”

¶1 (monitoring-overview.html).

3. AWS Glue Developer Guide

“AWS Glue Job Run Insights”

¶3-4 (job-run-insights.html).

4. AWS CloudTrail User Guide

“Logging AWS Glue API Calls with AWS CloudTrail”

Scope section (docs.aws.amazon.com)