A company uses AWS Glue Apache Spark jobs to handle extract, transform, and load (ETL) workloads. The company has enabled logging and monitoring for…

The Apache Spark UI is the primary interface for monitoring and debugging the execution of Spark applications. When an AWS Glue job runs on the Spark engine, AWS Glue provides access to the Spark UI. This UI offers a granular, real-time, and post-run view of the job's execution, including a detailed breakdown of jobs, stages, and tasks. A data engineer can use the Spark UI to examine metrics for each individual stage, such as duration, data shuffled, input/output records, and execution details, which is essential for diagnosing failures or performance bottlenecks at a specific point in the ETL process.

1. AWS Glue Developer Guide: Under the section "Monitoring with the Apache Spark web UI

" the documentation states

"You can use the Apache Spark web UI to monitor and debug AWS Glue ETL jobs running on the Spark engine... The Spark UI provides information about... A list of scheduler stages and tasks... A summary of RDD sizes and memory usage...". This directly supports using the Spark UI to examine stage details.

2. AWS Glue Developer Guide: The section "Monitoring with Amazon CloudWatch" lists the available metrics

which are aggregated at the job level (e.g.

glue.driver.aggregate.numCompletedStages

glue.driver.aggregate.numCompletedTasks). This demonstrates that CloudWatch does not provide the granular

per-stage view needed.

3. AWS Glue Developer Guide: The section "AWS Glue job run insights" describes the feature as a way to "simplify job debugging and optimization" by providing "the probable root causes for a job failure and provides remediation steps." This confirms it is a diagnostic summary tool

not a tool for direct examination of stage metrics.

4. AWS CloudTrail User Guide: The section "Logging AWS Glue API Calls with AWS CloudTrail" clarifies that CloudTrail records actions taken by a user

role

or an AWS service in AWS Glue

such as CreateJob or StartJobRun

not the internal workings of the job itself.