PROFESSIONAL-DATA-ENGINEER practice question 1 of 10

You orchestrate ETL pipelines by using Cloud Composer One of the tasks in the Apache Airflow directed acyclic graph (DAG) relies on a third-party service. You want to be notified when the task does not succeed. What should you do?

Answer options

A: Configure a Cloud Monitoring alert on the sla_missed metric associated with the task at risk to trigger a notification.

B: Assign a function with notification logic to the sla_miss_callback parameter for the operator responsible for the task at risk.

C: Assign a function with notification logic to the on_retry_callback parameter for the operator responsible for the task at risk.

D: Assign a function with notification logic to the on_failure_callback parameter for the operator responsible for the task at risk. Correct answer

Correct Answer:

D. Assign a function with notification logic to the on_failure_callback parameter for the operator responsible for the task at risk.

Explanation

The onfailurecallback parameter is a standard feature of Apache Airflow operators. It allows you to specify a Python callable (a function) that will be executed only when a task instance fails. This is the most direct and appropriate mechanism to trigger a custom action, such as sending a notification, in response to a task failure. By assigning a function with notification logic to this parameter for the specific task, you ensure that you are alerted precisely when that task does not succeed after all configured retries have been exhausted.

Why Incorrect

A. The slamissed metric is triggered when a task does not complete within its defined Service Level Agreement (SLA) timeframe. A task can miss its SLA and still succeed, so this does not specifically track failures.
B. The slamisscallback function is executed when a task misses its SLA. This is related to the timeliness of task completion, not its success or failure status.
C. The onretrycallback function is executed each time a task is retried. A task might retry and then ultimately succeed, so this callback does not signify a final failure.

References

1. Apache Airflow Documentation, BaseOperator API Reference: The official documentation for Airflow's BaseOperator (the parent class for all operators) defines the callback parameters.

onfailurecallback: "a function to be called when a task instance of this task fails."

onretrycallback: "a function to be called when a task instance of this task is retried."

slamisscallback: "function to be called when a task instance of this task misses its SLA."

Source: Apache Airflow Documentation, apacheairflow.models.baseoperator.py.

2. Google Cloud Documentation, Writing DAGs (Airflow): Google's documentation for Cloud Composer explicitly directs users to the core Apache Airflow concepts for authoring DAGs, including the use of callbacks for notifications. It states, "Airflow provides several ways to set up notifications... A common way is to use Airflow's built-in callback mechanism."

Source: Google Cloud, Cloud Composer Documentation, "Writing DAGs (Airflow)", section on "Sending email notifications".

3. Google Cloud Documentation, Monitoring environments: This page lists the Airflow metrics exported to Cloud Monitoring. It includes airflow/dag/task/slamisses, confirming that SLA misses are tracked as a distinct metric from general task failures.

Source: Google Cloud, Cloud Composer Documentation, "Key metrics for monitoring environments", section "Airflow metrics".

Keep practicing

Take the free PROFESSIONAL-DATA-ENGINEER practice test

Ten exam-style questions with answers and explanations, plus the exam facts and study guides.

Start the free practice test