ASSOCIATE-DATA-PRACTITIONER practice question 1 of 25

You are designing a pipeline to process data files that arrive in Cloud Storage by 3:00 am each day. Data processing is performed in stages, where…

Full set: 106 questions
You are designing a pipeline to process data files that arrive in Cloud Storage by 3:00 am each day. Data processing is performed in stages, where the output of one stage becomes the input of the next. Each stage takes a long time to run. Occasionally a stage fails, and you have to address the problem. You need to ensure that the final output is generated as quickly as possible. What should you do?
Answer options
A: Design a Spark program that runs under Dataproc. Code the program to wait for user input when an error is detected. Rerun the last action after correcting any stage output data errors.
B: Design the pipeline as a set of PTransforms in Dataflow. Restart the pipeline after correcting any stage output data errors.
C: Design the workflow as a Cloud Workflow instance. Code the workflow to jump to a given stage based on an input parameter. Rerun the workflow after correcting any stage output data errors.
D: Design the processing as a directed acyclic graph (DAG) in Cloud Composer. Clear the state of the failed task after correcting any stage output data errors. Correct answer
Correct Answer:
D. Design the processing as a directed acyclic graph (DAG) in Cloud Composer. Clear the state of the failed task after correcting any stage output data errors.
Explanation

Cloud Composer, which is a managed Apache Airflow service, is designed for orchestrating complex workflows. The pipeline can be modeled as a directed acyclic graph (DAG), where each stage is a task. When a task (stage) fails, Airflow's UI allows an operator to inspect the failure, correct the underlying data or code issue, and then "clear" the state of only the failed task. The Airflow scheduler will then re-run just that task and any of its downstream dependencies, not the entire pipeline. This approach is the most efficient for recovering from failures in long-running, multi-stage processes, minimizing the time to final output.

Why Incorrect

A. A Spark program waiting for user input is an anti-pattern for automated pipelines and is not a robust or scalable error-handling mechanism.
B. Restarting the entire Dataflow pipeline from the beginning is inefficient and violates the requirement to generate the final output as quickly as possible.
C. While Cloud Workflows can orchestrate tasks, its state management and retry logic are less intuitive for this use case than Airflow's task-centric model.

References

1. Google Cloud Documentation, Cloud Composer, "Executing DAGs". It states, "Airflow represents a workflow as a Directed Acyclic Graph (DAG) of tasks. ... You can also clear task runs for a past or future DAG run. This is useful when tasks fail and you want to retry them after fixing a bug." (Section: "Triggering DAGs")

2. Google Cloud Documentation, Cloud Composer, "Managing DAGs". The documentation details using the Airflow UI to "clear" a task, which deletes the task instance's state and forces it to re-run. (Section: "Viewing task instances")

Scroll to Top