ASSOCIATE-DATA-PRACTITIONER practice question 1 of 25
You are designing a pipeline to process data files that arrive in Cloud Storage by 3:00 am each day. Data processing is performed in stages, where…
Full set: 106 questionsCloud Composer, which is a managed Apache Airflow service, is designed for orchestrating complex workflows. The pipeline can be modeled as a directed acyclic graph (DAG), where each stage is a task. When a task (stage) fails, Airflow's UI allows an operator to inspect the failure, correct the underlying data or code issue, and then "clear" the state of only the failed task. The Airflow scheduler will then re-run just that task and any of its downstream dependencies, not the entire pipeline. This approach is the most efficient for recovering from failures in long-running, multi-stage processes, minimizing the time to final output.
A. A Spark program waiting for user input is an anti-pattern for automated pipelines and is not a robust or scalable error-handling mechanism.
B. Restarting the entire Dataflow pipeline from the beginning is inefficient and violates the requirement to generate the final output as quickly as possible.
C. While Cloud Workflows can orchestrate tasks, its state management and retry logic are less intuitive for this use case than Airflow's task-centric model.
1. Google Cloud Documentation, Cloud Composer, "Executing DAGs". It states, "Airflow represents a workflow as a Directed Acyclic Graph (DAG) of tasks. ... You can also clear task runs for a past or future DAG run. This is useful when tasks fail and you want to retry them after fixing a bug." (Section: "Triggering DAGs")
2. Google Cloud Documentation, Cloud Composer, "Managing DAGs". The documentation details using the Airflow UI to "clear" a task, which deletes the task instance's state and forces it to re-run. (Section: "Viewing task instances")
Keep practicing
Take the free ASSOCIATE-DATA-PRACTITIONER practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other ASSOCIATE-DATA-PRACTITIONER practice questions
- Question 2Your organization plans to move their on-premises environment to Google Cloud. Your organ…
- Question 3You are responsible for managing Cloud Storage buckets for a research company. Your compa…
- Question 4You are predicting customer churn for a subscription-based service. You have a 50 PB hist…
- Question 5Your retail company collects customer data from various sources:Online transactions: Stor…
- Question 6Your company has developed a website that allows users to upload and share video files. T…
- Question 8Another team in your organization is requesting access to a BigQuery dataset. You need to…
