ASSOCIATE-DATA-PRACTITIONER practice question 14 of 25

You need to create a data pipeline that streams event information from applications in multiple Google Cloud regions into BigQuery for near real-time analysis. The data requires transformation before loading. You want to create the pipeline using a visual interface. What should you do?

Full set: 106 questions
Answer options
A: Push event information to a Pub/Sub topic. Create a Dataflow job using the Dataflow job builder. Correct answer
B: Push event information to a Pub/Sub topic. Create a Cloud Run function to subscribe to the Pub/Sub topic, apply transformations, and insert the data into BigQuery.
C: Push event information to a Pub/Sub topic. Create a BigQuery subscription in Pub/Sub.
D: Push event information to Cloud Storage, and create an external table in BigQuery. Create a BigQuery scheduled job that executes once each day to apply transformations.
Correct Answer:
A. Push event information to a Pub/Sub topic. Create a Dataflow job using the Dataflow job builder.
Explanation

This solution meets all requirements. Pub/Sub is a global, scalable messaging service ideal for ingesting streaming events from multiple regions. Dataflow provides a managed stream processing service for performing the necessary transformations. Crucially, Google Cloud provides pre-built Dataflow templates, such as "Pub/Sub Topic to BigQuery," which can be deployed and configured through a visual interface (the "Dataflow job builder" in the Google Cloud Console) without writing code. This combination provides a near real-time, transformational, and visually configured pipeline.

Why Incorrect

B. Using a Cloud Run function requires writing custom code and does not meet the requirement for creating the pipeline using a visual interface.
C. A BigQuery subscription for Pub/Sub is for simple streaming ingest with limited, SQL-based transformations, not the more complex transformations Dataflow handles.
D. This describes a batch processing pattern using Cloud Storage and scheduled jobs, which does not meet the "near real-time analysis" requirement.

References

1. Google Cloud Documentation, "Create a Dataflow job from a template," Dataflow. "Google Cloud provides a set of Dataflow templates that you can use to create jobs... You can run a template from the Google Cloud console, the gcloud command-line tool, or REST API calls." The console provides the visual interface.

2. Google Cloud Documentation, "Google-provided streaming templates," Dataflow. Lists the "Pub/Sub Topic to BigQuery" template, which "is a streaming pipeline that reads JSON-formatted messages from a Pub/Sub topic and writes them to a BigQuery table." It also supports a UDF for transformations.

3. Google Cloud Documentation, "Choose a data ingestion option," Analytics. This page compares different ingestion patterns, highlighting Pub/Sub and Dataflow for streaming ETL into BigQuery.

Scroll to Top