ASSOCIATE-DATA-PRACTITIONER practice question 22 of 25

Your company is building a near real-time streaming pipeline to process JSON telemetry data from small appliances. You need to process messages arriving at a Pub/Sub topic, capitalize letters in the serial number field, and write results to BigQuery. You want to use a managed service and write a minimal amount of code for underlying transformations. What should you do?

Full set: 106 questions
Answer options
A: Use a Pub/Sub to BigQuery subscription, write results directly to BigQuery, and schedule a transformation query to run every five minutes.
B: Use a Pub/Sub to Cloud Storage subscription, write a Cloud Run service that is triggered when objects arrive in the bucket, performs the transformations, and writes the results to BigQuery.
C: Use the “Pub/Sub to BigQuery” Dataflow template with a UDF, and write the results to BigQuery. Correct answer
D: Use a Pub/Sub push subscription, write a Cloud Run service that accepts the messages, performs the transformations, and writes the results to BigQuery.
Correct Answer:
C. Use the “Pub/Sub to BigQuery” Dataflow template with a UDF, and write the results to BigQuery.
Explanation

The "Pub/Sub to BigQuery" Dataflow template is a managed, serverless solution designed for streaming data from Pub/Sub to BigQuery. This approach minimizes operational overhead. The template supports User-Defined Functions (UDFs) written in JavaScript, which allows for lightweight, inline transformations like capitalizing a field. This combination perfectly meets the requirements of using a managed service to process a stream in near real-time while writing a minimal amount of custom code for the transformation logic.

Why Incorrect

A. Using a scheduled query introduces latency (up to five minutes) and is not a near real-time streaming solution.
B. This architecture (Pub/Sub -> GCS -> Cloud Run) is overly complex, adds unnecessary latency, and requires more code than a Dataflow template.
D. While Cloud Run can work, it requires writing more boilerplate code for message handling and BigQuery API interaction compared to using a pre-built Dataflow template with a UDF.

References

1. Google Cloud. (2024). Pub/Sub to BigQuery template. Dataflow documentation. "The Pub/Sub to BigQuery template is a streaming pipeline that reads JSON-formatted messages from a Pub/Sub topic and writes them to a BigQuery table." The page also details the javascriptTextTransformGcsPath and javascriptTextTransformFunctionName parameters for applying a UDF. Retrieved from https://cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-bigquery

2. Google Cloud. (2024). Create user-defined functions for Dataflow templates. Dataflow documentation. This guide provides examples and instructions on how to write and apply a JavaScript UDF to a Dataflow template for custom transformations. Retrieved from https://cloud.google.com/dataflow/docs/guides/templates/create-udf

Scroll to Top