ASSOCIATE-DATA-PRACTITIONER practice question 24 of 25

You work for an online retail company. Your company collects customer purchase data in CSV files and pushes them to Cloud Storage every 10 minutes…

Full set: 106 questions
You work for an online retail company. Your company collects customer purchase data in CSV files and pushes them to Cloud Storage every 10 minutes. The data needs to be transformed and loaded into BigQuery for analysis. The transformation involves cleaning the data, removing duplicates, and enriching it with product information from a separate table in BigQuery. You need to implement a low-overhead solution that initiates data processing as soon as the files are loaded into Cloud Storage. What should you do?
Answer options
A: Use Cloud Composer sensors to detect files loading in Cloud Storage. Create a Dataproc cluster, and use a Composer task to execute a job on the cluster to process and load the data into BigQuery.
B: Schedule a direct acyclic graph (DAG) in Cloud Composer to run hourly to batch load the data from Cloud Storage to BigQuery, and process the data in BigQuery using SQL.
C: Use Dataflow to implement a streaming pipeline using an OBJECT_FINALIZE notification from Pub/Sub to read the data from Cloud Storage, perform the transformations, and write the data to BigQuery. Correct answer
D: Create a Cloud Data Fusion job to process and load the data from Cloud Storage into BigQuery. Create an OBJECT_FINALI ZE notification in Pub/Sub, and trigger a Cloud Run function to start the Cloud Data Fusion job as soon as new files are loaded.
Correct Answer:
C. Use Dataflow to implement a streaming pipeline using an OBJECT_FINALIZE notification from Pub/Sub to read the data from Cloud Storage, perform the transformations, and write the data to BigQuery.
Explanation

This solution is event-driven, serverless, and low-overhead. Cloud Storage can publish an OBJECTFINALIZE event to a Pub/Sub topic whenever a new file is uploaded. A streaming Dataflow pipeline can subscribe to this topic, triggering the processing job immediately. Dataflow is designed for scalable data transformation, including cleaning, deduplication, and enrichment by joining with BigQuery data, before writing the final results to a BigQuery table. This approach avoids the overhead of managing clusters or waiting for a batch schedule, directly addressing the requirements for immediate processing and minimal management.

Why Incorrect

A. Creating a Dataproc cluster for each file arrival every 10 minutes is inefficient and has high startup overhead, contradicting the low-overhead requirement.
B. An hourly batch schedule does not meet the requirement to initiate processing "as soon as the files are loaded," which happens every 10 minutes.
D. While functional, this adds unnecessary complexity and potential latency by using Cloud Run as an intermediary to trigger a Cloud Data Fusion job.

References

1. Google Cloud Documentation, "Cloud Storage triggers". This page describes how to use Pub/Sub notifications for Cloud Storage events like OBJECTFINALIZE to trigger downstream processing. (cloud.google.com/functions/docs/calling/storage)

2. Google Cloud Documentation, "Streaming with Cloud Pub/Sub". This section details how services like Dataflow can subscribe to Pub/Sub topics to process data in real-time as events occur. (cloud.google.com/pubsub/docs/overview#integrations)

3. Google Cloud Documentation, "Common BigQuery data ingestion patterns". The "Stream processing with Dataflow" pattern is described for scenarios requiring real-time transformation and loading from sources like Cloud Storage. (cloud.google.com/bigquery/docs/ingestion-patterns)

Scroll to Top