ASSOCIATE-DATA-PRACTITIONER practice question 24 of 25
You work for an online retail company. Your company collects customer purchase data in CSV files and pushes them to Cloud Storage every 10 minutes…
Full set: 106 questionsThis solution is event-driven, serverless, and low-overhead. Cloud Storage can publish an OBJECTFINALIZE event to a Pub/Sub topic whenever a new file is uploaded. A streaming Dataflow pipeline can subscribe to this topic, triggering the processing job immediately. Dataflow is designed for scalable data transformation, including cleaning, deduplication, and enrichment by joining with BigQuery data, before writing the final results to a BigQuery table. This approach avoids the overhead of managing clusters or waiting for a batch schedule, directly addressing the requirements for immediate processing and minimal management.
A. Creating a Dataproc cluster for each file arrival every 10 minutes is inefficient and has high startup overhead, contradicting the low-overhead requirement.
B. An hourly batch schedule does not meet the requirement to initiate processing "as soon as the files are loaded," which happens every 10 minutes.
D. While functional, this adds unnecessary complexity and potential latency by using Cloud Run as an intermediary to trigger a Cloud Data Fusion job.
1. Google Cloud Documentation, "Cloud Storage triggers". This page describes how to use Pub/Sub notifications for Cloud Storage events like OBJECTFINALIZE to trigger downstream processing. (cloud.google.com/functions/docs/calling/storage)
2. Google Cloud Documentation, "Streaming with Cloud Pub/Sub". This section details how services like Dataflow can subscribe to Pub/Sub topics to process data in real-time as events occur. (cloud.google.com/pubsub/docs/overview#integrations)
3. Google Cloud Documentation, "Common BigQuery data ingestion patterns". The "Stream processing with Dataflow" pattern is described for scenarios requiring real-time transformation and loading from sources like Cloud Storage. (cloud.google.com/bigquery/docs/ingestion-patterns)
Keep practicing
Take the free ASSOCIATE-DATA-PRACTITIONER practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other ASSOCIATE-DATA-PRACTITIONER practice questions
- Question 1You are designing a pipeline to process data files that arrive in Cloud Storage by 3:00 a…
- Question 2Your organization plans to move their on-premises environment to Google Cloud. Your organ…
- Question 3You are responsible for managing Cloud Storage buckets for a research company. Your compa…
- Question 4You are predicting customer churn for a subscription-based service. You have a 50 PB hist…
- Question 5Your retail company collects customer data from various sources:Online transactions: Stor…
- Question 6Your company has developed a website that allows users to upload and share video files. T…
