You are designing an Apache Beam pipeline to enrich data from Cloud Pub/Sub with static reference data from BigQuery. The reference data is small…

The pipeline must be a streaming job because it ingests data from Cloud Pub/Sub, which is an unbounded, continuous data source. The pipeline will use PubSubIO to read from Pub/Sub and BigQueryIO to both read the static reference data and write the enriched output.

The key requirement is enriching the streaming data with a small, static reference dataset. The most efficient and standard Apache Beam pattern for this scenario is using side-inputs. A side input allows a ParDo transform to access an entire PCollection (the reference data) in-memory while processing each element of the main PCollection (the streaming data), making it ideal for lookups.

A. A batch job is incorrect because Cloud Pub/Sub provides a continuous, unbounded stream of data that requires a streaming pipeline for real-time processing.

B. JdbcIO is for connecting to databases via a JDBC driver, not for BigQuery. Side-outputs are used to split a single PCollection into multiple outputs, not for data enrichment.

D. Side-outputs are for splitting a PCollection into multiple distinct collections based on some criteria, which is different from the required enrichment task.

1. Apache Beam Programming Guide, "Side inputs": "A side input is an additional input that a ParDo transform can access each time it processes an element in the input PCollection... Side inputs are useful when your ParDo needs to inject data that is computed at pipeline construction time, or data that is computed in an earlier stage of the same pipeline... especially when the side input is much smaller than the main input PCollection." This directly supports using side-inputs for the small, static reference data.

2. Google Cloud Documentation, "Common stream analytics patterns": Under the "Data enrichment" section, it describes the pattern of joining a streaming data source with a slowly changing or static dataset. For small datasets, it states: "If the dataset is small enough to fit into memory on a worker, you can load the dataset from Cloud Storage or a database like Cloud SQL or Bigtable and use it as a side input for a Beam ParDo transform." This validates the use of side-inputs for small enrichment tables.

3. Apache Beam Documentation, "Built-in I/O Transforms": This page lists the I/O connectors. It specifies ReadFromPubSub and WriteToPubSub (part of PubSubIO) for Cloud Pub/Sub, and ReadFromBigQuery and WriteToBigQuery (part of BigQueryIO) for BigQuery, confirming the correct I/O transforms.