Which PDI step or entry processes data within the Hadoop cluster?

The Pentaho MapReduce job entry is the component designed to execute data processing logic directly within the Hadoop cluster. It orchestrates a MapReduce job by taking PDI transformations for the Mapper and optional Combiner/Reducer phases and submitting them to the Hadoop framework. Hadoop then distributes these tasks across its data nodes, processing the data in parallel where it resides. This "push-down" processing leverages the distributed computing power of the cluster, which is the core intent of the question. The other options involve data movement to or from the cluster, but the processing itself occurs on the PDI server.

A. The Hadoop File Output step writes data from the PDI engine to HDFS. The processing happens in PDI before the write.

B. The Hadoop File Input step reads data from HDFS into the PDI engine. The processing happens in PDI after the read.

D. The Hadoop Copy Files entry is a file management utility for copying or moving files within HDFS, not a data processing component.

1. Hitachi Vantara Documentation

Pentaho 9.4

"Pentaho MapReduce": "The Pentaho MapReduce job entry runs a MapReduce job on a Hadoop cluster. The MapReduce job is made up of a series of transformations that you create in PDI... The MapReduce job is processed on the Hadoop cluster

which allows for a great deal of scalability." (This directly confirms that processing occurs on the cluster).

2. Hitachi Vantara Documentation

Pentaho 9.4

"Hadoop File Input": "The Hadoop File Input step reads data from one or more files in a Hadoop cluster." (This describes reading data from Hadoop into the PDI transformation engine

not processing within it).

3. Hitachi Vantara Documentation

Pentaho 9.4

"Hadoop File Output": "The Hadoop File Output step writes data to one or more files in a Hadoop cluster." (This describes writing data from the PDI engine to Hadoop

with processing occurring before the write).