Which three file formats are splittable on HDFS? (Choose three). Choose 3 answers Choose 3 answers

A file format is considered splittable in HDFS if a large file can be logically divided into chunks that can be processed independently and in parallel by different nodes.

- Text files (.txt): These are splittable because data is typically separated by newlines. A processing framework like MapReduce can read a block of data, and if the split starts in the middle of a line, it can easily discard the first partial line and synchronize at the beginning of the next full line.

- Parquet: This is a columnar storage format designed for parallel processing in the Hadoop ecosystem. It structures data into row groups, which are designed to be read independently, making the format inherently splittable.

- Avro: This is a row-based format that includes synchronization markers between blocks of data. These markers allow a reader to seek to any point in a file, find the next marker, and begin reading, which makes Avro files splittable.

B. xml: Standard XML files are not inherently splittable because there is no reliable way to find the start or end of a record within an arbitrary file block without parsing from the beginning.

D. xisx: Assuming this is a typo for XLSX (Excel), this format is a compressed ZIP archive. Compressed archives cannot be split for parallel processing as they must be read sequentially from the start to be decompressed.

1. Avro Splittability: The official Apache Avro specification details the Object Container File format

which includes periodic "sync markers." These markers enable splitting. A reader can seek to any position in a file and then scan forward to the next sync marker to begin processing records.

Source: Apache Avro™ 1.11.1 Specification

Section: "Object Container Files". The specification states

"A sync marker is used to permit efficient splitting of files for parallel processing."

2. Parquet Splittability: The Apache Parquet format is explicitly designed for parallel processing. Data is organized into "row groups

" and the file's metadata allows readers to seek directly to the start of any row group

making it highly splittable.

Source: Apache Parquet Official Documentation

"Apache Parquet Format" section. The documentation describes the file structure consisting of row groups

which are the fundamental unit of parallel execution.

3. Text File Splittability: Hadoop's TextInputFormat treats plain text files as splittable. Its LineRecordReader can handle splits that occur in the middle of a line by reading past the initial partial line and starting at the next full newline character.

Source: Courseware from the University of California

Berkeley

CS 162: Operating Systems and System Programming. Lecture 22: "Distributed File Systems

" slide 32

discusses how MapReduce splits input files and how LineRecordReader handles records that cross split boundaries.