HCE-5920 practice question 6 of 15

Which three file formats are splittable on HDFS? (Choose three). Choose 3 answers Choose 3 answers

Full set: 60 questions
Answer options
A: txt Correct answer
B: xml
C: Parquet Correct answer
D: xisx
E: Avro Correct answer
Correct Answer:
A. txt; C. Parquet; E. Avro
Explanation

A file format is considered splittable in HDFS if a large file can be logically divided into chunks that can be processed independently and in parallel by different nodes.
- Text files (.txt): These are splittable because data is typically separated by newlines. A processing framework like MapReduce can read a block of data, and if the split starts in the middle of a line, it can easily discard the first partial line and synchronize at the beginning of the next full line.
- Parquet: This is a columnar storage format designed for parallel processing in the Hadoop ecosystem. It structures data into row groups, which are designed to be read independently, making the format inherently splittable.
- Avro: This is a row-based format that includes synchronization markers between blocks of data. These markers allow a reader to seek to any point in a file, find the next marker, and begin reading, which makes Avro files splittable.

Why Incorrect

B. xml: Standard XML files are not inherently splittable because there is no reliable way to find the start or end of a record within an arbitrary file block without parsing from the beginning.
D. xisx: Assuming this is a typo for XLSX (Excel), this format is a compressed ZIP archive. Compressed archives cannot be split for parallel processing as they must be read sequentially from the start to be decompressed.

References

1. Avro Splittability: The official Apache Avro specification details the Object Container File format

which includes periodic "sync markers." These markers enable splitting. A reader can seek to any position in a file and then scan forward to the next sync marker to begin processing records.

Source: Apache Avro™ 1.11.1 Specification

Section: "Object Container Files". The specification states

"A sync marker is used to permit efficient splitting of files for parallel processing."

2. Parquet Splittability: The Apache Parquet format is explicitly designed for parallel processing. Data is organized into "row groups

" and the file's metadata allows readers to seek directly to the start of any row group

making it highly splittable.

Source: Apache Parquet Official Documentation

"Apache Parquet Format" section. The documentation describes the file structure consisting of row groups

which are the fundamental unit of parallel execution.

3. Text File Splittability: Hadoop's TextInputFormat treats plain text files as splittable. Its LineRecordReader can handle splits that occur in the middle of a line by reading past the initial partial line and starting at the next full newline character.

Source: Courseware from the University of California

Berkeley

CS 162: Operating Systems and System Programming. Lecture 22: "Distributed File Systems

" slide 32

discusses how MapReduce splits input files and how LineRecordReader handles records that cross split boundaries.

Scroll to Top