HCE-5920 practice question 6 of 15
Which three file formats are splittable on HDFS? (Choose three). Choose 3 answers Choose 3 answers
Full set: 60 questionsA file format is considered splittable in HDFS if a large file can be logically divided into chunks that can be processed independently and in parallel by different nodes.
- Text files (.txt): These are splittable because data is typically separated by newlines. A processing framework like MapReduce can read a block of data, and if the split starts in the middle of a line, it can easily discard the first partial line and synchronize at the beginning of the next full line.
- Parquet: This is a columnar storage format designed for parallel processing in the Hadoop ecosystem. It structures data into row groups, which are designed to be read independently, making the format inherently splittable.
- Avro: This is a row-based format that includes synchronization markers between blocks of data. These markers allow a reader to seek to any point in a file, find the next marker, and begin reading, which makes Avro files splittable.
B. xml: Standard XML files are not inherently splittable because there is no reliable way to find the start or end of a record within an arbitrary file block without parsing from the beginning.
D. xisx: Assuming this is a typo for XLSX (Excel), this format is a compressed ZIP archive. Compressed archives cannot be split for parallel processing as they must be read sequentially from the start to be decompressed.
1. Avro Splittability: The official Apache Avro specification details the Object Container File format
which includes periodic "sync markers." These markers enable splitting. A reader can seek to any position in a file and then scan forward to the next sync marker to begin processing records.
Source: Apache Avro™ 1.11.1 Specification
Section: "Object Container Files". The specification states
"A sync marker is used to permit efficient splitting of files for parallel processing."
2. Parquet Splittability: The Apache Parquet format is explicitly designed for parallel processing. Data is organized into "row groups
" and the file's metadata allows readers to seek directly to the start of any row group
making it highly splittable.
Source: Apache Parquet Official Documentation
"Apache Parquet Format" section. The documentation describes the file structure consisting of row groups
which are the fundamental unit of parallel execution.
3. Text File Splittability: Hadoop's TextInputFormat treats plain text files as splittable. Its LineRecordReader can handle splits that occur in the middle of a line by reading past the initial partial line and starting at the next full newline character.
Source: Courseware from the University of California
Berkeley
CS 162: Operating Systems and System Programming. Lecture 22: "Distributed File Systems
" slide 32
discusses how MapReduce splits input files and how LineRecordReader handles records that cross split boundaries.
Keep practicing
Take the free HCE-5920 practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other HCE-5920 practice questions
- Question 1You are connecting to a secure Hadoop dueler from Pentaho and want to use impersonation. …
- Question 2You are encryption your database connection password to use in the kettle. properties fil…
- Question 3you want to make a dynamic PDI transformation that is driven with variables that are load…
- Question 4A Big Data customer wants to run POI transformations on Spark on their production Hadoop …
- Question 5You are running a PDI Job and you need to identify when something fails for a specific jo…
- Question 7In a PDI transformation you are retrieving data from a large lookup table using a Databas…
