CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice question 16 of 25
Given: python CopyEdit spark.sparkContext.setLogLevel("") Which set contains the suitable configuration settings for Spark driver LOG_LEVELs?
Full set: 135 questionsApache Spark utilizes the log4j logging library. The spark.sparkContext.setLogLevel() method allows developers to control the verbosity of Spark's logging output during runtime. The accepted log levels are a standard set defined by log4j. These levels, in order of increasing severity, are ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF. The set provided in option B (ERROR, WARN, TRACE, OFF) consists entirely of these valid log levels. The other options contain invalid level names.
A. The level FAIL is not a valid log4j or Spark log level. The standard level for reporting failures is ERROR or FATAL.
C. The level NONE is not a valid log level. To disable all logging, the correct level is OFF.
D. The level NONE is not a valid log level. The correct level to turn off logging is OFF.
1. Apache Spark 3.5.1 Documentation
PySpark API Reference: In the documentation for the pyspark.SparkContext.setLogLevel method
the valid log levels are explicitly listed.
Source: pyspark.SparkContext.setLogLevel(logLevel)
Quote: "Control our logLevel. This overrides any user-defined log settings. Valid log levels include: ALL
DEBUG
ERROR
FATAL
INFO
OFF
TRACE
WARN"
Link: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.SparkContext.setLogLevel.html
2. Apache Spark 3.5.1 Documentation
Configuration: The "Configuration" page under the "Configuring Logging" section describes how Spark's logging is configured using log4j
which underpins the available log levels.
Source: Configuration - Configuring Logging
Quote: "Spark uses log4j for logging... You can configure it by adding a log4j2.properties file in the conf directory." This confirms the use of the log4j standard
whose levels are listed in the API reference.
Link: https://spark.apache.org/docs/latest/configuration.html#configuring-logging
Keep practicing
Take the free CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice test
Ten exam-style questions with answers and explanations, plus the exam facts and study guides.
More questions
Other CERTIFIED-ASSOCIATE-DEVELOPER-FOR-APACHE-SPARK-3-5-QUESTIONS practice questions
- Question 1You have: DataFrame A: 128 GB of transactions DataFrame B: 1 GB user lookup table Which s…
- Question 2A Spark DataFrame df is cached using the MEMORY_AND_DISK storage level, but the DataFrame…
- Question 344 of 55. A data engineer is working on a real-time analytics pipeline using Spark Struct…
- Question 7A developer wants to test Spark Connect with an existing Spark application. What are the …
- Question 8A data engineer is running a Spark job to process a dataset of 1 TB stored in distributed…
- Question 10A data engineer is building a Structured Streaming pipeline and wants the pipeline to rec…
