Which of the following hyperparameter optimization methods automatically makes informed selections of hyperparameter values based on previous trials for each iterative model evaluation?

Tree of Parzen Estimators (TPE) is a sequential, model-based (Bayesian) optimization algorithm. It iteratively builds a probabilistic model based on the history of evaluated hyperparameter sets and their corresponding performance. In each step, it uses this model to select the next set of hyperparameters that are most likely to yield an improvement according to an acquisition function (Expected Improvement). This "informed" selection process allows TPE to focus the search on more promising regions of the hyperparameter space, making it more efficient than methods that do not learn from past results.

A. Random Search samples hyperparameter configurations independently and randomly; it does not learn from past trials to inform future selections.
B. Halving Random Search is an early-stopping method that prunes poor configurations, but the initial selections are random and not informed by past results.
D. Grid Search exhaustively evaluates a predefined, static grid of values. The search path is fixed and does not adapt based on trial outcomes.

1. Official Databricks Documentation: Databricks Machine Learning Guide

"Hyperopt concepts". The documentation states

"Tree of Parzen Estimators (TPE) is a Bayesian approach that iteratively and adaptively chooses new hyperparameter settings to explore based on past results." This directly supports that TPE makes informed selections.

Source: Databricks Documentation > Machine Learning > Hyperparameter tuning > Hyperopt concepts.

2. Peer-Reviewed Academic Publication: Bergstra

J.

Bardenet

R.

Bengio

Y.

& Kégl

B. (2011). Algorithms for Hyper-Parameter Optimization. In Advances in Neural Information Processing Systems 24 (NIPS 2011).

Section: 2. Sequential Model-Based Optimization. The paper describes sequential model-based optimization (of which TPE is an example) as a formal process of using the history of (hyperparameter

loss) pairs to inform the choice of the next hyperparameter to evaluate.

3. Official Databricks Documentation: Databricks Machine Learning Guide

"Parallelize hyperparameter tuning with scikit-learn and MLflow". This guide contrasts TPE with Random Search

explaining

"Whereas random search is a parallelizable tuning algorithm

TPE makes decisions sequentially." This highlights the iterative

results-based nature of TPE.

Source: Databricks Documentation > Machine Learning > Hyperparameter tuning > Parallelize hyperparameter tuning with scikit-learn and MLflow.