Bayesian Optimization and Hyperband (BOHB) Hyperparameter Tuning With an Example

Machine learning models often require tuning of hyperparameters to achieve their best performance. Hyperparameter tuning can be a daunting and time-consuming task, as it involves experimenting with different parameter combinations to find the optimal settings. Bayesian Optimization and Hyperband (BOHB) is a cutting-edge technique that leverages Bayesian optimization and the Hyperband algorithm to efficiently search for the best hyperparameters for machine learning models. In this article, we will delve into what BOHB is and its advantages and provide a practical example of tuning hyperparameters for an XGBoost model using BOHB.

What Is BOHB?

BOHB stands for Bayesian Optimization and Hyperband. It combines two powerful concepts:

1. Bayesian Optimization: This is a probabilistic model-based optimization technique that uses a surrogate model (usually a Gaussian process) to model the objective function (e.g., model accuracy) and makes informed decisions about where to explore the hyperparameter space next. It's particularly useful when the objective function is expensive to evaluate.
2. Hyperband: Hyperband is a resource allocation algorithm that efficiently uses a limited budget (e.g., time or computation) to tune hyperparameters. It progressively allocates resources to the most promising hyperparameter configurations and discards underperforming ones.

BOHB combines these two concepts to create an efficient hyperparameter optimization process.

Advantages of BOHB

1. Efficiency: BOHB efficiently utilizes resources by focusing on promising hyperparameter configurations. It eliminates poorly performing configurations early in the process, saving valuable computation time.
2. Scalability: BOHB scales well to large hyperparameter spaces and can handle both continuous and categorical hyperparameters. This makes it suitable for tuning a wide range of machine-learning models.
3. Automatic configuration: BOHB automates the hyperparameter tuning process, reducing the need for manual intervention. This is particularly beneficial when dealing with complex models and hyperparameter spaces.
4. State-of-the-art performance: BOHB often outperforms traditional hyperparameter optimization methods, such as grid search and random search, in terms of finding optimal hyperparameters.

Example: Population-Based Training With XGBoost

Now, let's dive into a practical example of using BOHB to optimize the hyperparameters of an XGBoost model.

Step 1: Import Libraries and Load Data

Import the necessary libraries, including xgboost for the XGBoost classifier, NumPy for numerical operations, load_iris to load the Iris dataset and libraries related to BOHB.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
import xgboost as xgb
from hpbandster.core.worker import Worker
from hpbandster.optimizers import BOHB

Step 2: Load and Prepare the Data

Load the Iris dataset using load_irs() and split it into training and testing sets with a 75-25 split ratio. This dataset will be used to train and validate the XGBoost model.

# Load the Iris dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25)

Step 3: Define the BOHB Worker Class

This step defines the objective function that BOHB will use to evaluate the performance of each hyperparameter configuration. The objective function should return a negative value, where lower values indicate better performance. In this case, the objective function is simply the negative of the accuracy score on the test set.  

# Define the objective function
def objective_function(config):
    """Objective function for BoHB."""

    # Create an XGBoost classifier
    clf = xgb.XGBClassifier(**config)

    # Train the classifier
    clf.fit(X_train, y_train)

    # Evaluate the classifier on the test set
    score = clf.score(X_test, y_test)

    return -score

Step 4: Define the Hyperparameter Search Space

This step defines the configuration space for the hyperparameters that we want to tune. The configuration space is a dictionary that maps each hyperparameter to a range of possible values.

# Define the configuration space
config_space = {
    'n_estimators': hp.quniform('n_estimators', 100, 1000, log=True),
    'max_depth': hp.quniform('max_depth', 1, 10, q=1),
    'learning_rate': hp.loguniform('learning_rate', 0.001, 0.1),
    'gamma': hp.loguniform('gamma', 0.01, 1.0),
    'colsample_bytree': hp.uniform('colsample_bytree', 0.5, 1.0),
    'reg_alpha': hp.loguniform('reg_alpha', 0.01, 1.0),
    'reg_lambda': hp.loguniform('reg_lambda', 0.01, 1.0),
}

Step 5: Initialize the BOHB Optimizer

This step creates a BOHB optimizer object. The config_space parameter specifies the configuration space for the hyperparameters.

# Create a BoHB optimizer
optimizer = BOHB(config_space=config_space)

Step 6: Run the Optimization

This step runs the BOHB optimization. The objective function parameter specifies the objective function to use, and the budget parameter specifies the number of evaluations to perform.

# Run the BoHB optimization
optimizer.run(objective_function, budget=100)

Step 7: Retrieve the Best Hyperparameters and Score

This step gets the best hyperparameter configuration that was found by BOHB and prints the best hyperparameter configuration to the console.

# Get the best configuration
best_config = optimizer.get_best_config()

# Print the best configuration
print('Best configuration:', best_config)

After running the above code. The best configuration of the parameters found out by BOHB are:

Best configuration: {'n_estimators': 242.0, 
                     'max_depth': 7.0, 
                     'learning_rate': 0.025304661570619978, 
                     'gamma': 0.05979385282331321, 
                     'colsample_bytree': 0.7106730811630383, 
                     'reg_alpha': 0.03662072703875718, 
                     'reg_lambda': 0.07257624812809639
                    }

Conclusion

In the ever-evolving field of machine learning, the quest for optimal model performance often begins with fine-tuning hyperparameters. Bayesian Optimization and Hyperband (BOHB) emerge as a formidable solution to this challenge, efficiently navigating the complex hyperparameter space to discover configurations that maximize model performance. This article has shed light on BOHB's inner workings, advantages, and practical application using the popular XGBoost algorithm.

BOHB's efficiency, scalability, and automation make it a compelling choice for hyperparameter optimization. Its ability to adaptively allocate resources to promising configurations while discarding underperforming ones accelerates the optimization process and conserves valuable computational resources.

The provided code example demonstrates that BOHB seamlessly integrates into the machine learning workflow. By leveraging this powerful technique, data scientists and machine learning practitioners can streamline their hyperparameter tuning efforts, allowing them to focus on model development and deployment.

Do you have any questions related to this article? Leave a comment and ask your question; I will do my best to answer it.

Thanks for reading!