Hyperparameter Tuning Using Bayesian Optimization in MATLAB

For comparative studies on hyperparameter optimization methods, refer to the following articles:

Hyperparameter Tuning Based on Bayesian Optimization-1

Jethro, WeChat Official Account: Xiaobailou Laboratory 【MATLAB Reinforcement Learning Toolbox】 Hyperparameter Tuning Based on Bayesian Optimization-1

01

—

The Bayesian optimization algorithm is a global optimization method used to minimize a scalar objective function f(h) within a bounded domain (where h represents the set of hyperparameters to be optimized). This objective function can be deterministic or stochastic, meaning that it may return different results when evaluated at the same input variable h. Furthermore, the components of the input variable h can be continuous real numbers, integers, or categorical variables, i.e., a discrete set of names [1]. The core idea of Bayesian optimization is to predict the behavior of the objective function at unassessed points by constructing a probabilistic model (such as a Gaussian process model) and selecting the optimal sampling point for evaluation based on these predictions. This process typically includes the following steps, as illustrated in Figure 1 [2,3]:

Initialization: Start sampling from random points and evaluate the objective function values.

Update Model: Update the probability distribution of the objective function using a Gaussian process or other statistical models based on existing observational data.

Select Next Sampling Point: Decide the next point to evaluate through a strategy called the “Acquisition Function.” Common acquisition functions include Expected Improvement, Probability of Improvement, and Lower Confidence Bound.

Iterative Optimization: Repeat the above steps until a stopping criterion is met (such as reaching the maximum number of iterations or finding a sufficiently good solution).

Figure 1 Structure Diagram of Bayesian Optimization

Note: h represents the set of hyperparameters; Gp represents the posterior distribution of the Gaussian regression process; f* represents the currently obtained best Gaussian process regression function value; μ(h) and σ(h) represent the mean and variance of the objective function f(h) under the selected candidate hyperparameter set h; Trapz() represents a custom objective function; for specific meanings, please refer to the literature.

Objective Function:

The bayesopt command attempts to minimize the objective function (objective function). Specifically, when a function needs to be maximized, simply set the objective function to the negative value of the function to be maximized. For example, if the objective function is f(h), then -f(h) should be set as the objective function for bayesopt. In this way, bayesopt will attempt to minimize -f(h), thereby achieving the effect of maximizing f(h). Bayesopt will pass a variable table to the objective function, these variables having the declared names and types. Additionally, the objective function can return extra information, such as coupled constraints and user data, which help with constraint handling and result analysis during the optimization process.

The syntax of the objective function is as follows:

[objective,coupledconstraints,userdata] = fun(h)%objective — The value of the objective function at point h, a real number;%coupledconstraints - The value of coupled constraints (if any, optional output), a real value vector. %Negative values indicate that constraints are met, positive values indicate unmet constraints.%userdata — The function can return optional data for further use, such as for plotting or logging (optional output).

Variable Settings:

In MATLAB, use the optimizableVariable function to create variable description objects for Bayesian optimization. Each variable has a unique name and value range. The basic syntax for creating a variable is as follows:

variable = optimizableVariable(Name,Range)%where Name is the variable name, Range is the variable value range,%For continuous and integer variables, Range is a vector containing the lower and upper bounds;%For categorical variables, Range is a cell array containing possible value names.

Variable Types: The Type parameter specifies the following three types:

‘real’: Continuous real values between finite bounds. The vector [lower upper] is the variable range Range, indicating lower and upper bounds.

‘integer’: Integer values within a finite range, similar to ‘real’.

‘categorical’: A cell array of possible value names specified in the Range parameter, such as {‘red’,’green’,’blue’}.

Log Transformation: For ‘real’ and ‘integer’ type variables, the bayesopt can specify searching in the logarithmic scale space by setting the Transform name-value pair to ‘log’. For this transformation, ensure that the lower bound of Range is strictly positive (for ‘real’ type) or non-negative (for ‘integer’ type). For example:

var2 = optimizableVariable('ivar',[1 1000],'Type','integer','Transform','log')

Bayesopt’s range includes endpoints. Therefore, 0 cannot be used as the lower limit for logarithmic transformation variables. If you want to set the lower limit of ivar to 0 in a logarithmic transformation variable, you can set the lower limit of ivar to 1 and then use ivar-1 in the objective function. Log transformation can convert the multiplicative operations of the original data into additive operations, making numerical calculations more stable and avoiding underflow or overflow issues caused by excessively large numbers. In Bayesian optimization, log transformation can compress the value range of the objective function into a smaller range, thus reducing the sparsity problem of the data.

Excluding Variables: If you want to exclude a certain parameter (e.g., ivar) from the optimization, you can handle it by setting ivar.optimize=false.

Acquisition Function Types:

In Bayesian optimization, the acquisition function is one of the key components, mainly responsible for determining the location of the next sampling point. It quantifies the value of each potential sampling point by combining the predicted values of the surrogate model and uncertainties, thus guiding the next choice in the optimization process. Specifically, the acquisition function needs to strike a balance between exploration and exploitation to avoid getting stuck in local optima and to quickly find the global optimum. Here are six common types of acquisition functions and their characteristics:

Expected Improvement (EI):

'expected-improvement'

EI is one of the most popular acquisition functions, performing excellently in balancing exploration and exploitation. EI calculates the potential improvement that candidate points may bring compared to the current optimal value, considering uncertainty. EI is suitable for most situations, but its downside is that it may get stuck in local optima in certain cases.

Expected Improvement Plus (EI+):

'expected-improvement-plus'

EI+ adds extra parameters on top of EI to reduce the risk of getting stuck in local optima. It starts with a larger parameter value during the initial sampling and gradually decreases as the number of samples increases, thus tending to explore more in the early stages.

Expected Improvement Per Second (EI/s):

'expected-improvement-per-second'

EI/s is designed for situations with limited computational resources, optimizing the efficiency of each iteration by considering time costs.

Expected Improvement Per Second Plus (EI/s+):

'expected-improvement-per-second-plus' (default)

EI/s+ is the default acquisition function selected by MATLAB, combining the advantages of EI/s and EI+, considering both time costs and avoiding the issue of getting stuck in local optima.

Lower Confidence Bound (LCB):

'lower-confidence-bound'

LCB is an acquisition function that tends to explore, selecting the next sampling point based on confidence intervals. LCB is suitable for situations requiring extensive exploration, as it tends to sample points with higher uncertainty.

Probability of Improvement (PI):

'probability-of-improvement'

PI calculates the probability that candidate points can improve the current optimal value. PI leans more towards exploitation, as it only considers whether it can improve the current optimal value, not the magnitude of the improvement.

Acquisition Function Selection Experience:

• If you want to explore unknown areas more in the early stages, you can choose LCB or PI.

• If you want to balance exploitation and exploration, EI is a good choice.

• If computational resources are limited and you want to improve efficiency, you can choose EI/s or EI/s+.

• If you are concerned about getting stuck in local optima, you can choose EI+ or EI/s+.

Therefore, when selecting an acquisition function, you should decide the most suitable type based on optimization goals, computational resources, and preferences for exploration versus exploitation.