Hyperparameter tuning by randomized-search¶

In the previous notebook, we showed how to use a grid-search approach to search for the best hyperparameters maximizing the statistical performance of a predictive model.

However, a grid-search approach has limitations. It does not scale when the number of parameters to tune is increasing. Also, the grid will imposed a regularity during the search which might be problematic.

In this notebook, we will present the another method to tune hyperparameters called randomized search.

Our predictive model¶

Let us reload the dataset as we did previously:

from sklearn import set_config

set_config(display="diagram")

import pandas as pd

adult_census = pd.read_csv("../datasets/adult-census.csv")

We extract the column containing the target.

target_name = "class"
target = adult_census[target_name]
target

       <=50K
       <=50K
        >50K
        >50K
       <=50K
          ...  
   <=50K
    >50K
   <=50K
   <=50K
    >50K
Name: class, Length: 48842, dtype: object

We drop from our data the target and the "education-num" column which duplicates the information with "education" columns.

data = adult_census.drop(columns=[target_name, "education-num"])
data.head()

	age	workclass	education	marital-status	occupation	relationship	race	sex	capital-gain	hours-per-week	native-country
0	25	Private	11th	Never-married	Machine-op-inspct	Own-child	Black	Male	0	40	United-States
1	38	Private	HS-grad	Married-civ-spouse	Farming-fishing	Husband	White	Male	0	50	United-States
2	28	Local-gov	Assoc-acdm	Married-civ-spouse	Protective-serv	Husband	White	Male	0	40	United-States
3	44	Private	Some-college	Married-civ-spouse	Machine-op-inspct	Husband	Black	Male	7688	40	United-States
4	18	?	Some-college	Never-married	?	Own-child	White	Female	0	30	United-States

Once the dataset is loaded, we split it into a training and testing sets.

from sklearn.model_selection import train_test_split

data_train, data_test, target_train, target_test = train_test_split(
    data, target, random_state=42)

We will create the same predictive pipeline as seen in the grid-search section.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OrdinalEncoder
from sklearn.compose import make_column_selector as selector

categorical_columns_selector = selector(dtype_include=object)
categorical_columns = categorical_columns_selector(data)

categorical_preprocessor = OrdinalEncoder(handle_unknown="use_encoded_value",
                                          unknown_value=-1)
preprocessor = ColumnTransformer([
    ('cat-preprocessor', categorical_preprocessor, categorical_columns)],
    remainder='passthrough', sparse_threshold=0)

# for the moment this line is required to import HistGradientBoostingClassifier
from sklearn.experimental import enable_hist_gradient_boosting
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.pipeline import Pipeline

model = Pipeline([
    ("preprocessor", preprocessor),
    ("classifier", HistGradientBoostingClassifier(random_state=42, max_leaf_nodes=4)),
])

model

Pipeline

Pipeline(steps=[('preprocessor',
                 ColumnTransformer(remainder='passthrough', sparse_threshold=0,
                                   transformers=[('cat-preprocessor',
                                                  OrdinalEncoder(handle_unknown='use_encoded_value',
                                                                 unknown_value=-1),
                                                  ['workclass', 'education',
                                                   'marital-status',
                                                   'occupation', 'relationship',
                                                   'race', 'sex',
                                                   'native-country'])])),
                ('classifier',
                 HistGradientBoostingClassifier(max_leaf_nodes=4,
                                                random_state=42))])

preprocessor: ColumnTransformer

ColumnTransformer(remainder='passthrough', sparse_threshold=0,
                  transformers=[('cat-preprocessor',
                                 OrdinalEncoder(handle_unknown='use_encoded_value',
                                                unknown_value=-1),
                                 ['workclass', 'education', 'marital-status',
                                  'occupation', 'relationship', 'race', 'sex',
                                  'native-country'])])

cat-preprocessor

['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']

OrdinalEncoder

OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)

remainder

passthrough

passthrough

HistGradientBoostingClassifier

HistGradientBoostingClassifier(max_leaf_nodes=4, random_state=42)

Tuning using a randomized-search¶

With the GridSearchCV estimator, the parameters need to be specified explicitly. We already mentioned that exploring a large number of values for different parameters will be quickly untractable.

Instead, we can randomly generate the parameter candidates. Indeed, such approach avoids the regularity of the grid. Hence, adding more evaluations can increase the resolution in each direction. This is the case in the frequent situation where the choice of some hyperparameters is not very important, as for hyperparameter 2 in the figure below.

Indeed, the number of evaluation points need to be divided across the two different hyperparameters. With a grid, the danger is that the region of good hyperparameters fall between the line of the grid: this region is aligned with the grid given that hyperparameter 2 has a weak influence. Rather, stochastic search will sample hyperparameter 1 independently from hyperparameter 2 and find the optimal region.

The RandomizedSearchCV class allows for such stochastic search. It is used similarly to the GridSearchCV but the sampling distributions need to be specified instead of the parameter values. For instance, we will draw candidates using a log-uniform distribution because the parameters we are interested in take positive values with a natural log scaling (.1 is as close to 1 as 10 is).

Note

Random search (with RandomizedSearchCV) is typically beneficial compared to grid search (with GridSearchCV) to optimize 3 or more hyperparameters.

We will optimize 3 other parameters in addition to the ones we optimized in the notebook presenting the GridSearchCV:

l2_regularization: it corresponds to the constant to regularized the loss function
min_samples_leaf: it corresponds to the minimum number of samples required in a leaf;
max_bins: it corresponds to the maximum number of bins to construct the histograms.

We recall the meaning of the 2 remaining parameters:

learning_rate: it corresponds to the speed at which the gradient-boosting will correct the residuals at each boosting iteration;
max_leaf_nodes: it corresponds to the maximum number of leaves for each tree in the ensemble.

Note

scipy.stats.loguniform can be used to generate floating numbers. To generate random values for integer-valued parameters (e.g. min_samples_leaf) we can adapt is as follows:

from scipy.stats import loguniform


class loguniform_int:
    """Integer valued version of the log-uniform distribution"""
    def __init__(self, a, b):
        self._distribution = loguniform(a, b)

    def rvs(self, *args, **kwargs):
        """Random variable sample"""
        return self._distribution.rvs(*args, **kwargs).astype(int)

Now, we can define the randomized search using the different distributions. Executing 10 iterations of 5-fold cross-validation for random parametrizations of this model on this dataset can take from 10 seconds to several minutes, depending on the speed of the host computer and the number of available processors.

%%time
from sklearn.model_selection import RandomizedSearchCV

param_distributions = {
    'classifier__l2_regularization': loguniform(1e-6, 1e3),
    'classifier__learning_rate': loguniform(0.001, 10),
    'classifier__max_leaf_nodes': loguniform_int(2, 256),
    'classifier__min_samples_leaf': loguniform_int(1, 100),
    'classifier__max_bins': loguniform_int(2, 255),
}

model_random_search = RandomizedSearchCV(
    model, param_distributions=param_distributions, n_iter=10,
    cv=5, verbose=1,
)
model_random_search.fit(data_train, target_train)

Fitting 5 folds for each of 10 candidates, totalling 50 fits

CPU times: user 31 s, sys: 769 ms, total: 31.8 s
Wall time: 31.8 s

RandomizedSearchCV

RandomizedSearchCV(cv=5,
                   estimator=Pipeline(steps=[('preprocessor',
                                              ColumnTransformer(remainder='passthrough',
                                                                sparse_threshold=0,
                                                                transformers=[('cat-preprocessor',
                                                                               OrdinalEncoder(handle_unknown='use_encoded_value',
                                                                                              unknown_value=-1),
                                                                               ['workclass',
                                                                                'education',
                                                                                'marital-status',
                                                                                'occupation',
                                                                                'relationship',
                                                                                'race',
                                                                                'sex',
                                                                                'native-country'])])),
                                             ('classifier',
                                              Hi...
                   param_distributions={'classifier__l2_regularization': ,
                                        'classifier__learning_rate': ,
                                        'classifier__max_bins': <__main__.loguniform_int object at 0x7efd58f6c490>,
                                        'classifier__max_leaf_nodes': <__main__.loguniform_int object at 0x7efd5dc09110>,
                                        'classifier__min_samples_leaf': <__main__.loguniform_int object at 0x7efd5e8950d0>},
                   verbose=1)

preprocessor: ColumnTransformer

ColumnTransformer(remainder='passthrough', sparse_threshold=0,
                  transformers=[('cat-preprocessor',
                                 OrdinalEncoder(handle_unknown='use_encoded_value',
                                                unknown_value=-1),
                                 ['workclass', 'education', 'marital-status',
                                  'occupation', 'relationship', 'race', 'sex',
                                  'native-country'])])

cat-preprocessor

['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race', 'sex', 'native-country']

OrdinalEncoder

OrdinalEncoder(handle_unknown='use_encoded_value', unknown_value=-1)

remainder

passthrough

passthrough

HistGradientBoostingClassifier

HistGradientBoostingClassifier(max_leaf_nodes=4, random_state=42)

Then, we can compute the accuracy score on the test set.

accuracy = model_random_search.score(data_test, target_test)

print(f"The test accuracy score of the best model is "
      f"{accuracy:.2f}")

The test accuracy score of the best model is 0.87

from pprint import pprint

print("The best parameters are:")
pprint(model_random_search.best_params_)

The best parameters are:
{'classifier__l2_regularization': 0.3336133907581423,
 'classifier__learning_rate': 0.05009918607333982,
 'classifier__max_bins': 80,
 'classifier__max_leaf_nodes': 20,
 'classifier__min_samples_leaf': 9}

We can inspect the results using the attributes cv_results as we did previously.

def shorten_param(param_name):
    if "__" in param_name:
        return param_name.rsplit("__", 1)[1]
    return param_name

# get the parameter names
column_results = [
    f"param_{name}" for name in param_distributions.keys()]
column_results += [
    "mean_test_score", "std_test_score", "rank_test_score"]

cv_results = pd.DataFrame(model_random_search.cv_results_)
cv_results = cv_results[column_results].sort_values(
    "mean_test_score", ascending=False)
cv_results = cv_results.rename(shorten_param, axis=1)
cv_results

	l2_regularization	learning_rate	max_leaf_nodes	min_samples_leaf	max_bins	mean_test_score	std_test_score	rank_test_score
2	0.333613	0.050099	20	9	80	0.857471	0.002679	1
9	0.00984	0.12749	3	2	37	0.851274	0.002628	2
0	0.002382	0.187691	189	14	20	0.849772	0.001998	3
8	0.008506	0.090429	3	55	12	0.834757	0.002356	4
7	0.000058	0.217731	5	23	6	0.832819	0.004337	5
6	0.001091	0.044767	206	2	5	0.826650	0.001620	6
4	0.000446	0.008104	7	8	249	0.816167	0.001061	7
5	19.851949	0.286559	26	69	2	0.801725	0.002883	8
1	0.000017	0.01488	4	59	7	0.785318	0.003433	9
3	308.772059	0.001166	4	7	2	0.758947	0.000013	10

In practice, a randomized hyperparameter search is usually run with a large number of iterations. In order to avoid the computation cost and still make a decent analysis, we load the results obtained from a similar search with 200 iterations.

# model_random_search = RandomizedSearchCV(
#     model, param_distributions=param_distributions, n_iter=200,
#     n_jobs=2, cv=5)
# model_random_search.fit(data_train, target_train)
# cv_results =  pd.DataFrame(model_random_search.cv_results_)
# cv_results.to_csv("../figures/randomized_search_results.csv")

cv_results = pd.read_csv("../figures/randomized_search_results.csv",
                         index_col=0)

As we have more than 2 parameters in our grid-search, we cannot visualize the results using a heatmap. However, we can us a parallel coordinates plot.

(cv_results[column_results].rename(
    shorten_param, axis=1).sort_values("mean_test_score"))

	l2_regularization	learning_rate	max_leaf_nodes	min_samples_leaf	max_bins	mean_test_score	std_test_score	rank_test_score
357	0.000026	3.075318	3	68	31	0.241053	0.000013	500
200	0.000444	6.236325	2	2	30	0.344629	0.207156	499
413	0.000001	8.828574	64	1	144	0.448205	0.253714	497
344	0.000003	7.091079	5	1	95	0.448205	0.253714	497
232	0.000097	9.976823	28	5	3	0.448205	0.253714	496
...	...	...	...	...	...	...	...	...
327	4.733808	0.036786	61	5	241	0.869673	0.002417	5
328	2.036232	0.224702	28	49	236	0.869837	0.000808	4
21	4.994918	0.077047	53	7	192	0.870793	0.001993	3
343	0.000404	0.244503	15	15	229	0.871339	0.002741	2
208	0.011775	0.076653	24	2	155	0.871393	0.001588	1

500 rows × 8 columns

import numpy as np
import plotly.express as px

fig = px.parallel_coordinates(
    cv_results.rename(shorten_param, axis=1).apply({
        "learning_rate": np.log10,
        "max_leaf_nodes": np.log2,
        "max_bins": np.log2,
        "min_samples_leaf": np.log10,
        "l2_regularization": np.log10,
        "mean_test_score": lambda x: x}),
    color="mean_test_score",
    color_continuous_scale=px.colors.sequential.Viridis,
)
fig.show()

Scikit-learn course

Hyperparameter tuning by randomized-search¶

Our predictive model¶

Tuning using a randomized-search¶