Analysis of hyperparameter search results

In the previous notebook we showed how to implement a randomized search for tuning the hyperparameters of a HistGradientBoostingClassifier to fit the adult_census dataset. In practice, a randomized hyperparameter search is usually run with a large number of iterations.

In order to avoid the computational cost and still make a decent analysis, we load the results obtained from a similar search with 500 iterations.

import pandas as pd

cv_results = pd.read_csv(
    "../figures/randomized_search_results.csv", index_col=0
)
cv_results

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	param_classifier__l2_regularization	param_classifier__learning_rate	param_classifier__max_bins	param_classifier__max_leaf_nodes	param_classifier__min_samples_leaf	params	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
0	0.540456	0.062725	0.052069	0.002661	2.467047	0.550075	86	22	6	{'classifier__l2_regularization': 2.4670474863...	0.856558	0.862271	0.857767	0.854491	0.856675	0.857552	0.002586	48
1	1.110536	0.033403	0.074142	0.002165	0.015449	0.001146	60	19	1	{'classifier__l2_regularization': 0.0154488709...	0.758974	0.758941	0.758941	0.758941	0.758941	0.758947	0.000013	323
2	1.137484	0.053150	0.092993	0.029005	1.095093	0.004274	151	17	10	{'classifier__l2_regularization': 1.0950934559...	0.783267	0.758941	0.776413	0.779143	0.758941	0.771341	0.010357	311
3	3.935108	0.202993	0.118105	0.023658	0.003621	0.001305	18	164	37	{'classifier__l2_regularization': 0.0036210968...	0.758974	0.758941	0.758941	0.758941	0.758941	0.758947	0.000013	323
4	0.255219	0.038301	0.056048	0.016736	0.000081	5.407382	97	8	3	{'classifier__l2_regularization': 8.1060737427...	0.758974	0.758941	0.758941	0.758941	0.758941	0.758947	0.000013	323
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
495	0.452411	0.023006	0.055563	0.000846	0.000075	0.364373	92	17	4	{'classifier__l2_regularization': 7.4813767874...	0.858332	0.865001	0.862681	0.860360	0.860770	0.861429	0.002258	34
496	1.133042	0.014456	0.078186	0.002199	5.065946	0.001222	7	17	1	{'classifier__l2_regularization': 5.0659455480...	0.758974	0.758941	0.758941	0.758941	0.758941	0.758947	0.000013	323
497	0.911828	0.017167	0.076563	0.005130	2.460025	0.044408	16	7	7	{'classifier__l2_regularization': 2.4600250010...	0.839907	0.849713	0.846847	0.846028	0.844390	0.845377	0.003234	140
498	1.168120	0.121819	0.061283	0.000760	0.000068	0.287904	227	146	5	{'classifier__l2_regularization': 6.7755366885...	0.861881	0.865001	0.862408	0.859951	0.861862	0.862221	0.001623	33
499	0.823774	0.120686	0.060351	0.014958	0.445218	0.005112	19	8	19	{'classifier__l2_regularization': 0.4452178932...	0.764569	0.765902	0.765902	0.764947	0.765083	0.765281	0.000535	319

500 rows × 18 columns

We define a function to remove the prefixes in the hyperparameters column names.

def shorten_param(param_name):
    if "__" in param_name:
        return param_name.rsplit("__", 1)[1]
    return param_name


cv_results = cv_results.rename(shorten_param, axis=1)
cv_results

	mean_fit_time	std_fit_time	mean_score_time	std_score_time	l2_regularization	learning_rate	max_bins	max_leaf_nodes	min_samples_leaf	params	split0_test_score	split1_test_score	split2_test_score	split3_test_score	split4_test_score	mean_test_score	std_test_score	rank_test_score
0	0.540456	0.062725	0.052069	0.002661	2.467047	0.550075	86	22	6	{'classifier__l2_regularization': 2.4670474863...	0.856558	0.862271	0.857767	0.854491	0.856675	0.857552	0.002586	48
1	1.110536	0.033403	0.074142	0.002165	0.015449	0.001146	60	19	1	{'classifier__l2_regularization': 0.0154488709...	0.758974	0.758941	0.758941	0.758941	0.758941	0.758947	0.000013	323
2	1.137484	0.053150	0.092993	0.029005	1.095093	0.004274	151	17	10	{'classifier__l2_regularization': 1.0950934559...	0.783267	0.758941	0.776413	0.779143	0.758941	0.771341	0.010357	311
3	3.935108	0.202993	0.118105	0.023658	0.003621	0.001305	18	164	37	{'classifier__l2_regularization': 0.0036210968...	0.758974	0.758941	0.758941	0.758941	0.758941	0.758947	0.000013	323
4	0.255219	0.038301	0.056048	0.016736	0.000081	5.407382	97	8	3	{'classifier__l2_regularization': 8.1060737427...	0.758974	0.758941	0.758941	0.758941	0.758941	0.758947	0.000013	323
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
495	0.452411	0.023006	0.055563	0.000846	0.000075	0.364373	92	17	4	{'classifier__l2_regularization': 7.4813767874...	0.858332	0.865001	0.862681	0.860360	0.860770	0.861429	0.002258	34
496	1.133042	0.014456	0.078186	0.002199	5.065946	0.001222	7	17	1	{'classifier__l2_regularization': 5.0659455480...	0.758974	0.758941	0.758941	0.758941	0.758941	0.758947	0.000013	323
497	0.911828	0.017167	0.076563	0.005130	2.460025	0.044408	16	7	7	{'classifier__l2_regularization': 2.4600250010...	0.839907	0.849713	0.846847	0.846028	0.844390	0.845377	0.003234	140
498	1.168120	0.121819	0.061283	0.000760	0.000068	0.287904	227	146	5	{'classifier__l2_regularization': 6.7755366885...	0.861881	0.865001	0.862408	0.859951	0.861862	0.862221	0.001623	33
499	0.823774	0.120686	0.060351	0.014958	0.445218	0.005112	19	8	19	{'classifier__l2_regularization': 0.4452178932...	0.764569	0.765902	0.765902	0.764947	0.765083	0.765281	0.000535	319

500 rows × 18 columns

As we have more than 2 parameters in our randomized-search, we cannot visualize the results using a heatmap. We could still do it pair-wise, but having a two-dimensional projection of a multi-dimensional problem can lead to a wrong interpretation of the scores.

import seaborn as sns
import numpy as np

df = pd.DataFrame(
    {
        "max_leaf_nodes": cv_results["max_leaf_nodes"],
        "learning_rate": cv_results["learning_rate"],
        "score_bin": pd.cut(
            cv_results["mean_test_score"], bins=np.linspace(0.5, 1.0, 6)
        ),
    }
)
sns.set_palette("YlGnBu_r")
ax = sns.scatterplot(
    data=df,
    x="max_leaf_nodes",
    y="learning_rate",
    hue="score_bin",
    s=50,
    color="k",
    edgecolor=None,
)
ax.set_xscale("log")
ax.set_yscale("log")

_ = ax.legend(
    title="mean_test_score", loc="center left", bbox_to_anchor=(1, 0.5)
)

In the previous plot we see that the top performing values are located in a band of learning rate between 0.01 and 1.0, but we have no control in how the other hyperparameters interact with such values for the learning rate. Instead, we can visualize all the hyperparameters at the same time using a parallel coordinates plot.

import numpy as np
import plotly.express as px

fig = px.parallel_coordinates(
    cv_results.rename(shorten_param, axis=1).apply(
        {
            "learning_rate": np.log10,
            "max_leaf_nodes": np.log2,
            "max_bins": np.log2,
            "min_samples_leaf": np.log10,
            "l2_regularization": np.log10,
            "mean_test_score": lambda x: x,
        }
    ),
    color="mean_test_score",
    color_continuous_scale=px.colors.sequential.Viridis,
)
fig.show()