📃 Solution for Exercise M6.01

The aim of this notebook is to investigate if we can tune the hyperparameters of a bagging regressor and evaluate the gain obtained.

We will load the California housing dataset and split it into a training and a testing set.

from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split

data, target = fetch_california_housing(as_frame=True, return_X_y=True)
target *= 100  # rescale the target in k$
data_train, data_test, target_train, target_test = train_test_split(
    data, target, random_state=0, test_size=0.5
)

Note

If you want a deeper overview regarding this dataset, you can refer to the Appendix - Datasets description section at the end of this MOOC.

Create a BaggingRegressor and provide a DecisionTreeRegressor to its parameter estimator. Train the regressor and evaluate its generalization performance on the testing set using the mean absolute error.

# solution
from sklearn.metrics import mean_absolute_error
from sklearn.tree import DecisionTreeRegressor
from sklearn.ensemble import BaggingRegressor

tree = DecisionTreeRegressor()
bagging = BaggingRegressor(estimator=tree, n_jobs=2)
bagging.fit(data_train, target_train)
target_predicted = bagging.predict(data_test)
print(
    "Basic mean absolute error of the bagging regressor:\n"
    f"{mean_absolute_error(target_test, target_predicted):.2f} k$"
)

Basic mean absolute error of the bagging regressor:
36.53 k$

Now, create a RandomizedSearchCV instance using the previous model and tune the important parameters of the bagging regressor. Find the best parameters and check if you are able to find a set of parameters that improve the default regressor still using the mean absolute error as a metric.

Tip

You can list the bagging regressor’s parameters using the get_params method.

# solution
for param in bagging.get_params().keys():
    print(param)

bootstrap
bootstrap_features
estimator__ccp_alpha
estimator__criterion
estimator__max_depth
estimator__max_features
estimator__max_leaf_nodes
estimator__min_impurity_decrease
estimator__min_samples_leaf
estimator__min_samples_split
estimator__min_weight_fraction_leaf
estimator__monotonic_cst
estimator__random_state
estimator__splitter
estimator
max_features
max_samples
n_estimators
n_jobs
oob_score
random_state
verbose
warm_start

from scipy.stats import randint
from sklearn.model_selection import RandomizedSearchCV

param_grid = {
    "n_estimators": randint(10, 30),
    "max_samples": [0.5, 0.8, 1.0],
    "max_features": [0.5, 0.8, 1.0],
    "estimator__max_depth": randint(3, 10),
}
search = RandomizedSearchCV(
    bagging, param_grid, n_iter=20, scoring="neg_mean_absolute_error"
)
_ = search.fit(data_train, target_train)

/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/numpy/ma/core.py:2846: RuntimeWarning: invalid value encountered in cast
  _data = np.array(data, dtype=dtype, copy=copy,

import pandas as pd

columns = [f"param_{name}" for name in param_grid.keys()]
columns += ["mean_test_error", "std_test_error"]
cv_results = pd.DataFrame(search.cv_results_)
cv_results["mean_test_error"] = -cv_results["mean_test_score"]
cv_results["std_test_error"] = cv_results["std_test_score"]
cv_results[columns].sort_values(by="mean_test_error")

	param_n_estimators	param_max_samples	param_max_features	param_estimator__max_depth	mean_test_error	std_test_error
6	28	0.8	1.0	9	39.417639	0.997963
3	13	1.0	0.8	9	39.476256	1.247757
18	13	1.0	0.8	8	40.719825	1.318149
0	29	0.8	0.8	7	42.350030	1.222091
11	22	0.8	0.8	7	42.986859	1.617616
15	21	1.0	0.5	9	44.540225	1.279482
19	25	0.8	0.5	9	44.882318	1.681919
4	12	1.0	0.5	9	45.051546	2.864171
8	27	0.5	0.5	8	46.603009	2.008999
5	11	1.0	1.0	5	48.289698	0.987867
13	29	0.5	0.5	6	48.620279	1.538058
9	16	0.8	0.8	5	48.813829	1.364801
17	22	0.5	0.5	6	49.762982	1.662406
1	17	1.0	0.5	5	52.231828	1.535115
10	20	0.5	0.8	4	52.650240	0.973787
14	20	0.8	0.5	5	52.722602	1.456663
12	19	0.5	0.5	5	53.159672	2.687279
7	26	1.0	1.0	3	56.819449	1.034640
16	10	0.5	0.8	3	56.855750	0.262953
2	26	0.8	0.8	3	57.579853	1.037738

target_predicted = search.predict(data_test)
print(
    "Mean absolute error after tuning of the bagging regressor:\n"
    f"{mean_absolute_error(target_test, target_predicted):.2f} k$"
)

Mean absolute error after tuning of the bagging regressor:
39.51 k$

We see that the predictor provided by the bagging regressor does not need much hyperparameter tuning compared to a single decision tree.