📃 Solution for Exercise M7.03

As with the classification metrics exercise, we will evaluate the regression metrics within a cross-validation framework to get familiar with the syntax.

We will use the Ames house prices dataset.

import pandas as pd
import numpy as np

ames_housing = pd.read_csv("../datasets/house_prices.csv")
data = ames_housing.drop(columns="SalePrice")
target = ames_housing["SalePrice"]
data = data.select_dtypes(np.number)
target /= 1000

Note

If you want a deeper overview regarding this dataset, you can refer to the Appendix - Datasets description section at the end of this MOOC.

The first step will be to create a linear regression model.

# solution
from sklearn.linear_model import LinearRegression

model = LinearRegression()

Then, use the cross_val_score to estimate the generalization performance of the model. Use a KFold cross-validation with 10 folds. Make the use of the \(R^2\) score explicit by assigning the parameter scoring (even though it is the default score).

# solution
from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, data, target, cv=10, scoring="r2")
print(f"R2 score: {scores.mean():.3f} ± {scores.std():.3f}")

R2 score: 0.794 ± 0.103

Then, instead of using the \(R^2\) score, use the mean absolute error (MAE). You may need to refer to the documentation for the scoring parameter.

# solution
scores = cross_val_score(
    model, data, target, cv=10, scoring="neg_mean_absolute_error"
)
errors = -scores
print(f"Mean absolute error: {errors.mean():.3f} k$ ± {errors.std():.3f}")

Mean absolute error: 21.892 k$ ± 2.225

The scoring parameter in scikit-learn expects score. It means that the higher the values, and the smaller the errors are, the better the model is. Therefore, the error should be multiplied by -1. That’s why the string given the scoring starts with neg_ when dealing with metrics which are errors.

Finally, use the cross_validate function and compute multiple scores/errors at once by passing a list of scorers to the scoring parameter. You can compute the \(R^2\) score and the mean absolute error for instance.

# solution
from sklearn.model_selection import cross_validate

scoring = ["r2", "neg_mean_absolute_error"]
cv_results = cross_validate(model, data, target, scoring=scoring)

import pandas as pd

scores = {
    "R2": cv_results["test_r2"],
    "MAE": -cv_results["test_neg_mean_absolute_error"],
}
scores = pd.DataFrame(scores)
scores

	R2	MAE
0	0.848721	21.256799
1	0.816374	22.084083
2	0.813513	22.113367
3	0.814138	20.448279
4	0.637473	24.370341

In the Regression Metrics notebook, we introduced the concept of loss function, which is the metric optimized when training a model. In the case of LinearRegression, the fitting process consists in minimizing the mean squared error (MSE). Some estimators, such as HistGradientBoostingRegressor, can use different loss functions, to be set using the loss hyperparameter.

Notice that the evaluation metrics and the loss functions are not necessarily the same. Let’s see an example:

# solution
from collections import defaultdict
from sklearn.ensemble import HistGradientBoostingRegressor

scoring = ["neg_mean_squared_error", "neg_mean_absolute_error"]
loss_functions = ["squared_error", "absolute_error"]
scores = defaultdict(list)

for loss_func in loss_functions:
    model = HistGradientBoostingRegressor(loss=loss_func)
    cv_results = cross_validate(model, data, target, scoring=scoring)
    mse = -cv_results["test_neg_mean_squared_error"]
    mae = -cv_results["test_neg_mean_absolute_error"]
    scores["loss"].append(loss_func)
    scores["MSE"].append(f"{mse.mean():.1f} ± {mse.std():.1f}")
    scores["MAE"].append(f"{mae.mean():.1f} ± {mae.std():.1f}")
scores = pd.DataFrame(scores)
scores.set_index("loss")

	MSE	MAE
loss
squared_error	892.2 ± 243.6	17.6 ± 0.9
absolute_error	923.8 ± 344.6	16.7 ± 1.5

Even if the score distributions overlap due to the presence of outliers in the dataset, it is true that the average MSE is lower when loss="squared_error", whereas the average MAE is lower when loss="absolute_error" as expected. Indeed, the choice of a loss function is made depending on the evaluation metric that we want to optimize for a given use case.

If you feel like going beyond the contents of this MOOC, you can try different combinations of loss functions and evaluation metrics.

Notice that there are some metrics that cannot be directly optimized by optimizing a loss function. This is the case for metrics that evolve in a discontinuous manner with respect to the internal parameters of the model, as learning solvers based on gradient descent or similar optimizers require continuity (the details are beyond the scope of this MOOC).

For instance, classification models are often evaluated using metrics computed on hard class predictions (i.e. whether a sample belongs to a given class) rather than from continuous values such as predict_proba (i.e. the estimated probability of belonging to said given class). Because of this, classifiers are typically trained by optimizing a loss function computed from some continuous output of the model. We call it a “surrogate loss” as it substitutes the metric of interest. For instance LogisticRegression minimizes the log_loss applied to the predict_proba output of the model. By minimizing the surrogate loss, we maximize the accuracy. However scikit-learn does not provide surrogate losses for all possible classification metrics.