Making clusters part of a supervised pipeline

This notebook explores how K-means clustering can be used as a feature engineering step to improve the performance of a regression model.

Here we use the California Housing dataset, which includes information about the geographic location (latitude and longitude).

Our goal is to predict the median house value (MedHouseVal) using a ridge regression model, to investigate whether adding features derived from applying K-means to geographic coordinates can improve the pipeline’s predictive performance.

from sklearn.datasets import fetch_california_housing

data, target = fetch_california_housing(return_X_y=True, as_frame=True)
target *= 100  # rescale the target in k$

We can first design a predictive pipeline that completly ignores the coordinates:

import pandas as pd
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import Ridge
from sklearn.compose import make_column_transformer

data_train, data_test, target_train, target_test = train_test_split(
    data, target, test_size=0.2, random_state=0
)
geo_columns = ["Latitude", "Longitude"]
model_drop_geo = make_pipeline(
    make_column_transformer(("drop", geo_columns), remainder="passthrough"),
    StandardScaler(),
    Ridge(alpha=1e-12),
)
test_error_drop_geo = -cross_val_score(
    model_drop_geo, data_train, target_train, scoring="neg_mean_absolute_error"
)
print(
    "The test MAE without geographical features is: "
    f"{test_error_drop_geo.mean():.2f} ± {test_error_drop_geo.std():.2f} k$"
)

The test MAE without geographical features is: 57.35 ± 0.53 k$

We observe a Mean Absolute Error of approximately 57k$ when dropping the geographical features.

As seen in the previous notebook, we suspect that the price information may be linked to the distance to the nearest urban center, and proximity to the coast:

import plotly.express as px

fig = px.scatter_map(
    data,
    lat="Latitude",
    lon="Longitude",
    color=target,
    zoom=5,
    height=600,
    labels={"color": "price (k$)"},
)
fig.update_layout(
    mapbox_style="open-street-map",
    mapbox_center={
        "lat": data["Latitude"].mean(),
        "lon": data["Longitude"].mean(),
    },
    margin={"r": 0, "t": 0, "l": 0, "b": 0},
)
fig.show(renderer="notebook")