Dimensionality reduction of text data

In the Clustering Chapter we briefly introduced how to deal with text data. There, we presented the concept of vectorization, where we treat each word as a feature (a column), and represent each document as a vector (a row). As this process creates as many features as unique words occurring in the data, the dimension of the feature space can be very large.

In this notebook we use the Wikinews dataset to explore how to reduce that dimensionality for visualization and analysis. This turns out to be trickier than in the tabular setting. The heuristics we used before, such as the 90% variance threshold and the Kaiser criterion, behave very differently on text data and can lead to impractical choices.

We also compare linear and non-linear reduction techniques. Each technique tells a different story about the same data, and knowing when to use which is one of the goals of this notebook.

import pandas as pd

data = pd.read_csv("../datasets/wiki_news.csv")
data

	category	text
0	business	The besieged airline industry received some go...
1	business	The Dow Jones Industrial Average fell more tha...
2	business	The International Criminal Court has presented...
3	business	The online payment service PayPal has received...
4	business	The world's second largest auto parts maker, D...
...	...	...
1245	tech	YouTubeThe online video sharing site YouTube h...
1246	tech	If you've noticed a fuzzy yellowish object at ...
1247	tech	NASA officials decided late Monday to go ahead...
1248	tech	The head of the United States National Nuclear...
1249	tech	Scientists have announced that the largest rad...

1250 rows × 2 columns

In the Clustering Chapter we encoded the text using CountVectorizer first, then we just mentioned that skrub.StringEncoder encodes text similarly to CountVectorizer but it additionally reduces the dimension of the feature space.

In this notebook, we use TfidfVectorizer to vectorize the “text” column. The min_df and max_df hyperparameters discard any word that appears in fewer than 5 documents, or in more than 80% of documents, respectively. The logic is that very rare terms may just be typos, proper nouns, or highly specific terms that won’t generalize across the corpus; whereas words appearing in almost every document may not help distinguish one document from another: articles, conjunctions, auxiliary verbs, etc.

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer = TfidfVectorizer(min_df=5, max_df=0.8)
data_encoded = vectorizer.fit_transform(data["text"])
data_encoded

<Compressed Sparse Row sparse matrix of dtype 'float64'
	with 191581 stored elements and shape (1250, 6678)>

The output from the cell above tells us that, after discarding terms according to the min_df and max_df, we are left with 6,678 unique terms distributed in the 1,250 documents. Most entries are zero, since any given document uses only a small fraction of all possible terms; out of the 1250 × 6678 ≈ 8.3 million possible entries, only ~190,000 are non-zero. Working directly in this 6678-dimensional space is computationally expensive and unnecessary, as most dimensions carry little information and many terms are correlated (synonyms, verb conjugations, etc). Dimensionality reduction, such as PCA, can compress this into a much smaller set of directions that capture the dominant patterns across documents.

Let’s now use PCA to keep just 2 dimensions. But first we define a helper function that allows us to plot the different categories and explore the data structure at a glance.

import textwrap
import plotly.graph_objects as go
from sklearn.decomposition import PCA


def wrap(text, width=80, max_lines=3):
    lines = textwrap.wrap(text, width)
    if len(lines) > max_lines:
        return "<br>".join(lines[:max_lines]) + "..."
    return "<br>".join(lines)


def plot_2d_projection(estimator, data, categories_to_plot):
    X_2d = estimator.fit_transform(data_encoded)

    fig = go.Figure()

    for cat in categories_to_plot:
        idx = data["category"] == cat
        fig.add_trace(
            go.Scatter(
                x=X_2d[idx, 0],
                y=X_2d[idx, 1],
                mode="markers",
                name=cat,
                marker=dict(size=5, opacity=0.6),
                text=data.loc[idx, "text"].apply(wrap),
                hovertemplate="<b>%{text}</b><extra></extra>",
            )
        )

    estimator_name = type(estimator).__name__
    fig.update_layout(
        title=f"TF-IDF + {estimator_name} (2D projection)",
        xaxis_title="PC1",
        yaxis_title="PC2",
        yaxis=dict(scaleanchor="x", scaleratio=1),  # set equal axes
    )
    fig.show(renderer="notebook")


all_categories = data["category"].unique()
pca = PCA(n_components=2)  # no need to set the random seed, can you tell why?
plot_2d_projection(pca, data, all_categories)

All categories crowd near the origin, but “sport” and “tech” extend into distinct regions of the PC space. Their characteristic vocabulary is distinctive enough to pull them apart with 2 components.

Remember also that the First Principal Component carries more variance than the Second Principal Component, and so on. Because of this, the fact that “sport” extends largely in the negative PC1 direction suggests sports vocabulary is the most distinctive across this corpus of documents and is well-captured by that direction. Similarly, “tech” is spread in the positive PC2 direction. This suggests that the words that define PC2 are disproportionately tech-related terms.

“Business” and “entertainment” stay near the center, suggesting their vocabulary is spread across many directions rather than concentrated along the first two components. Let’s focus on “entertainment” and use a pairplot to explore whether higher components better capture the vocabulary specific to this category.

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt


def set_equal_axes(*args, **kwargs):
    plt.xlim(-lim, lim)
    plt.ylim(-lim, lim)
    plt.gca().set_aspect("equal")
    plt.xticks(rotation=45)


n_components = 4
category_of_interest = "entertainment"
pca.set_params(n_components=n_components)
X_pca = pca.fit_transform(data_encoded)
entertainment = pd.DataFrame(
    X_pca[data["category"] == category_of_interest],
    columns=[f"PC{i + 1}" for i in range(n_components)],
)
lim = max(abs(entertainment.values.min()), abs(entertainment.values.max()))
g = sns.PairGrid(entertainment, corner=True, aspect=1.2)
g.map_offdiag(sns.scatterplot, alpha=0.6, s=20)
g.map_diag(sns.histplot)
g.map_offdiag(set_equal_axes)
_ = g.figure.suptitle(
    f"TF-IDF + PCA on {category_of_interest}\n(first {n_components} components)"
)

The first panel shows PC1 vs PC2, which corresponds to the 2D scatter plot we explored earlier with all categories. The cloud is roughly isotropic, with a few outliers pulling away from the center. As neither component dominates, PC1 and PC2 are driven by other categories’ vocabulary, and entertainment articles are essentially scattered at random along those first directions.

From PC3 onward, the scatter panels show a clear diagonal elongation, meaning entertainment articles tend to land on defined regions of these components rather than randomly. This correlation between components suggests they capture coherent vocabulary patterns within the category. The panels involving PC4 show elongation along the PC4 axis, suggesting it is the first component to capture variance specific to entertainment.

Now let’s get back to our 2D scatter plot to visualize “sport” and “entertainment” together in the first two components.

categories_to_plot = ["entertainment", "sport"]
plot_2d_projection(pca, data, categories_to_plot)