KGP Talkie

Cardinality in Machine Learning

Understand cardinality in categorical variables and its effect on model performance. Learn to handle high-cardinality features using Python techniques.

Sep 29, 2020Updated Jul 12, 202629 min readFollow

Topics You Will Master

What cardinality means for categorical features
How high-cardinality variables impact model performance and memory
Techniques for grouping rare labels and reducing cardinality
Encoding high-cardinality features using Python

When we prepare categorical variables for a model, one key thing to check is the number of unique categories or labels.

The number of unique labels within a categorical variable is known as its cardinality. A high number of labels within a variable is referred to as high cardinality. High-cardinality features (like zip codes, cities, or IP addresses) are hard for many models, and for decision trees in particular.

In this blog, we will explore the concept of cardinality using the Titanic dataset. We will learn how to measure cardinality and spot the problems it causes during train-test splits. We will see its impact on algorithms like Random Forests, AdaBoost, Logistic Regression, and Gradient Boosting. Then we apply a deck-based grouping technique to reduce cardinality and prevent overfitting.

Prerequisites: Python 3.x, Pandas, Scikit-learn.

High Cardinality Challenges

High cardinality in a category column can cause several problems for a model:

  • Tree Bias: Tree-based algorithms tend to favor variables with many levels over those with few levels, regardless of their actual predictive power.
  • Overfitting and Noise: Variables with too many labels introduce high-frequency noise with little information, making the model prone to memorizing the training set.
  • Train-Test Inconsistency: Some labels may appear only in the training set, which causes overfitting. Other labels may appear only in the test set, so the model cannot predict on them.

We will demonstrate these effects below using the Titanic dataset to predict passenger survival.

Setup and Libraries

Before we process the dataset, import the libraries for data handling, model building, and evaluation:

PYTHON 
# to read the dataset into a dataframe and perform operations on it
import pandas as pd

# to perform basic array operations
import numpy as np

# to build machine learning models
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import AdaBoostClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier

# to evaluate the models
from sklearn.metrics import roc_auc_score

# to separate data into train and test
from sklearn.model_selection import train_test_split

Load the Titanic dataset using Pandas:

PYTHON 
data = pd.read_csv('https://raw.githubusercontent.com/laxmimerit/All-CSV-ML-Data-Files-Download/master/titanic.csv')
data.head()
OUTPUT
PassengerIdSurvivedPclassNameSexAgeSibSpParchTicketFareCabinEmbarked
0103Braund, Mr. Owen Harrismale22.010A/5 211717.2500NaNS
1211Cumings, Mrs. John Bradley (Florence Briggs Th...female38.010PC 1759971.2833C85C
2313Heikkinen, Miss. Lainafemale26.000STON/O2. 31012827.9250NaNS
3411Futrelle, Mrs. Jacques Heath (Lily May Peel)female35.01011380353.1000C123S
4503Allen, Mr. William Henrymale35.0003734508.0500NaNS

Check the cardinality (the number of unique categories) of each category column in the dataset:

PYTHON 
print('Number of categories in the variable Name: {}'.format(
    len(data.Name.unique())))

print('Number of categories in the variable Gender: {}'.format(
    len(data.Sex.unique())))

print('Number of categories in the variable Ticket: {}'.format(
    len(data.Ticket.unique())))

print('Number of categories in the variable Cabin: {}'.format(
    len(data.Cabin.unique())))

print('Number of categories in the variable Embarked: {}'.format(
    len(data.Embarked.unique())))

print('Total number of passengers in the Titanic: {}'.format(len(data)))
OUTPUT 
Number of categories in the variable Name: 891
Number of categories in the variable Gender: 2
Number of categories in the variable Ticket: 681
Number of categories in the variable Cabin: 148
Number of categories in the variable Embarked: 4
Total number of passengers in the Titanic: 891

The printout confirms that while Sex (2 categories) and Embarked (4 categories) have low cardinality, Ticket (681 categories), Name (891 categories), and Cabin (148 categories) exhibit high cardinality.

To demonstrate the impact of cardinality, we will focus on the Cabin column. Display the unique values of Cabin to see their original structure:

PYTHON 
data.Cabin.unique()
OUTPUT 
array([nan, 'C85', 'C123', 'E46', 'G6', 'C103', 'D56', 'A6',
       'C23 C25 C27', 'B78', 'D33', 'B30', 'C52', 'B28', 'C83', 'F33',
       'F G73', 'E31', 'A5', 'D10 D12', 'D26', 'C110', 'B58 B60', 'E101',
       'F E69', 'D47', 'B86', 'F2', 'C2', 'E33', 'B19', 'A7', 'C49', 'F4',
       'A32', 'B4', 'B80', 'A31', 'D36', 'D15', 'C93', 'C78', 'D35',
       'C87', 'B77', 'E67', 'B94', 'C125', 'C99', 'C118', 'D7', 'A19',
       'B49', 'D', 'C22 C26', 'C106', 'C65', 'E36', 'C54',
       'B57 B59 B63 B66', 'C7', 'E34', 'C32', 'B18', 'C124', 'C91', 'E40',
       'T', 'C128', 'D37', 'B35', 'E50', 'C82', 'B96 B98', 'E10', 'E44',
       'A34', 'C104', 'C111', 'C92', 'E38', 'D21', 'E12', 'E63', 'A14',
       'B37', 'C30', 'D20', 'B79', 'E25', 'D46', 'B73', 'C95', 'B38',
       'B39', 'B22', 'C86', 'C70', 'A16', 'C101', 'C68', 'A10', 'E68',
       'B41', 'A20', 'D19', 'D50', 'D9', 'A23', 'B50', 'A26', 'D48',
       'E58', 'C126', 'B71', 'B51 B53 B55', 'D49', 'B5', 'B20', 'F G63',
       'C62 C64', 'E24', 'C90', 'C45', 'E8', 'B101', 'D45', 'C46', 'D30',
       'E121', 'D11', 'E77', 'F38', 'B3', 'D6', 'B82 B84', 'D17', 'A36',
       'B102', 'B69', 'E49', 'C47', 'D28', 'E17', 'A24', 'C50', 'B42',
       'C148'], dtype=object)

We will reduce the cardinality of Cabin by extracting only the first letter. The first letter indicates the deck (e.g., A, B, C) where the cabin was located, which acts as a proxy for social class and proximity to the ship's surface:

PYTHON 
# extract first letter of Cabin
data['Cabin_reduced'] = data['Cabin'].astype(str).str[0]

data[['Cabin', 'Cabin_reduced']].head()
OUTPUT
CabinCabin_reduced
0NaNn
1C85C
2NaNn
3C123C
4NaNn

Verify the new cardinality of the reduced Cabin_reduced variable:

PYTHON 
print('Number of categories in the variable Cabin: {}'.format(
    len(data.Cabin.unique())))

print('Number of categories in the variable Cabin_reduced: {}'.format(
    len(data.Cabin_reduced.unique())))
OUTPUT 
Number of categories in the variable Cabin: 148
Number of categories in the variable Cabin_reduced: 9

Splitting the first letter reduces the cardinality from 148 categories down to just 9.

Split the dataset into training (70%) and testing (30%) sets:

PYTHON 
use_cols = ['Cabin', 'Cabin_reduced', 'Sex']

X_train, X_test, y_train, y_test = train_test_split(
    data[use_cols],
    data['Survived'],
    test_size=0.3,
    random_state=0)

X_train.shape, X_test.shape
OUTPUT 
((623, 3), (268, 3))

Inconsistent Category Distributions

When a variable has high cardinality, categories often land in the training set but not the test set, or the other way around. Count the categories in Cabin that exist only in the training set:

PYTHON 
unique_to_train_set = [
    x for x in X_train.Cabin.unique() if x not in X_test.Cabin.unique()
]

len(unique_to_train_set)
OUTPUT 
100

There are 100 cabins that only appear in the training dataset. Now count the categories that only exist in the testing dataset:

PYTHON 
unique_to_test_set = [
    x for x in X_test.Cabin.unique() if x not in X_train.Cabin.unique()
]

len(unique_to_test_set)
OUTPUT 
28

Repeat the same analysis on the Cabin_reduced variable to evaluate the impact of reduced cardinality:

PYTHON 
unique_to_train_set = [
    x for x in X_train['Cabin_reduced'].unique()
    if x not in X_test['Cabin_reduced'].unique()
]

len(unique_to_train_set)
OUTPUT 
1

Count the reduced categories that appear only in the testing set:

PYTHON 
unique_to_test_set = [
    x for x in X_test['Cabin_reduced'].unique()
    if x not in X_train['Cabin_reduced'].unique()
]

len(unique_to_test_set)
OUTPUT 
0

By reducing cardinality, only 1 category remains unique to the training set, and no categories are unique to the test set.

Encoding and Mapping Categorical Variables

To train models, we must map our string categories to integer values. Create an integer mapping dictionary for the original Cabin feature:

PYTHON 
import itertools

cabin_dict = {k: i for i, k in enumerate(X_train['Cabin'].unique(), 0)}
print(dict(itertools.islice(cabin_dict.items(), 100)))
OUTPUT 
{'E17': 0, 'D33': 1, nan: 2, 'D26': 3, 'B58 B60': 4, 'C128': 5, 'D17': 6, 'A14': 7, 'F33': 8, 'B19': 9, 'D21': 10, 'C148': 11, 'C30': 12, 'D56': 13, 'E24': 14, 'E40': 15, 'E31': 16, 'E44': 17, 'E38': 18, 'D37': 19, 'E8': 20, 'C92': 21, 'E63': 22, 'C125': 23, 'F4': 24, 'E67': 25, 'C126': 26, 'B73': 27, 'E36': 28, 'C78': 29, 'E46': 30, 'C111': 31, 'E101': 32, 'D15': 33, 'E12': 34, 'G6': 35, 'A32': 36, 'B4': 37, 'A10': 38, 'A5': 39, 'C95': 40, 'E25': 41, 'C90': 42, 'D6': 43, 'A36': 44, 'D': 45, 'D50': 46, 'B96 B98': 47, 'C93': 48, 'E77': 49, 'C101': 50, 'D11': 51, 'C123': 52, 'C32': 53, 'B35': 54, 'C91': 55, 'T': 56, 'B101': 57, 'E58': 58, 'A23': 59, 'B77': 60, 'D28': 61, 'B82 B84': 62, 'B79': 63, 'C45': 64, 'C2': 65, 'B5': 66, 'C104': 67, 'B20': 68, 'A19': 69, 'B51 B53 B55': 70, 'B80': 71, 'B38': 72, 'B22': 73, 'B18': 74, 'C22 C26': 75, 'A16': 76, 'F2': 77, 'D47': 78, 'E121': 79, 'C23 C25 C27': 80, 'B28': 81, 'E10': 82, 'D36': 83, 'C46': 84, 'B39': 85, 'D30': 86, 'E33': 87, 'C50': 88, 'D20': 89, 'C124': 90, 'A34': 91, 'C110': 92, 'D19': 93, 'B86': 94, 'D35': 95, 'C99': 96, 'D46': 97, 'F38': 98, 'A24': 99}

Map the string values to integers inside both train and test sets:

PLAINTEXT 
X_train.loc[:, 'Cabin_mapped'] = X_train.loc[:, 'Cabin'].map(cabin_dict)
X_test.loc[:, 'Cabin_mapped'] = X_test.loc[:, 'Cabin'].map(cabin_dict)

X_train[['Cabin_mapped', 'Cabin']].head(10)
OUTPUT
Cabin_mappedCabin
8570E17
521D33
3862NaN
1243D26
5782NaN
5492NaN
1184B58 B60
122NaN
1572NaN
1272NaN

Repeat the mapping process on the Cabin_reduced variable:

PYTHON 
# create replace dictionary
cabin_dict = {k: i for i, k in enumerate(X_train['Cabin_reduced'].unique(), 0)}

# replace labels by numbers with dictionary
X_train.loc[:, 'Cabin_reduced'] = X_train.loc[:, 'Cabin_reduced'].map(cabin_dict)
X_test.loc[:, 'Cabin_reduced'] = X_test.loc[:, 'Cabin_reduced'].map(cabin_dict)

X_train[['Cabin_reduced', 'Cabin']].head(20)
OUTPUT
Cabin_reducedCabin
8570E17
521D33
3862NaN
1241D26
5782NaN
5492NaN
1183B58 B60
122NaN
1572NaN
1272NaN
6532NaN
2352NaN
7852NaN
2412NaN
3514C128
8621D17
8512NaN
7532NaN
5322NaN
4852NaN

Map the categorical Sex column to binary values:

PYTHON 
X_train.loc[:, 'Sex'] = X_train.loc[:, 'Sex'].map({'male': 0, 'female': 1})
X_test.loc[:, 'Sex'] = X_test.loc[:, 'Sex'].map({'male': 0, 'female': 1})

X_train.Sex.head()
OUTPUT 
857    0
52     1
386    0
326    0
578    1
Name: Sex, dtype: int64

Verify missing values in the processed training set columns:

PYTHON 
X_train[['Cabin_mapped', 'Cabin_reduced', 'Sex']].isnull().sum()
OUTPUT 
Cabin_mapped     0
Cabin_reduced    0
Sex              0
dtype: int64

Verify missing values in the test set columns:

PYTHON 
X_test[['Cabin_mapped', 'Cabin_reduced', 'Sex']].isnull().sum()
OUTPUT 
Cabin_mapped     30
Cabin_reduced     0
Sex               0
dtype: int64

The high-cardinality Cabin_mapped feature adds 30 missing values in the test set. The categories that only appear in the test set were not in the dictionary we built from the training set, so they map to NaN.

Show the count of unique classes in each feature:

PYTHON 
len(X_train.Cabin_mapped.unique()), len(X_train.Cabin_reduced.unique())
OUTPUT 
(121, 9)

Model Evaluation

We will now train several models to see how cardinality affects their performance and how well they generalize.

Random Forests

Train a Random Forest classifier using the high-cardinality Cabin_mapped feature:

PYTHON 
# call the model
rf = RandomForestClassifier(n_estimators=200, random_state=39)

# train the model
rf.fit(X_train[['Cabin_mapped', 'Sex']], y_train)

# make predictions on train and test set
pred_train = rf.predict_proba(X_train[['Cabin_mapped', 'Sex']])
pred_test = rf.predict_proba(X_test[['Cabin_mapped', 'Sex']].fillna(0))

print('Train set')
print('Random Forests roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))
print('Test set')
print('Random Forests roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))
OUTPUT 
Train set
Random Forests roc-auc: 0.8617329342096702
Test set
Random Forests roc-auc: 0.8078571428571428

The model scores higher on the training set (0.86) than the test set (0.80), which shows overfitting.

Train the Random Forest classifier using the reduced cardinality Cabin_reduced feature:

PYTHON 
# call the model
rf = RandomForestClassifier(n_estimators=200, random_state=39)

# train the model
rf.fit(X_train[['Cabin_reduced', 'Sex']], y_train)

# make predictions on train and test set
pred_train = rf.predict_proba(X_train[['Cabin_reduced', 'Sex']])
pred_test = rf.predict_proba(X_test[['Cabin_reduced', 'Sex']])

print('Train set')
print('Random Forests roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))
print('Test set')
print('Random Forests roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))
OUTPUT 
Train set
Random Forests roc-auc: 0.8199550985878832
Test set
Random Forests roc-auc: 0.8332142857142857

Reducing cardinality lowers overfitting. So the model generalizes better and gets a higher ROC-AUC on the test set.

AdaBoost

Train an AdaBoost classifier using the high-cardinality feature:

PYTHON 
# call the model
ada = AdaBoostClassifier(n_estimators=200, random_state=44)

# train the model
ada.fit(X_train[['Cabin_mapped', 'Sex']], y_train)

# make predictions on train and test set
pred_train = ada.predict_proba(X_train[['Cabin_mapped', 'Sex']])
pred_test = ada.predict_proba(X_test[['Cabin_mapped', 'Sex']].fillna(0))

print('Train set')
print('Adaboost roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))
print('Test set')
print('Adaboost roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))
OUTPUT 
Train set
Adaboost roc-auc: 0.8399546647578144
Test set
Adaboost roc-auc: 0.809375

Train the AdaBoost classifier using the reduced cardinality feature:

PYTHON 
# call the model
ada = AdaBoostClassifier(n_estimators=200, random_state=44)

# train the model
ada.fit(X_train[['Cabin_reduced', 'Sex']], y_train)

# make predictions on train and test set
pred_train = ada.predict_proba(X_train[['Cabin_reduced', 'Sex']])
pred_test = ada.predict_proba(X_test[['Cabin_reduced', 'Sex']].fillna(0))

print('Train set')
print('Adaboost roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))
print('Test set')
print('Adaboost roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))
OUTPUT 
Train set
Adaboost roc-auc: 0.8195863430294354
Test set
Adaboost roc-auc: 0.8332142857142857

Like Random Forests, the AdaBoost model trained on the reduced variable generalizes better and overfits less.

Logistic Regression

Train a Logistic Regression model using the high-cardinality feature:

PYTHON 
# call the model
logit = LogisticRegression(random_state=44, solver='lbfgs')

# train the model
logit.fit(X_train[['Cabin_mapped', 'Sex']], y_train)

# make predictions on train and test set
pred_train = logit.predict_proba(X_train[['Cabin_mapped', 'Sex']])
pred_test = logit.predict_proba(X_test[['Cabin_mapped', 'Sex']].fillna(0))

print('Train set')
print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))
print('Test set')
print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))
OUTPUT 
Train set
Logistic regression roc-auc: 0.8094564109238411
Test set
Logistic regression roc-auc: 0.7591071428571431

Train the Logistic Regression model using the reduced cardinality feature:

PYTHON 
# call the model
logit = LogisticRegression(random_state=44, solver='lbfgs')

# train the model
logit.fit(X_train[['Cabin_reduced', 'Sex']], y_train)

# make predictions on train and test set
pred_train = logit.predict_proba(X_train[['Cabin_reduced', 'Sex']])
pred_test = logit.predict_proba(X_test[['Cabin_reduced', 'Sex']].fillna(0))

print('Train set')
print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))
print('Test set')
print('Logistic regression roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))
OUTPUT 
Train set
Logistic regression roc-auc: 0.7672664367367301
Test set
Logistic regression roc-auc: 0.7957738095238095

Gradient Boosted Classifier

Train a Gradient Boosting classifier using the high-cardinality feature:

PYTHON 
# call the model
gbc = GradientBoostingClassifier(n_estimators=300, random_state=44)

# train the model
gbc.fit(X_train[['Cabin_mapped', 'Sex']], y_train)

# make predictions on train and test set
pred_train = gbc.predict_proba(X_train[['Cabin_mapped', 'Sex']])
pred_test = gbc.predict_proba(X_test[['Cabin_mapped', 'Sex']].fillna(0))

print('Train set')
print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))
print('Test set')
print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))
OUTPUT 
Train set
Gradient Boosted Trees roc-auc: 0.8731860480249887
Test set
Gradient Boosted Trees roc-auc: 0.816845238095238

Train the Gradient Boosting classifier using the reduced cardinality feature:

PYTHON 
# model trained on reduced-cardinality Cabin_reduced feature

# call the model
gbc = GradientBoostingClassifier(n_estimators=300, random_state=44)

# train the model
gbc.fit(X_train[['Cabin_reduced', 'Sex']], y_train)

# make predictions on train and test set
pred_train = gbc.predict_proba(X_train[['Cabin_reduced', 'Sex']])
pred_test = gbc.predict_proba(X_test[['Cabin_reduced', 'Sex']].fillna(0))

print('Train set')
print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_train, pred_train[:,1])))
print('Test set')
print('Gradient Boosted Trees roc-auc: {}'.format(roc_auc_score(y_test, pred_test[:,1])))
OUTPUT 
Train set
Gradient Boosted Trees roc-auc: 0.8204756946703976
Test set
Gradient Boosted Trees roc-auc: 0.8332142857142857

We see that reducing the cardinality of the variable gives a big boost on unseen data for every model we tried.

Conclusion

In this blog, we explored how high-cardinality categorical variables affect the performance of machine learning algorithms. Using the Titanic dataset, we saw how high cardinality spreads categories unevenly across train-test splits. It also adds null values when we encode, and it leads to severe overfitting. By grouping cabin values into their decks, we reduced cardinality from 148 to 9 categories. This improved how well every classifier generalized.

Key takeaways:

  • High Cardinality: Features with too many categories often cause models to overfit by memorizing noise instead of learning meaningful patterns.
  • Data Leakage and Splitting Issues: With high cardinality, some categories land in only one split. This leads to encoding and prediction errors.
  • Grouping and Dimensionality Reduction: We can group categories using domain knowledge, such as taking the first letter of a cabin ID. This is a strong way to reduce cardinality and boost model performance.

Next steps:

Found this useful? Keep building with me.

New tutorials every week on YouTube: or go deeper with a full structured course.

Subscribe on YouTubeBrowse Udemy CoursesJoin the Newsletter
Related Tutorials

Latest recommendations you might like

Missing Values and Their Mechanisms
Machine Learning

Missing Values and Their Mechanisms

Understand MCAR, MAR, and MNAR missing data mechanisms and their impact on machine learning. Covers detection, analysis, and treatment strategies using Python.

Sep 28, 2020Read Tutorial
Feature Engineering: Variable Magnitude
Machine Learning

Feature Engineering: Variable Magnitude

Understand the impact of feature magnitude on ML algorithms, and learn scaling techniques in Python including Standard, MinMax, and Robust scaling.

Oct 4, 2020Read Tutorial
Feature Engineering: Outlier Detection
Machine Learning

Feature Engineering: Outlier Detection

Detect and handle outliers in Python using IQR and Z-score methods, with boxplot and Q-Q plot visualization and practical boundary calculations.

Oct 3, 2020Read Tutorial
Feature Engineering: Linear Model Assumptions
Machine Learning

Feature Engineering: Linear Model Assumptions

Detect and fix violations of linear model assumptions: linearity, normality, homoscedasticity, and multicollinearity, with Q-Q plots and log transforms.

Oct 2, 2020Read Tutorial

Find this tutorial useful?

Subscribe to our YouTube channels for more practical production walk-throughs.

Watch in EnglishWatch in Hindi

Discussion & Comments