Handling Missing Values
1) Drop columns with missing values
The simplest solution. But the model may lose access to some potentially important information.
# Get names of columns with missing values
cols_with_missing = [col for col in X_train.columns
if X_train[col].isnull().any()]
# Drop columns in training and validation data
reduced_X_train = X_train.drop(cols_with_missing, axis=1)
reduced_X_valid = X_valid.drop(cols_with_missing, axis=1)
print("MAE from Approach 1 (Drop columns with missing values):")
print(score_dataset(reduced_X_train, reduced_X_valid, y_train, y_valid))
2) Imputation
Fill in missing values with some numbers. Numbers may not be exact, but it performs better than dropping entire columns.
from sklearn.impute import SimpleImputer
# Imputation
my_imputer = SimpleImputer()
imputed_X_train = pd.DataFrame(my_imputer.fit_transform(X_train))
imputed_X_valid = pd.DataFrame(my_imputer.transform(X_valid))
# Imputation removed column names; put them back
imputed_X_train.columns = X_train.columns
imputed_X_valid.columns = X_valid.columns
print("MAE from Approach 2 (Imputation):")
print(score_dataset(imputed_X_train, imputed_X_valid, y_train, y_valid))
3) Extension to imputation
Provide the model with information on which values are imputed. In some cases, this can be a meaningful improvement, but in other cases, it doesn't help at all.
# Make copy to avoid changing original data (when imputing)
X_valid_plus = X_valid.copy()
X_train_plus = X_train.copy()
# Make new columns indicating what will be imputed
for col in cols_with_missing:
X_train_plus[col + '_was_missing'] = X_train_plus[col].isnull()
X_valid_plus[col + '_was_missing'] = X_valid_plus[col].isnull()
# Imputation
my_imputer = SimpleImputer()
imputed_X_train_plus = pd.DataFrame(my_imputer.fit_transform(X_train_plus))
imputed_X_valid_plus = pd.DataFrame(my_imputer.transform(X_valid_plus))
# Imputation removed column names; put them back
imputed_X_train_plus.columns = X_train_plus.columns
imputed_X_valid_plus.columns = X_valid_plus.columns
print("MAE from Approach 3 (An Extension to Imputation):")
print(score_dataset(imputed_X_train_plus, imputed_X_valid_plus, y_train, y_valid))
Categorical Variables
1) Drop categorical variables
Simply remove and ignore categorical variables. This approach works well when there is a ranking to the categories. Categorical variables that have order in the values are called ordinal variables.
drop_X_train = X_train.select_dtypes(exclude=['object'])
drop_X_valid = X_valid.select_dtypes(exclude=['object'])
print("MAE from Approach 1 (Drop categorical variables):")
print(score_dataset(drop_X_train, drop_X_valid, y_train, y_valid))
2) Ordinal Encoding
Assign unique integer values to each category. This approach works well with ordinal variables which are categorical variables that have order.
from sklearn.preprocessing import OrdinalEncoder
# Make copy to avoid changing original data
label_X_train = X_train.copy()
label_X_valid = X_valid.copy()
# Apply ordinal encoder to each column with categorical data
ordinal_encoder = OrdinalEncoder()
label_X_train[object_cols] = ordinal_encoder.fit_transform(X_train[object_cols])
label_X_valid[object_cols] = ordinal_encoder.transform(X_valid[object_cols])
print("MAE from Approach 2 (Ordinal Encoding):")
print(score_dataset(label_X_train, label_X_valid, y_train, y_valid))
3) One-Hot Encoding
Creates new columns for each value indicating the presence of rach values. This approach works well with variables that do not have a clear ranking also known as nominal variables.
from sklearn.preprocessing import OneHotEncoder
# Apply one-hot encoder to each column with categorical data
OH_encoder = OneHotEncoder(handle_unknown='ignore', sparse=False)
OH_cols_train = pd.DataFrame(OH_encoder.fit_transform(X_train[object_cols]))
OH_cols_valid = pd.DataFrame(OH_encoder.transform(X_valid[object_cols]))
# One-hot encoding removed index; put it back
OH_cols_train.index = X_train.index
OH_cols_valid.index = X_valid.index
# Remove categorical columns (will replace with one-hot encoding)
num_X_train = X_train.drop(object_cols, axis=1)
num_X_valid = X_valid.drop(object_cols, axis=1)
# Add one-hot encoded columns to numerical features
OH_X_train = pd.concat([num_X_train, OH_cols_train], axis=1)
OH_X_valid = pd.concat([num_X_valid, OH_cols_valid], axis=1)
print("MAE from Approach 3 (One-Hot Encoding):")
print(score_dataset(OH_X_train, OH_X_valid, y_train, y_valid))
Pipeline
Pipeline is a way to bundle your preprocessing and modeling steps into one step. Not all data scientist uses pipelines, but they can have some important benefits: Cleaner Code, Fewer Bugs, Easier to Productionize, and More Options for Model Validation. There are three steps to building pipelines.
Step 1: Define preprocessing step
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder
# Preprocessing for numerical data
numerical_transformer = SimpleImputer(strategy='constant')
# Preprocessing for categorical data
categorical_transformer = Pipeline(steps=[
('imputer', SimpleImputer(strategy='most_frequent')),
('onehot', OneHotEncoder(handle_unknown='ignore'))
])
# Bundle preprocessing for numerical and categorical data
preprocessor = ColumnTransformer(
transformers=[
('num', numerical_transformer, numerical_cols),
('cat', categorical_transformer, categorical_cols)
])
Step 2: Define the model
from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(n_estimators=100, random_state=0)
Step 3: Create and evaluate the pipeline
from sklearn.metrics import mean_absolute_error
# Bundle preprocessing and modeling code in a pipeline
my_pipeline = Pipeline(steps=[('preprocessor', preprocessor),
('model', model)
])
# Preprocessing of training data, fit model
my_pipeline.fit(X_train, y_train)
# Preprocessing of validation data, get predictions
preds = my_pipeline.predict(X_valid)
# Evaluate the model
score = mean_absolute_error(y_valid, preds)
print('MAE:', score)
Cross-Validation
Cross-Validation is a process of validating the model on different subsets of data. Validating on fixed sets of validation data can lead to an inaccurate measure of model quality. It may perform well on a given validation set, but it may not for other sets of data. Cross-Validation is used to prevent this.
In cross-validation, the dataset is divided into small subsets of equal size. Each subset is called folds. Then, we run one iteration for each fold. On each iteration, each fold is set as a validation (holdout) and every other fold as training data. This gives us a measure of model quality validated on the entire dataset.
The tradeoff of cross-validation is longer running time. While it may insignificant for a small dataset, cross-validating on larger datasets may take a much longer time. It may also be unnecessary. On larger datasets, a single validation set may be sufficient enough to yield the same scores as cross-validation.
Implementation
Define pipeline first.
from sklearn.ensemble import RandomForestRegressor
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
# Define pipeline first
my_pipeline = Pipeline(steps=[('preprocessor', SimpleImputer()),
('model', RandomForestRegressor(n_estimators=50,
random_state=0))
])
Obtain cross-validation scores with cross_val_score() function from scikit-learn. WE set the number of folds with the cv parameter
from sklearn.model_selection import cross_val_score
# Multiply by -1 since sklearn calculates *negative* MAE
scores = -1 * cross_val_score(my_pipeline, X, y,
cv=5, # number of folds
scoring='neg_mean_absolute_error')
print("MAE scores:\n", scores)
Output:
MAE scores:
[301628.7893587 303164.4782723 287298.331666 236061.84754543
260383.45111427]
Take the average across each iteration.
print("Average MAE score (across experiments):")
print(scores.mean())
Output:
Average MAE score (across experiments):
277707.3795913405