Data Science Review: Intro to ML III

#blog#datascience#machinelearning

intro-to-ml

Bagging and Boosting

In previous postings, we used the random forest method for prediction, which makes predictions by averaging several different trees. We call this type of method that combines different machine learning models an ensemble method or ensemble learning.

There are two major ways to combine different machine learning models in ensemble learning: bagging and boosting.

bagging and boosting

Bagging trains its component models separately and produces predictions by voting or averaging predictions made by each component model. Random forest is one example of a bagging algorithm.

Boosting, on the other hand, is a method that chains component models so that each model focuses on the errors made by the previous model improving the overall performance. Gradient Boosting is one example of boosting algorithm.

Gradient Boosting

Gradient boosting is a method that goes through cycles to iteratively add models to an ensemble. As the name suggests, gradient boosting minimizes the loss function of the model by adding a new model with its parameter calculated using gradient descent. This will be repeated until the loss function is close to zero or a specific number of the model is created.

gboost

Implementation using XGBoost

XGBoost (eXtreme Gradient Boosting) is a powerful library with the implementation of gradient boost with several additional features for its performance and speed.

  • n_estimators: number of models to add to the ensemble. Typically ranges from 100-1000.

  • early_stopping_rounds: number of rounds of deterioration of validation score before stopping the model. The model will stop before reaching the 'n_estimators' if the model stops improving. 5 is a reasonable choice.

  • eval_set: when using early_stopping_rounds, some data for calculating validation scores is required.

  • learning_rate: the rate at which the predictions from each model affect the final prediction. This allows n_estimators to be set with a larger number without overfitting. In general, small learning_rate and large n_estimators yield an accurate model, but it will take longer to train.

  • n_jobs: number of threads. This is usually set to a number of the core the machine running this calculation has. It is useful for a larger dataset.

from xgboost import XGBRegressor

my_model = XGBRegressor(n_estimators=1000, learning_rate=0.05, n_jobs=4)
my_model.fit(X_train, y_train, 
             early_stopping_rounds=5, 
             eval_set=[(X_valid, y_valid)], 
             verbose=False)

Test parameters

Let's test different values for n_estimators and learning_rate to see how it performs.

First, build function that trains and calculates mean absolute error for given model.

def get_mae(model):
    model.fit(X_train, y_train)
    predictions = model.predict(X_valid)
    mae = mean_absolute_error(y_valid, predictions)
    return mae

Test 1: defualt parameters

# defualt parameters
model_1 = XGBRegressor(random_state=0) 
print("test_1 mae: ", get_mae(model_1))

Output:

test_1 mae:  17662.736729452055

Test 2: high n_estimators, low learning_rate

# n_estimators = 1000, learning rate=0.05
model_2 = XGBRegressor(n_estimators=1000, learning_rate=0.05) 
print("test_2 mae: ", get_mae(model_2))

Output:

test_2 mae:  16688.691513270547

Test 3: low n_estimators, high learning_rate

# n_estimator=100, learning_rate=0.5
model_3 = XGBRegressor(n_estimators=100, learning_rate=0.5) 
print("test_3 mae: ", get_mae(model_3))

Output:

test_3 mae:  20930.964656464042

Data Leakage

In some cases, information about the target that was available for training data may not be available for prediction. The model's performance will be higher in training but fails to achieve the same performance in production. This phenomenon is called data leakage.

  • Target Leakage: occurs when your dataset includes data that will not be available at the time you make predictions. This usually causes unrealistically high model accuracy.

target-leakage

  • Train Test Contamination: occurs when validation data affects the preprocessing of training data. Ex) Fitting imputer before splitting training and validation data.

Links