A few weeks ago, I introduced a model-agnostic gradient boosting (XGBoost, LightGBM, CatBoost-like) procedure for supervised regression and classififcatin, that can use any base learner (available in R and Python package mlsauce):

• https://thierrymoudiki.github.io/blog/2024/10/06/python/r/genericboosting
• https://thierrymoudiki.github.io/blog/2024/10/14/r/genericboosting-r

The rationale is different from other histogram-based gradient boosting algorithms, as histograms are only used here for feature engineering of continuous features. So far, I don’t see huge differences with the original implementation of the GenericBooster, but it’s still a work in progress. I envisage to try it out on a data set that contains a ‘higher’ mix of continuous and categorical features (as categorical features are not histogram-engineered).

Here are a few results that can give you an idea of the performance of the algorithm. Keep in mind that the models are not tuned, and that the GenericBooster can be tuned (in addition to the boosting model’s hyperparamters) with the base learner’s hyperparameters. That makes, potentially, a lot of degrees of freedom and room for improvement/exploration.

!pip install git+https://github.com/Techtonique/mlsauce.git --verbose --upgrade --no-cache-dir

import os
import mlsauce as ms
from sklearn.datasets import load_breast_cancer, load_iris, load_wine, load_digits
from sklearn.model_selection import train_test_split
from time import time

load_models = [load_breast_cancer, load_iris, load_wine, load_digits]

for model in load_models:

    data = model()
    X = data.data
    y= data.target

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 13)

    clf = ms.LazyBoostingClassifier(verbose=0, ignore_warnings=True, #n_jobs=2,
                                    custom_metric=None, preprocess=False)

    start = time()
    models, predictioms = clf.fit(X_train, X_test, y_train, y_test, hist=True)
    models2, predictioms = clf.fit(X_train, X_test, y_train, y_test, hist=False)
    print(f"\nElapsed: {time() - start} seconds\n")

    display(models)
    display(models2)

2it [00:00,  2.27it/s]
100%|██████████| 38/38 [00:41<00:00,  1.09s/it]
2it [00:00,  5.14it/s]
100%|██████████| 38/38 [00:43<00:00,  1.14s/it]


Elapsed: 85.95083284378052 seconds

2it [00:00,  6.46it/s]
100%|██████████| 38/38 [00:12<00:00,  3.11it/s]
2it [00:00, 10.38it/s]
100%|██████████| 38/38 [00:11<00:00,  3.18it/s]


Elapsed: 24.71835470199585 seconds

2it [00:00,  5.45it/s]
100%|██████████| 38/38 [00:14<00:00,  2.63it/s]
2it [00:00,  9.26it/s]
100%|██████████| 38/38 [00:14<00:00,  2.58it/s]


Elapsed: 29.76035761833191 seconds

2it [00:01,  1.90it/s]
100%|██████████| 38/38 [09:30<00:00, 15.02s/it]
2it [00:01,  1.03it/s]
100%|██████████| 38/38 [09:27<00:00, 14.94s/it]


Elapsed: 1141.7054164409637 seconds