A classifier that’s very accurate (and deep)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Version v0.15.0
of nnetsauce
is now available on your favorite platforms: PyPI, conda and GitHub for Python; R universe and GitHub for R.
The changes in this version are mostly related to Automated Machine learning (AutoML):
- lazy prediction for classification and regression: see this post for more details on the subject (and remember to use
pip install nnetsauce
directly, instead of installing from a GitHub branch namedlazy-predict
) - lazy prediction for multivariate time series (MTS): see this post for more details on the subject (and remember to use
pip install nnetsauce
directly, instead of installing from a GitHub branch namedlazy-predict
) - lazy prediction with deep quasi-randomized nnetworks will be described in this post
Note that in the example below, for the offically released version (v0.15.0), Gradient boosting classifiers are available. This doesn’t change the best model chosen by the algorithm (which is never Gradient boosting, because the deep model is already doing “too much”). Not sure if Gradient boosting will be kept as a default model here because, in this context, it’s relatively slow.
To finish, for Windows users, if you run into issues when trying to install nnetsauce
(but you shouldn’t): remember that you can use the Windows Subsystem for Linux (WSL).
Contents
- 0 – Install and import packages
- 1 – breast cancer data
- 2 – iris data
- 3 – wine data
- 4 – digits data
- 5 – R examples
Here is a jupyter notebook allowing you to reproduce these results. Do not hesitate
to modify these examples by choosing – in LazyDeepClassifier
– a different number of layers n_layers
, or the number of
engineered features per layer, n_hidden_features
.
0 – Install and import packages
!pip install nnetsauce --upgrade import os import nnetsauce as ns import matplotlib.pyplot as plt from sklearn.datasets import load_breast_cancer, load_iris, load_wine, load_digits from sklearn.model_selection import train_test_split from sklearn.metrics import classification_report, ConfusionMatrixDisplay from time import time
1 – breast cancer data
data = load_breast_cancer() X = data.data y= data.target X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123) clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True) start = time() models, predictions = clf.fit(X_train, X_test, y_train, y_test) print(f"\n\n Elapsed: {time()-start} seconds \n") model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test) display(models) 100%|██████████| 27/27 [00:44<00:00, 1.65s/it] Elapsed: 44.49836850166321 seconds
Accuracy | Balanced Accuracy | ROC AUC | F1 Score | Time Taken | |
---|---|---|---|---|---|
Model | |||||
Perceptron | 0.99 | 0.99 | 0.99 | 0.99 | 2.15 |
SGDClassifier | 0.98 | 0.98 | 0.98 | 0.98 | 1.99 |
RandomForestClassifier | 0.98 | 0.98 | 0.98 | 0.98 | 2.76 |
PassiveAggressiveClassifier | 0.98 | 0.98 | 0.98 | 0.98 | 1.89 |
LogisticRegression | 0.98 | 0.98 | 0.98 | 0.98 | 1.14 |
ExtraTreesClassifier | 0.98 | 0.98 | 0.98 | 0.98 | 3.13 |
BaggingClassifier | 0.97 | 0.97 | 0.97 | 0.97 | 1.62 |
LabelPropagation | 0.97 | 0.97 | 0.97 | 0.97 | 1.69 |
LabelSpreading | 0.97 | 0.97 | 0.97 | 0.97 | 1.08 |
AdaBoostClassifier | 0.97 | 0.97 | 0.97 | 0.97 | 3.42 |
LinearSVC | 0.97 | 0.97 | 0.97 | 0.97 | 1.65 |
KNeighborsClassifier | 0.97 | 0.97 | 0.97 | 0.97 | 1.07 |
SVC | 0.97 | 0.96 | 0.96 | 0.97 | 1.07 |
CalibratedClassifierCV | 0.97 | 0.96 | 0.96 | 0.97 | 1.82 |
DecisionTreeClassifier | 0.96 | 0.96 | 0.96 | 0.96 | 1.50 |
QuadraticDiscriminantAnalysis | 0.96 | 0.95 | 0.95 | 0.96 | 1.16 |
LinearDiscriminantAnalysis | 0.96 | 0.95 | 0.95 | 0.96 | 1.39 |
ExtraTreeClassifier | 0.96 | 0.94 | 0.94 | 0.96 | 0.83 |
RidgeClassifier | 0.96 | 0.94 | 0.94 | 0.96 | 3.15 |
RidgeClassifierCV | 0.96 | 0.94 | 0.94 | 0.96 | 1.51 |
GaussianNB | 0.93 | 0.90 | 0.90 | 0.93 | 1.50 |
NearestCentroid | 0.93 | 0.90 | 0.90 | 0.93 | 3.00 |
NuSVC | 0.93 | 0.90 | 0.90 | 0.93 | 1.27 |
BernoulliNB | 0.91 | 0.89 | 0.89 | 0.91 | 1.15 |
DummyClassifier | 0.64 | 0.50 | 0.50 | 0.50 | 0.96 |
model_dictionary["Perceptron"]
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=Perceptron(random_state=42))))In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=Perceptron(random_state=42))))
CustomClassifier(obj=CustomClassifier(obj=Perceptron(random_state=42)))
CustomClassifier(obj=Perceptron(random_state=42))
Perceptron(random_state=42)
Perceptron(random_state=42) print(classification_report(y_test, model_dictionary["Perceptron"].fit(X_train, y_train).predict(X_test))) precision recall f1-score support 0 1.00 0.98 0.99 41 1 0.99 1.00 0.99 73 accuracy 0.99 114 macro avg 0.99 0.99 0.99 114 weighted avg 0.99 0.99 0.99 114 ConfusionMatrixDisplay.from_estimator(model_dictionary["Perceptron"], X_test, y_test) plt.show()
2 - iris data
data = load_iris() X = data.data y= data.target X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123) clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True) start = time() models, predictions = clf.fit(X_train, X_test, y_train, y_test) print(f"\n\n Elapsed: {time()-start} seconds \n") model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test) display(models) 100%|██████████| 27/27 [00:05<00:00, 4.51it/s] Elapsed: 5.992894172668457 seconds
Accuracy | Balanced Accuracy | ROC AUC | F1 Score | Time Taken | |
---|---|---|---|---|---|
Model | |||||
BaggingClassifier | 1.00 | 1.00 | None | 1.00 | 0.18 |
DecisionTreeClassifier | 1.00 | 1.00 | None | 1.00 | 0.11 |
KNeighborsClassifier | 0.97 | 0.97 | None | 0.97 | 0.10 |
CalibratedClassifierCV | 0.97 | 0.97 | None | 0.97 | 0.24 |
SGDClassifier | 0.97 | 0.97 | None | 0.97 | 0.21 |
ExtraTreeClassifier | 0.97 | 0.97 | None | 0.97 | 0.10 |
RidgeClassifier | 0.97 | 0.97 | None | 0.97 | 1.05 |
RidgeClassifierCV | 0.97 | 0.97 | None | 0.97 | 0.21 |
RandomForestClassifier | 0.97 | 0.97 | None | 0.97 | 0.88 |
LogisticRegression | 0.97 | 0.97 | None | 0.97 | 0.14 |
LinearSVC | 0.97 | 0.97 | None | 0.97 | 0.11 |
PassiveAggressiveClassifier | 0.93 | 0.94 | None | 0.93 | 0.11 |
Perceptron | 0.93 | 0.94 | None | 0.93 | 0.11 |
BernoulliNB | 0.93 | 0.94 | None | 0.93 | 0.10 |
LabelSpreading | 0.93 | 0.91 | None | 0.93 | 0.31 |
LabelPropagation | 0.93 | 0.91 | None | 0.93 | 0.20 |
ExtraTreesClassifier | 0.93 | 0.91 | None | 0.93 | 0.45 |
GaussianNB | 0.90 | 0.91 | None | 0.90 | 0.10 |
AdaBoostClassifier | 0.90 | 0.91 | None | 0.90 | 0.38 |
SVC | 0.87 | 0.88 | None | 0.87 | 0.19 |
NuSVC | 0.87 | 0.88 | None | 0.87 | 0.12 |
NearestCentroid | 0.87 | 0.88 | None | 0.87 | 0.10 |
LinearDiscriminantAnalysis | 0.63 | 0.67 | None | 0.54 | 0.20 |
QuadraticDiscriminantAnalysis | 0.20 | 0.33 | None | 0.07 | 0.11 |
DummyClassifier | 0.20 | 0.33 | None | 0.07 | 0.10 |
model_dictionary["BaggingClassifier"]
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=BaggingClassifier(random_state=42))))In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=BaggingClassifier(random_state=42))))
CustomClassifier(obj=CustomClassifier(obj=BaggingClassifier(random_state=42)))
CustomClassifier(obj=BaggingClassifier(random_state=42))
BaggingClassifier(random_state=42)
BaggingClassifier(random_state=42) print(classification_report(y_test, model_dictionary["BaggingClassifier"].fit(X_train, y_train).predict(X_test))) precision recall f1-score support 0 1.00 1.00 1.00 13 1 1.00 1.00 1.00 6 2 1.00 1.00 1.00 11 accuracy 1.00 30 macro avg 1.00 1.00 1.00 30 weighted avg 1.00 1.00 1.00 30 ConfusionMatrixDisplay.from_estimator(model_dictionary["BaggingClassifier"], X_test, y_test) plt.show()
3 - wine data
data = load_wine() X = data.data y= data.target X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123) clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True) start = time() models, predictions = clf.fit(X_train, X_test, y_train, y_test) print(f"\n\n Elapsed: {time()-start} seconds \n") model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test) display(models) 100%|██████████| 27/27 [00:05<00:00, 5.09it/s] Elapsed: 5.312330007553101 seconds
Accuracy | Balanced Accuracy | ROC AUC | F1 Score | Time Taken | |
---|---|---|---|---|---|
Model | |||||
RidgeClassifierCV | 1.00 | 1.00 | None | 1.00 | 0.15 |
RidgeClassifier | 1.00 | 1.00 | None | 1.00 | 0.13 |
Perceptron | 1.00 | 1.00 | None | 1.00 | 0.15 |
SVC | 0.97 | 0.98 | None | 0.97 | 0.13 |
SGDClassifier | 0.97 | 0.98 | None | 0.97 | 0.13 |
CalibratedClassifierCV | 0.97 | 0.98 | None | 0.97 | 0.27 |
PassiveAggressiveClassifier | 0.97 | 0.98 | None | 0.97 | 0.14 |
LogisticRegression | 0.97 | 0.98 | None | 0.97 | 0.16 |
LinearSVC | 0.97 | 0.98 | None | 0.97 | 0.14 |
BaggingClassifier | 0.97 | 0.97 | None | 0.97 | 0.21 |
RandomForestClassifier | 0.97 | 0.97 | None | 0.97 | 0.61 |
LinearDiscriminantAnalysis | 0.97 | 0.97 | None | 0.97 | 0.16 |
LabelSpreading | 0.94 | 0.94 | None | 0.94 | 0.44 |
LabelPropagation | 0.94 | 0.94 | None | 0.94 | 0.26 |
ExtraTreesClassifier | 0.94 | 0.94 | None | 0.94 | 0.50 |
KNeighborsClassifier | 0.92 | 0.92 | None | 0.92 | 0.13 |
ExtraTreeClassifier | 0.89 | 0.88 | None | 0.89 | 0.13 |
DecisionTreeClassifier | 0.83 | 0.87 | None | 0.83 | 0.12 |
GaussianNB | 0.86 | 0.85 | None | 0.85 | 0.13 |
NearestCentroid | 0.86 | 0.81 | None | 0.86 | 0.13 |
NuSVC | 0.83 | 0.79 | None | 0.84 | 0.13 |
AdaBoostClassifier | 0.81 | 0.78 | None | 0.81 | 0.43 |
BernoulliNB | 0.81 | 0.75 | None | 0.80 | 0.15 |
QuadraticDiscriminantAnalysis | 0.53 | 0.58 | None | 0.52 | 0.13 |
DummyClassifier | 0.31 | 0.33 | None | 0.14 | 0.13 |
model_dictionary["RidgeClassifierCV"]
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=RidgeClassifierCV())))In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=RidgeClassifierCV())))
CustomClassifier(obj=CustomClassifier(obj=RidgeClassifierCV()))
CustomClassifier(obj=RidgeClassifierCV())
RidgeClassifierCV()
RidgeClassifierCV() print(classification_report(y_test, model_dictionary["RidgeClassifierCV"].fit(X_train, y_train).predict(X_test))) precision recall f1-score support 0 1.00 1.00 1.00 8 1 1.00 1.00 1.00 11 2 1.00 1.00 1.00 17 accuracy 1.00 36 macro avg 1.00 1.00 1.00 36 weighted avg 1.00 1.00 1.00 36 ConfusionMatrixDisplay.from_estimator(model_dictionary["RidgeClassifierCV"], X_test, y_test) plt.show()
4 - digits data
data = load_digits() X = data.data y= data.target X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123) clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True) start = time() models, predictions = clf.fit(X_train, X_test, y_train, y_test) print(f"\n\n Elapsed: {time()-start} seconds \n") model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test) display(models) 100%|██████████| 27/27 [01:41<00:00, 3.75s/it] Elapsed: 101.15918207168579 seconds
Accuracy | Balanced Accuracy | ROC AUC | F1 Score | Time Taken | |
---|---|---|---|---|---|
Model | |||||
SVC | 0.99 | 0.99 | None | 0.99 | 3.47 |
LogisticRegression | 0.97 | 0.97 | None | 0.97 | 3.67 |
CalibratedClassifierCV | 0.97 | 0.97 | None | 0.97 | 8.78 |
RandomForestClassifier | 0.97 | 0.97 | None | 0.97 | 4.46 |
LinearDiscriminantAnalysis | 0.96 | 0.96 | None | 0.96 | 4.78 |
LinearSVC | 0.96 | 0.96 | None | 0.96 | 4.18 |
ExtraTreesClassifier | 0.96 | 0.96 | None | 0.96 | 3.50 |
PassiveAggressiveClassifier | 0.96 | 0.96 | None | 0.96 | 2.89 |
KNeighborsClassifier | 0.96 | 0.96 | None | 0.96 | 3.60 |
SGDClassifier | 0.96 | 0.96 | None | 0.96 | 2.62 |
RidgeClassifier | 0.94 | 0.94 | None | 0.94 | 4.45 |
Perceptron | 0.94 | 0.94 | None | 0.94 | 4.38 |
RidgeClassifierCV | 0.94 | 0.94 | None | 0.93 | 3.02 |
NuSVC | 0.93 | 0.93 | None | 0.93 | 4.55 |
BaggingClassifier | 0.92 | 0.92 | None | 0.92 | 4.84 |
LabelPropagation | 0.91 | 0.91 | None | 0.92 | 2.63 |
LabelSpreading | 0.91 | 0.91 | None | 0.92 | 3.64 |
QuadraticDiscriminantAnalysis | 0.88 | 0.88 | None | 0.88 | 2.39 |
NearestCentroid | 0.85 | 0.85 | None | 0.85 | 4.27 |
DecisionTreeClassifier | 0.83 | 0.83 | None | 0.83 | 3.36 |
BernoulliNB | 0.82 | 0.82 | None | 0.82 | 2.85 |
GaussianNB | 0.81 | 0.81 | None | 0.80 | 4.63 |
ExtraTreeClassifier | 0.81 | 0.80 | None | 0.81 | 3.35 |
AdaBoostClassifier | 0.39 | 0.37 | None | 0.32 | 5.05 |
DummyClassifier | 0.08 | 0.10 | None | 0.01 | 4.98 |
model_dictionary["SVC"]
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=SVC(random_state=42))))In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=SVC(random_state=42))))
CustomClassifier(obj=CustomClassifier(obj=SVC(random_state=42)))
CustomClassifier(obj=SVC(random_state=42))
SVC(random_state=42)
SVC(random_state=42) print(classification_report(y_test, model_dictionary["SVC"].fit(X_train, y_train).predict(X_test))) precision recall f1-score support 0 1.00 1.00 1.00 39 1 0.94 1.00 0.97 34 2 1.00 0.97 0.99 36 3 1.00 1.00 1.00 33 4 0.95 1.00 0.98 42 5 1.00 0.95 0.97 37 6 1.00 1.00 1.00 43 7 1.00 1.00 1.00 31 8 1.00 0.95 0.97 37 9 0.97 1.00 0.98 28 accuracy 0.99 360 macro avg 0.99 0.99 0.99 360 weighted avg 0.99 0.99 0.99 360 ConfusionMatrixDisplay.from_estimator(model_dictionary["SVC"], X_test, y_test) plt.show()
ConfusionMatrixDisplay.from_estimator(model_dictionary["LogisticRegression"], X_test, y_test) plt.show()
5 - R examples
I hope this is not too technical: please note that installing the R version of nnetsauce
will create an environment in the current directory (more precisely, a folder named r-reticulate
will be created where you’ll install the package).
Classification
remotes::install_github("Techtonique/nnetsauce_r") # install nnetsauce from GitHub library(datasets) set.seed(123) X <- as.matrix(iris[, 1:4]) y <- as.integer(iris$Species) - 1L (index_train <- base::sample.int(n = nrow(X), size = floor(0.8*nrow(X)), replace = FALSE)) X_train <- X[index_train, ] y_train <- y[index_train] X_test <- X[-index_train, ] y_test <- y[-index_train] obj <- nnetsauce::LazyDeepClassifier(n_layers = 3L) res <- obj$fit(X_train, X_test, y_train, y_test) print(res[[1]]) # best accuracy must be 1.0
Regression
X <- MASS::Boston[,-14] # dataset has an ethical problem y <- MASS::Boston$medv set.seed(13) (index_train <- base::sample.int(n = nrow(X), size = floor(0.8*nrow(X)), replace = FALSE)) X_train <- X[index_train, ] y_train <- y[index_train] X_test <- X[-index_train, ] y_test <- y[-index_train] obj <- nnetsauce::LazyDeepRegressor(n_layers = 3L, n_clusters=2L) res <- obj$fit(X_train, X_test, y_train, y_test) print(res[[1]]) # best RMSE must be close to 2.8, try n_clusters=3L for 2.69
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Copyright © 2023 | MH Corporate basic by MH Themes