Site icon R-bloggers

A classifier that’s very accurate (and deep)

[This article was first published on T. Moudiki's Webpage - R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.

Version v0.15.0 of nnetsauce is now available on your favorite platforms: PyPI, conda and GitHub for Python; R universe and GitHub for R.

The changes in this version are mostly related to Automated Machine learning (AutoML):

Note that in the example below, for the offically released version (v0.15.0), Gradient boosting classifiers are available. This doesn’t change the best model chosen by the algorithm (which is never Gradient boosting, because the deep model is already doing “too much”). Not sure if Gradient boosting will be kept as a default model here because, in this context, it’s relatively slow.

To finish, for Windows users, if you run into issues when trying to install nnetsauce (but you shouldn’t): remember that you can use the Windows Subsystem for Linux (WSL).

Contents

Here is a jupyter notebook allowing you to reproduce these results. Do not hesitate to modify these examples by choosing – in LazyDeepClassifier – a different number of layers n_layers, or the number of engineered features per layer, n_hidden_features.

0 – Install and import packages

top

!pip install nnetsauce --upgrade

import os
import nnetsauce as ns
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer, load_iris, load_wine, load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, ConfusionMatrixDisplay
from time import time

1 – breast cancer data

top

data = load_breast_cancer()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)

display(models)

100%|██████████| 27/27 [00:44<00:00,  1.65s/it]



 Elapsed: 44.49836850166321 seconds 
Accuracy Balanced Accuracy ROC AUC F1 Score Time Taken
Model
Perceptron 0.99 0.99 0.99 0.99 2.15
SGDClassifier 0.98 0.98 0.98 0.98 1.99
RandomForestClassifier 0.98 0.98 0.98 0.98 2.76
PassiveAggressiveClassifier 0.98 0.98 0.98 0.98 1.89
LogisticRegression 0.98 0.98 0.98 0.98 1.14
ExtraTreesClassifier 0.98 0.98 0.98 0.98 3.13
BaggingClassifier 0.97 0.97 0.97 0.97 1.62
LabelPropagation 0.97 0.97 0.97 0.97 1.69
LabelSpreading 0.97 0.97 0.97 0.97 1.08
AdaBoostClassifier 0.97 0.97 0.97 0.97 3.42
LinearSVC 0.97 0.97 0.97 0.97 1.65
KNeighborsClassifier 0.97 0.97 0.97 0.97 1.07
SVC 0.97 0.96 0.96 0.97 1.07
CalibratedClassifierCV 0.97 0.96 0.96 0.97 1.82
DecisionTreeClassifier 0.96 0.96 0.96 0.96 1.50
QuadraticDiscriminantAnalysis 0.96 0.95 0.95 0.96 1.16
LinearDiscriminantAnalysis 0.96 0.95 0.95 0.96 1.39
ExtraTreeClassifier 0.96 0.94 0.94 0.96 0.83
RidgeClassifier 0.96 0.94 0.94 0.96 3.15
RidgeClassifierCV 0.96 0.94 0.94 0.96 1.51
GaussianNB 0.93 0.90 0.90 0.93 1.50
NearestCentroid 0.93 0.90 0.90 0.93 3.00
NuSVC 0.93 0.90 0.90 0.93 1.27
BernoulliNB 0.91 0.89 0.89 0.91 1.15
DummyClassifier 0.64 0.50 0.50 0.50 0.96
model_dictionary["Perceptron"]
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=Perceptron(random_state=42))))
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=Perceptron(random_state=42))))
CustomClassifier(obj=CustomClassifier(obj=Perceptron(random_state=42)))
CustomClassifier(obj=Perceptron(random_state=42))
Perceptron(random_state=42)
Perceptron(random_state=42)
print(classification_report(y_test, model_dictionary["Perceptron"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      0.98      0.99        41
           1       0.99      1.00      0.99        73

    accuracy                           0.99       114
   macro avg       0.99      0.99      0.99       114
weighted avg       0.99      0.99      0.99       114

ConfusionMatrixDisplay.from_estimator(model_dictionary["Perceptron"], X_test, y_test)
plt.show()

2 – iris data

top

data = load_iris()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 27/27 [00:05<00:00,  4.51it/s]



 Elapsed: 5.992894172668457 seconds 
Accuracy Balanced Accuracy ROC AUC F1 Score Time Taken
Model
BaggingClassifier 1.00 1.00 None 1.00 0.18
DecisionTreeClassifier 1.00 1.00 None 1.00 0.11
KNeighborsClassifier 0.97 0.97 None 0.97 0.10
CalibratedClassifierCV 0.97 0.97 None 0.97 0.24
SGDClassifier 0.97 0.97 None 0.97 0.21
ExtraTreeClassifier 0.97 0.97 None 0.97 0.10
RidgeClassifier 0.97 0.97 None 0.97 1.05
RidgeClassifierCV 0.97 0.97 None 0.97 0.21
RandomForestClassifier 0.97 0.97 None 0.97 0.88
LogisticRegression 0.97 0.97 None 0.97 0.14
LinearSVC 0.97 0.97 None 0.97 0.11
PassiveAggressiveClassifier 0.93 0.94 None 0.93 0.11
Perceptron 0.93 0.94 None 0.93 0.11
BernoulliNB 0.93 0.94 None 0.93 0.10
LabelSpreading 0.93 0.91 None 0.93 0.31
LabelPropagation 0.93 0.91 None 0.93 0.20
ExtraTreesClassifier 0.93 0.91 None 0.93 0.45
GaussianNB 0.90 0.91 None 0.90 0.10
AdaBoostClassifier 0.90 0.91 None 0.90 0.38
SVC 0.87 0.88 None 0.87 0.19
NuSVC 0.87 0.88 None 0.87 0.12
NearestCentroid 0.87 0.88 None 0.87 0.10
LinearDiscriminantAnalysis 0.63 0.67 None 0.54 0.20
QuadraticDiscriminantAnalysis 0.20 0.33 None 0.07 0.11
DummyClassifier 0.20 0.33 None 0.07 0.10
model_dictionary["BaggingClassifier"]
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=BaggingClassifier(random_state=42))))
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=BaggingClassifier(random_state=42))))
CustomClassifier(obj=CustomClassifier(obj=BaggingClassifier(random_state=42)))
CustomClassifier(obj=BaggingClassifier(random_state=42))
BaggingClassifier(random_state=42)
BaggingClassifier(random_state=42)
print(classification_report(y_test, model_dictionary["BaggingClassifier"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        13
           1       1.00      1.00      1.00         6
           2       1.00      1.00      1.00        11

    accuracy                           1.00        30
   macro avg       1.00      1.00      1.00        30
weighted avg       1.00      1.00      1.00        30

ConfusionMatrixDisplay.from_estimator(model_dictionary["BaggingClassifier"], X_test, y_test)
plt.show()

3 – wine data

top

data = load_wine()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 27/27 [00:05<00:00,  5.09it/s]



 Elapsed: 5.312330007553101 seconds 
Accuracy Balanced Accuracy ROC AUC F1 Score Time Taken
Model
RidgeClassifierCV 1.00 1.00 None 1.00 0.15
RidgeClassifier 1.00 1.00 None 1.00 0.13
Perceptron 1.00 1.00 None 1.00 0.15
SVC 0.97 0.98 None 0.97 0.13
SGDClassifier 0.97 0.98 None 0.97 0.13
CalibratedClassifierCV 0.97 0.98 None 0.97 0.27
PassiveAggressiveClassifier 0.97 0.98 None 0.97 0.14
LogisticRegression 0.97 0.98 None 0.97 0.16
LinearSVC 0.97 0.98 None 0.97 0.14
BaggingClassifier 0.97 0.97 None 0.97 0.21
RandomForestClassifier 0.97 0.97 None 0.97 0.61
LinearDiscriminantAnalysis 0.97 0.97 None 0.97 0.16
LabelSpreading 0.94 0.94 None 0.94 0.44
LabelPropagation 0.94 0.94 None 0.94 0.26
ExtraTreesClassifier 0.94 0.94 None 0.94 0.50
KNeighborsClassifier 0.92 0.92 None 0.92 0.13
ExtraTreeClassifier 0.89 0.88 None 0.89 0.13
DecisionTreeClassifier 0.83 0.87 None 0.83 0.12
GaussianNB 0.86 0.85 None 0.85 0.13
NearestCentroid 0.86 0.81 None 0.86 0.13
NuSVC 0.83 0.79 None 0.84 0.13
AdaBoostClassifier 0.81 0.78 None 0.81 0.43
BernoulliNB 0.81 0.75 None 0.80 0.15
QuadraticDiscriminantAnalysis 0.53 0.58 None 0.52 0.13
DummyClassifier 0.31 0.33 None 0.14 0.13
model_dictionary["RidgeClassifierCV"]
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=RidgeClassifierCV())))
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=RidgeClassifierCV())))
CustomClassifier(obj=CustomClassifier(obj=RidgeClassifierCV()))
CustomClassifier(obj=RidgeClassifierCV())
RidgeClassifierCV()
RidgeClassifierCV()
print(classification_report(y_test, model_dictionary["RidgeClassifierCV"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00         8
           1       1.00      1.00      1.00        11
           2       1.00      1.00      1.00        17

    accuracy                           1.00        36
   macro avg       1.00      1.00      1.00        36
weighted avg       1.00      1.00      1.00        36

ConfusionMatrixDisplay.from_estimator(model_dictionary["RidgeClassifierCV"], X_test, y_test)
plt.show()

4 – digits data

top

data = load_digits()
X = data.data
y= data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = .2, random_state = 123)

clf = ns.LazyDeepClassifier(n_layers=3, verbose=0, ignore_warnings=True)
start = time()
models, predictions = clf.fit(X_train, X_test, y_train, y_test)
print(f"\n\n Elapsed: {time()-start} seconds \n")
model_dictionary = clf.provide_models(X_train, X_test, y_train, y_test)
display(models)

100%|██████████| 27/27 [01:41<00:00,  3.75s/it]



 Elapsed: 101.15918207168579 seconds 
Accuracy Balanced Accuracy ROC AUC F1 Score Time Taken
Model
SVC 0.99 0.99 None 0.99 3.47
LogisticRegression 0.97 0.97 None 0.97 3.67
CalibratedClassifierCV 0.97 0.97 None 0.97 8.78
RandomForestClassifier 0.97 0.97 None 0.97 4.46
LinearDiscriminantAnalysis 0.96 0.96 None 0.96 4.78
LinearSVC 0.96 0.96 None 0.96 4.18
ExtraTreesClassifier 0.96 0.96 None 0.96 3.50
PassiveAggressiveClassifier 0.96 0.96 None 0.96 2.89
KNeighborsClassifier 0.96 0.96 None 0.96 3.60
SGDClassifier 0.96 0.96 None 0.96 2.62
RidgeClassifier 0.94 0.94 None 0.94 4.45
Perceptron 0.94 0.94 None 0.94 4.38
RidgeClassifierCV 0.94 0.94 None 0.93 3.02
NuSVC 0.93 0.93 None 0.93 4.55
BaggingClassifier 0.92 0.92 None 0.92 4.84
LabelPropagation 0.91 0.91 None 0.92 2.63
LabelSpreading 0.91 0.91 None 0.92 3.64
QuadraticDiscriminantAnalysis 0.88 0.88 None 0.88 2.39
NearestCentroid 0.85 0.85 None 0.85 4.27
DecisionTreeClassifier 0.83 0.83 None 0.83 3.36
BernoulliNB 0.82 0.82 None 0.82 2.85
GaussianNB 0.81 0.81 None 0.80 4.63
ExtraTreeClassifier 0.81 0.80 None 0.81 3.35
AdaBoostClassifier 0.39 0.37 None 0.32 5.05
DummyClassifier 0.08 0.10 None 0.01 4.98
model_dictionary["SVC"]
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=SVC(random_state=42))))
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
CustomClassifier(obj=CustomClassifier(obj=CustomClassifier(obj=SVC(random_state=42))))
CustomClassifier(obj=CustomClassifier(obj=SVC(random_state=42)))
CustomClassifier(obj=SVC(random_state=42))
SVC(random_state=42)
SVC(random_state=42)
print(classification_report(y_test, model_dictionary["SVC"].fit(X_train, y_train).predict(X_test)))

              precision    recall  f1-score   support

           0       1.00      1.00      1.00        39
           1       0.94      1.00      0.97        34
           2       1.00      0.97      0.99        36
           3       1.00      1.00      1.00        33
           4       0.95      1.00      0.98        42
           5       1.00      0.95      0.97        37
           6       1.00      1.00      1.00        43
           7       1.00      1.00      1.00        31
           8       1.00      0.95      0.97        37
           9       0.97      1.00      0.98        28

    accuracy                           0.99       360
   macro avg       0.99      0.99      0.99       360
weighted avg       0.99      0.99      0.99       360

ConfusionMatrixDisplay.from_estimator(model_dictionary["SVC"], X_test, y_test)
plt.show()

ConfusionMatrixDisplay.from_estimator(model_dictionary["LogisticRegression"], X_test, y_test)
plt.show()

5 – R examples

top

I hope this is not too technical: please note that installing the R version of nnetsauce will create an environment in the current directory (more precisely, a folder named r-reticulate will be created where you’ll install the package).

Classification

remotes::install_github("Techtonique/nnetsauce_r") # install nnetsauce from GitHub

library(datasets)

set.seed(123)
X <- as.matrix(iris[, 1:4])
y <- as.integer(iris$Species) - 1L

(index_train <- base::sample.int(n = nrow(X),
                                 size = floor(0.8*nrow(X)),
                                 replace = FALSE))
X_train <- X[index_train, ]
y_train <- y[index_train]
X_test <- X[-index_train, ]
y_test <- y[-index_train]

obj <- nnetsauce::LazyDeepClassifier(n_layers = 3L)
res <- obj$fit(X_train, X_test, y_train, y_test)
print(res[[1]]) # best accuracy must be 1.0

Regression

X <- MASS::Boston[,-14] # dataset has an ethical problem
y <- MASS::Boston$medv

set.seed(13)
(index_train <- base::sample.int(n = nrow(X),
                                 size = floor(0.8*nrow(X)),
                                 replace = FALSE))
X_train <- X[index_train, ]
y_train <- y[index_train]
X_test <- X[-index_train, ]
y_test <- y[-index_train]

obj <- nnetsauce::LazyDeepRegressor(n_layers = 3L, n_clusters=2L)
res <- obj$fit(X_train, X_test, y_train, y_test)
print(res[[1]]) # best RMSE must be close to 2.8, try n_clusters=3L for 2.69
To leave a comment for the author, please follow the link and comment on their blog: T. Moudiki's Webpage - R.

R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.