Superpixels in imager

Superpixels are used in image segmentation as a pre-processing step. Instead of segmenting pixels directly, we first group similar pixels into “super-pixels”, which can then be processed further (and more cheaply).

(image from Wikimedia)

The current version of imager doesn’t implement them, but it turns out that SLIC superpixels are particularly easy to implement. SLIC is essentially k-means applied to pixels, with some bells and whistles.

We could use k-means to segment images based on colour alone. To get good results on colour segmentation the CIELAB colour space is appropriate, because it tries to be perceptually uniform.

library(tidyverse)
library(imager)
im <- load.image("https://upload.wikimedia.org/wikipedia/commons/thumb/f/fd/Aster_Tataricus.JPG/1024px-Aster_Tataricus.JPG")
#Convert to CIELAB colour space, then create a data.frame with three colour channels as columns
d <- sRGBtoLab(im) %>% as.data.frame(wide="c")%>%
    dplyr::select(-x,-y)
#Run k-means with 2 centers
km <- kmeans(d,2)
#Turn cluster index into an image
seg <- as.cimg(km$cluster,dim=c(dim(im)[1:2],1,1))
plot(im,axes=FALSE)
highlight(seg==1)

We mostly manage to separate the petals from the rest, with a few errors here and there.
SLIC does pretty much the same thing, except we (a) use many more centers and (b) we add pixel coordinates as features in the clustering. The latter ensures that only adjacent pixels get grouped together.

The code below implements SLIC. It’s mostly straightforward:

#Compute SLIC superpixels
#im: input image
#nS: number of superpixels
#ratio: determines compactness of superpixels.
#low values will result in pixels with weird shapes
#... further arguments passed to kmeans
slic <- function(im,nS,compactness=1,...)
{
    #If image is in colour, convert to CIELAB
    if (spectrum(im) ==3) im <- sRGBtoLab(im)

    #The pixel coordinates vary over 1...width(im) and 1...height(im)
    #Pixel values can be over a widely different range
    #We need our features to have similar scales, so
    #we compute relative scales of spatial dimensions to colour dimensions
    sc.spat <- (dim(im)[1:2]*.28) %>% max #Scale of spatial dimensions
    sc.col <- imsplit(im,"c") %>% map_dbl(sd) %>% max

    #Scaling ratio for pixel values
    rat <- (sc.spat/sc.col)/(compactness*10)

    
    X <- as.data.frame(im*rat,wide="c") %>% as.matrix
    #Generate initial centers from a grid
    ind <- round(seq(1,nPix(im)/spectrum(im),l=nS))
    #Run k-means
    km <- kmeans(X,X[ind,],...)

    #Return segmentation as image (pixel values index cluster)
    seg <- as.cimg(km$cluster,dim=c(dim(im)[1:2],1,1))
    #Superpixel image: each pixel is given the colour of the superpixel it belongs to
    sp <- map(1:spectrum(im),~ km$centers[km$cluster,2+.]) %>% do.call(c,.) %>% as.cimg(dim=dim(im))
    #Correct for ratio
    sp <- sp/rat
    if (spectrum(im)==3)
    {
        #Convert back to RGB
        sp <- LabtosRGB(sp) 
    }
    list(km=km,seg=seg,sp=sp)
}

Use it as follows:

#400 superpixels
out <- slic(im,400)
#Superpixels
plot(out$sp,axes=FALSE)
#Segmentation
plot(out$seg,axes=FALSE)
#Show segmentation on original image
(im*add.colour(abs(imlap(out$seg)) == 0)) %>% plot(axes=FALSE)

The next step is to segment the superpixels but I’ll keep that for another time.