A computer vision challenge: finding boats in the Mona Lisa

About a decade or so photomosaics were all the rage: a near-recreation of a famous image by using many smaller images as elements. Here, for example, is the Mona Lisa, created using the Metapixel program by overlaying 32×32 images of vehicles and animals.

An image like this presents an interesting computer vision challenge: can you use deep learning techniques to find the pictures of boats and cars embedded in the image, amongst all the noise and clutter of the other images around and often on top of them? This is the challenge that Max Kaznady and his colleagues in the data science team took upon themselves, using the power of an Azure N-Series virtual machine with 24 cores and 4 K80 GPUs. The model was trained using the mxnet package running on Microsoft R Server, which takes advantage of the powerful GPUs to train a Residual Network (ResNet) DNN with 21 convolutional blocks. (You can read more about Resnet in this Microsoft Research paper.) Once the model was trained, an HDInsight Spark cluster running Microsoft R Server was used to parallelize the problem of finding boat and car images within the photomosaic. Here's the architecture of the system, with the steps marked in order in yellow; another blog post explains how to set up such an architecture yourself. You can also just use the Deep Learning Toolkit with the Data Science Virtual Machine.

To learn more about this application, check out this recorded presentation from the Data Science Summit presented by Max Kaznady and Tao Wu, or the blog post linked below.

Cortana Intelligence and Machine Learning Blog: Applying Deep Learning at Cloud Scale, with Microsoft R Server & Azure Data Lake