Convolutional Neural Network

Convolutional Neural Network (CNN or ConvNet) is a special type of network that enables the the network to recognize patterns from data. Through series of convolutional, non-linear, and pooling layers, the network extracts useful features from the data. For example, in image classification, visual features could be an edge, a color, a texture, a shape, or some more complicated patterns.

Because of this stregnth in pattern recognition, CNN is widely used in fields like image classification, computer vision, and natural language processing. However, the downside of the CNN is that it is not capable of extracting global spatial information. The network does not pay attention to where 'VW Beetle' is located in an image, but it only cares if the given image contains 'VW Beetle' regardless of their position.

Feature Extraction

The feature extraction process consists of three basic steps:

1. Convolution: filter an image for particular feature
2. Non-linear: detect feature within the filtered image
3. Pooling: condense the image to enhance the features

The network perform several feature extractions in parallel producing numerous visual features.

Convolution

Convolution is an efficient way of connecting input signals to fully connected neurons. Instead of connecting entire raw data to fully connected neurons, convolution allows the network to learn patterns in local regions. It does so by scanning a convolution matrix or a kernel over an image producing a weighted sum of the input data. Different matrices produce different effects on the input data.

Stride and Padding

Convolution provides more options to manipulate its output with stride and padding. Stride allows you to control the overlap between kernels. Bigger stride may reduces redundancies between neighboring features and produces smaller output size.

Padding is an option to conserve informations on the edges of an image. By adding zero padding to the borders of an image, kernels can capture inportations on the edges better. The size of an output is calculated using the formula below:

Where input size is $N \times N$, kernel size is $F \times F$, $S$ is the stide size, and $P$ is the number of layers of padding.

Nonlinearity (ReLU)

After convolution, the feature maps pass through an activation function (the most popular is ReLU). The kaggle computer vision course describes the purpose of an activation function as:

You could think about the activation function as scoring pixel values according to some measure of importance. The ReLU activation says that negative values are not important and so sets them to 0. ("Everything unimportant is equally unimportant.")

The activation function also allows a convolutional network to have depth. The final output of the convolutional network without nonlinearity is the sum of the effects of each layer which is

Pooling

Max-Pooling is a popular technique to reduce complexities and "sharpen" or "intensify" features. The most common size used in max-pooling is $2 \times 2$. It partitions input data into $2 \times 2$ regions and extracts the maximum value inside the region.

Fully Connected Layer

After feature extractions, fully-connected layers learns extracted feature maps similar to a conventional neural network.

Common CNN architecture

LeNet-5 (1998):

LeNet is the first CNN architecture. It was designed for recognizing handwritten digits

AlexNet (2012):

AlexNet is the architecture that popularized CNN. The Architecture of AlexNet is similar to LeNet, but it uses two parallel networks trained with two GPUs. It won ILSVRC 2012

VGG-16 VGG-19 (2014):

Deeper network developed for large scale image recognition.

GoogLeNet (Inception) (2014): A variant of the inception network designed by Google to win ILSVRC 2014. It utilizes inception modules which allow the network to choose between multiple convolutional modules.

ResNet (2015):

Residual neural network is a network designed by stacking residual blocks on top of each other. Residual blocks are blocks of layers with the addition of a skipped connection. This allows the network to learn residual functions with reference to the layer inputs, instead of learning unreferenced functions. ResNet won ILSVRC 2015.

Pre-trained Model Available in Keras

See here

Specialized CNN Architectures

MobileNet: CNN tuned for mobile devices

RCNNs, SDD, and Yolo: object detection

UNet: medical image segmentation

Links

• Understanding of a convolutional neural network (S. Albawi et al.)
• Computer Vision
• Different Types of CNN Architectures Explained: Examples