Deep Learning is eating the world.
The hype began around 2012 when a Neural Network achieved super human performance on Image Recognition tasks and only a few people could predict what was about to happen.
During the past decade, more and more algorithms are coming to life. More and more companies are starting to add them in their daily business.
Here, I tried to cover all the most important Deep Learning algorithms and architectures concieved over the years for use in a variety of applications such as Computer Vision and Natural Language Processing.
Some of them are used more frequently than others and each one has its own streghth and weeknesses.
My main goal is to give you a general idea of the field and help you understand what algorithm should you use in each specific case. Because I know it seems chaotic for someone who wants to start from scratch.
But after reading the guide, I am confident that you will be able to recognize what is what and you will be ready to begin using them right away.
So if you are looking for a truly complete guide on Deep Learning , let's get started.
Contents
Deep Learning is gaining crazy amounts of popularity in the scientific and corporate communities. Since 2012, the year when a Convolutional Neural Network achieved unprecedent accuracy on an image recognition competition ( ImageNet Large Scale Visual Recognition Challenge), more and more research papers come out every year and more and more companies started to incorporate Neural Networks into their businesses. It is estimated that Deep Learning is right now a 2.5 Billion Market and expected to become 18.16 Billion by 2023.
But what is Deep learning?
According to Wikipedia: “Deep learning (also known as deep structured learning or differential programming) is part of a broader family of machine learning methods based on artificial neural networks with representation learning. Learning can be supervised, semi-supervised or unsupervised”.
In my mind, Deep Learning is a collection of algorithms inspired by the workings of the human brain in processing data and creating patterns for use in decision making, which are expanding and improving on the idea of a single model architecture called Artificial Neural Network.
Neural Networks
Just like the human brain, Neural Networks consist of Neurons. Each Neuron receives signals as an input, multiplies them by weights, sums them up and applies a non-linear function. These neurons are stacked next to each other and organized in layers.
But what do we accomplish by doing that?
Datacamp
Algorithm
It turns out that Neural Networks are excellent function approximators.
We can assume that every behavior and every system can ultimately be represented as a mathematical function (sometimes an incredible complex one). If we somehow manage to find that function, we can essentially understand everything about the system. But finding the function can be extremely hard. So, we need to estimate it. Enter Neural Networks.
Backpropagation
Neural Networks are able to learn the desired function using big amounts of data and an iterative algorithm called backpropagation. We feed the network with data, it produces an output, we compare that output with a desired one (using a loss function) and we readjust the weights based on the difference.
And repeat. And repeat. The adjustment of weights is performed using a non-linear optimization technique called stochastic gradient descent.
After a while, the network will become really good at producing the output. Hence, the training is over. Hence, we manage to approximate our function. And if we pass an input with an unknown output to the network, it will give us an answer based on the approximated function.
Let’s use an example to make this clearer. Let’s say that for some reason we want to identify images with a tree. We feed the network with any kind of images and it produces an output. Since we know if the image has actually a tree or not, we can compare the output with our truth and adjust the network.
As we pass more and more images, the network will make fewer and fewer mistakes. Now we can feed it with an unknown image, and it will tell us if the image contains a tree. Pretty cool, right?
Over the years researchers came up with amazing improvements on the original idea. Each new architecture was targeted on a specific problem and one achieved better accuracy and speed. We can classify all those new models in specific categories:
Feedforward Neural Networks (FNN)
Feedforward Neural Networks are usually fully connected, which means that every neuron in a layer is connected with all the other neurons in the next layers. The described structure is called Multilayer Perceptron and originated back in 1958. Single-layer perceptron can only learn linearly separable patterns, but a multilayer perceptron is able to learn non-linear relationships between the data.
They are exceptionally well on tasks like classification and regression. Contrary to other machine learning algorithms, they don’t converge so easily. The more data they have, the higher their accuracy.
Convolutional Neural Networks (CNN)
Convolutional Neural Networks employ a function called convolution The concept behind them is that instead of connecting each neuron with all the next ones, we connect it with only a handful of them (the receptive field).
In a way, they try to regularize feedforward networks to avoid overfitting (when the model learns only pre-seen data and can’t generalize), which makes them very good in identifying spatial relationships between the data.
Face Recognition Based on Convolutional Neural Network
That’s why their primary use case is Computer Vision and applications such as image classification, video recognition, medical image analysis and self-driving cars where they achieve literally superhuman performance.
They are also ideal to combine with other types of models such as Recurrent Networks and Autoencoders. One such example is Sign Language Recognition.
Recurrent Neural Networks (RNN)
Recurrent networks are perfect for time-related data and they are used in time series forecasting. They use some form of feedback, where they return the output back to the input. You can think of it as a loop from the output to the input in order to pass information back to the network. Therefore, they are capable to remember past data and use that information in its prediction.
To achieve better performance researchers have modified the original neuron into more complex structures such as GRU units and LSTM Units. LSTM units have been used extensively in natural language processing in tasks such as language translation, speech generation, text to speech synthesis.
STFCN: Spatio-Temporal FCN for Semantic Video Segmentation
Recursive Neural Network
Recursive Neural Networks are another form of recurrent networks with the difference that they are structured in a tree-like form. As a result, they can model hierarchical structures in the training dataset.
They are traditionally used in NLP in applications such as Audio to text transcription and sentiment analysis because of their ties to binary trees, contexts, and natural-language-based parsers. However, they tend to be much slower than Recurrent Networks
AutoEncoders
Autoencoders are mostly used as an unsupervised algorithm and its main use-case is dimensionality reduction and compression. Their trick is that they try to make the output equal to the input. In other works, they are trying to reconstruct the data.
They consist of an encoder and a decoder. The encoder receives the input and it encodes it in a latent space of a lower dimension. The decoder takes that vector and decodes it back to the original input.
Autoencoder Neural Networks for Outlier Correction in ECG- Based Biometric Identification
That way we can extract from the middle of the network a representation of the input with fewer dimensions. Genius, right?
Of course, we can use this idea to reproduce the same but a bit different or even better data (training data augmentation, data denoising, etc)
Deep Belief Networks and Restricted Boltzmann Machines
Restricted Boltzmann Machines are stochastic neural networks with generative capabilities as they are able to learn a probability distribution over their inputs. Unlike other networks, they consist of only input and hidden layers( no outputs).
In the forward part of the training the take the input and produce a representation of it. In the backward pass they reconstruct the original input from the representation. (Exactly like autoencoders but in a single network).
Explainable Restricted Boltzmann Machine for Collaborative Filtering
Multiple RBM can be stacked to form a Deep Belief Network. They look exactly like Fully Connected layers, but they differ in how they are trained. That’s because they train layers in pairs, following the training process of RBMs (described before)
However, DBNs and RBMs have kind of abandoned by the scientific community in favor of Variational Autoencoders and GANs
Generative Adversarial Networks (GAN)
GANs were introduced in 2016 by Ian Goodfellow and they are based on a simple but elegant idea: You want to generate data, let’s say images. What do you do?
You build two models. You train the first one to generate fake data (generator) and the second one to distinguish real from fakes ones(discriminator). And you put them to compete against each other.
The generator becomes better and better at image generation, as its ultimate goal is to fool the discriminator. The discriminator becomes better and better at distinguish fake from real images, as its goal is to not be fooled. The result is that we now have incredibly realistic fake data from the discriminator.
Applications of Generative Adversarial Networks include video games, astronomical images, interior design, fashion. Basically, if you have images in your fields, you can potentially use GANs. Oooh, do you remember Deep Fakes? Yeah, that was all made by GANs.
Transformers
Transformers are also very new and they are mostly used in language applications as they are starting to make recurrent networks obsolete. They based on a concept called attention, which is used to force the network to focus on a particular data point.
Instead of having overly complex LSTM units, you use Attention mechanisms to weigh different parts of the input based on their significance. The attention mechanism is nothing more than another layer with weights and its sole purpose is to adjust the weights in a way that prioritizes segments of inputs while deprioritizing others.
Transformers, in fact, consist of a number of stacked encoders (form the encoder layer), a number of stacked decoders (the decoder layer) and a bunch of attention layers (self- attentions and encoder-decoder attentions)
http://jalammar.github.io/illustrated-transformer/
Transformers are designed to handle ordered sequences of data, such as natural language, for various tasks such as machine translation and text summarization. Nowadays BERT and GPT-2 are the two most prominent pretrained natural language systems, used in a variety of NLP tasks, and they are both based on Transformers.
Graph Neural Networks
Unstructured data are not a great fit for Deep Learning in general. And there are many real-world applications where data are unstructured and organized in a graph format. Think social networks, chemical compounds, knowledge graphs, spatial data.
Graph Neural Networks purpose is to model Graph data, meaning that they identify the relationships between the nodes in a graph and produce a numeric representation of it. Just like an embedding. So, they can later be used in any other machine learning model for all sorts of tasks like clustering, classification, etc.
Deep Learning in Natural Language Processing (NLP)
Word Embeddings
Word Embeddings are the representations of words into numeric vectors in a way that capture the semantic and syntactic similarity between them. This is necessary because neural networks can only learn from numeric data so we had to find a way to encode words and text into numbers.
Word2Vec is the most popular technique and it tries to learn the embeddings by predicting a word based on its context (CBOW) or by predicting the surrounding words based on the word (Skip-Gram). Word2Vec is nothing more than a simple neural network with 2 layers that has words as inputs and outputs. Words are fed to the Neural Network in the form of one-hot encoding.
In the case of CBOW, the inputs are the adjacent words and the output is the desired word. In the case of Skip-Gram, it’s the other way around.
Glove is another model that extends the idea of Word2Vec by combining it with matrix factorization techniques such as Latent Semantic Analysis, which are proven to be really good as global text statistics but unable to capture local context. So the union of those two gives us the best of both worlds.
FastText by Facebook utilizes a different approach by making use of character-level representation instead of words.
Contextual Word Embeddings replace Word2Vec with Recurrent Neural Networks to predict, given a current word in a sentence, the next word. That way we can capture long term dependencies between words and each vector contains both the information on the current word and on the past ones. The most famous version is called ELMo and it consists of a two-layer bi-directional LSTM network.
Attention Mechanisms and Transformers are making RNN’s obsolete (as mentioned before), by weighting the most related words and forgetting the unimportant ones.
Sequence Modeling
Sequence models are an integral part of Natural Language Processing as it appears on lots of common applications such as Machine Translation, Speech Recognition, Autocompletion and Sentiment Classification. Sequence models are able to process a sequence of inputs or events such as a document of words.
For example, imagine that you want to translate a sentence from English to French.
To do that you need a Sequence to Sequence model (seq2sec). Seq2sec models include an encoder and a decoder. The encoder takes the sequence(sentence in English) and produces an output, a representation of the input in a latent space. This representation is fed to the decoder, which gives us the new sequence (sentence in France).
The most common architectures for encoder and decoder are Recurrent Neural Networks (mostly LSTMs) because they are great in capturing long term dependencies and Transformers that tend to be faster and easier to parallelize. Sometimes they are also combined with Convolutional Networks for better accuracy.
BERT and GPT-2 are considered the two best language models and they are in fact Transformer based Sequence models .
Deep Learning in Computer Vision
Stanford University School of Engineering
Localization and Object Detection
Image Localization is the task of locating objects in an image and mark them with a bounding box, while object detection includes also the classification of the object.
The interconnected tasks are tackled by a fundamental model (and its improvements) in Computer Vision called R-CNN. RCNN and it’s predecessors Fast RCNN and Faster RCNN take advantage of regions proposals and Convolutional Neural Network.
An external system or the network itself( in the case of Faster RCNN) proposes some regions of interest in the form of a fixed-sized box, which might contain objects. These boxes are classified and corrected via a CNN (such as AlexNet), which decided if the box contains an object, what the object is and fixes the dimensions of the bounding box.
Single-shot detectors
https://github.com/karolmajek/darknet-pjreddie
Single-shot detectors and it’s most famous member YOLO (You Only Look Once) ditch the idea of region proposals and they use a set of predefined boxes.
These boxes are forwarded to a CNN, which predicts for each one a number of bounding boxes with a confidence score, it detects one object centered in it and it classifies the object into a category. In the end, we keep only the bounding boxes with a high score.
Over the years YOLOv2, YOLOv3, and YOLO900 improved on the original idea both on speed and accuracy.
Semantic Segmentation
ICNet for Real-Time Semantic Segmentation on High-Resolution Images
One of the fundamentals tasks in computer vision is the classification of all pixels in an image in classes based on their context, aka Semantic Segmentation. In this direction, Fully Convolutional Networks (FCN) and U-Nets are the two most widely-used models.
Fully Convolutional Networks (FCN) is an encoder-decoder architecture with one convolutional and one deconvolutional network. The encoder downsamples the image to capture semantic and contextual information while the decoder upsamples to retrieve spatial information. That way we manage to retrieve the context of the image with the smaller time and space complexity possible.
U-Nets are based on the ingenious idea of skip-connections. Their encoder has the same size as the decoder and skip-connections transfer information from the first one to the latter in order to increase the resolution of the final output.
Pose Estimation
Pose Estimation is the problem of localizing human joints in images and videos and it can either be 2D or 3D. In 2D we estimate the (x,y) coordinates of each joint while in 3D the (x,y,z) coordinates.
PoseNet dominates the field (it’s the go-to model for most smartphone applications) of pose estimation and it uses Convolutional Neural Networks (didn’t see that coming, did you?). We feed the image to a CNN and we use a single-pose or a multi-pose algorithm to detect poses. Each pose is associated with a confidence score and some key points coordinates. In the end, we keep the ones with the highest confidence score.
Wrapping up
There you have it. All the essential Deep Learning algorithms at the time.
Of course, I couldn’t include all the published architectures, because they are literally thousands. But most of them are based on one of those basic models and improve it with different techniques and tricks.
I am also confident that I will need to update this guide pretty soon, as new papers are coming out as we speak. But that is the beauty of Deep Learning. There is so much room for new breakthroughs that it’s kind of scary.
If you think that I forgot something, don’t hesitate to contact us on Social media or via email. I want this post to be as complete as possible.
Now it’s your time. Go and build your own amazing applications using these algorithms. Or even create a new one, that will make the cut in our list. Why not?
Have fun and keep learning AI.
* Disclosure: Please note that some of the links above might be affiliate links, and at no additional cost to you, we will earn a commission if you decide to make a purchase after clicking through.