Deep Neural Networks With Python
In this Deep Neural Networks article, we look at Deep Learning, its types, the challenges it faces, and Deep Belief Networks.
Join the DZone community and get the full member experience.Join For Free
What Is Python Deep Neural Networks?
Before finding out what a deep neural network is, let’s learn about Artificial Neural Networks.
Artificial Neural Networks
An ANN (Artificial Neural Network) is inspired by the biological neural network. It can learn to perform tasks by observing examples, we do not need to program them with task-specific rules. An ANN can look at images labeled 'cat’ or ‘no cat’ and learn to identify more images itself.
Such a network is a collection of artificial neurons- connected nodes; these model neurons in a biological brain. A connection is like a synapse in a brain and is capable of transmitting signals from one artificial neuron to another. This neuron processes the signal it receives and signals to more artificial neurons it is connected to.
This way, we can have input, output, and hidden layers.
Some applications of Artificial Neural Networks have been Computer Vision, Speech Recognition, Machine Translation, Social Network Filtering, Medical Diagnosis, and playing board and video games.
Deep Neural Networks
Coming back, a Deep Neural Network is an ANN that has multiple layers between the input and the output layers. Such a network sifts through multiple layers and calculates the probability of each output.
A DNN is capable of modeling complex non-linear relationships.
Structure of Deep Neural Network
A DNN is usually a feedforward network. This means data from the input layer flows to the output layer without looping back.
Such a network with only one hidden layer would be a non-deep(or shallow) feedforward neural network. But in a deep neural network, the number of hidden layers could be, say, 1000. But it must be greater than 2 to be considered a DNN.
A DNN creates a map of virtual neurons and randomly assigns weights to the connections between these neurons. It multiplies the weights with the inputs to return an output between 0 and 1. If it fails to recognize a pattern, it uses an algorithm to adjust the weights.
Types of Deep Neural Networks
Broadly, we can classify Deep Neural Networks into two categories:
Recurrent Neural Networks (RNNs)
A Recurrent Neural Network is a sort of ANN where the connections between its nodes form a directed graph along a sequence. An RNN can use its internal state/ memory to process input sequences. Thus we can use it for tasks like unsegmented, connected handwriting recognition and speech recognition. Kinds of RNN:
- Finite Impulse Recurrent Network: A Directed Acyclic Graph (DAG) that we can replace with a strictly feedforward neural network.
- Infinite Impulse Recurrent Network: A Directed Cyclic Graph that we cannot unroll.
A basic RNN is a network of neurons held into layers where each node in a layer connects one-way (and directly) to every other node in the next layer. In an RNN, data can flow in any direction. We make use of LSTM (Long Short-Term Memory) and use RNNs in applications like language modeling.
Convolutional Neural Network (CNN or ConvNet)
A CNN is a sort of deep ANN that is feedforward. We use it for applications like analyzing visual imagery, Computer Vision, acoustic modeling for Automatic Speech Recognition (ASR), Recommender Systems, and Natural Language Processing (NLP).
A CNN uses multilayer perceptrons for minimal preprocessing. In such a network, the connectivity pattern between neurons mimics how an animal visual cortex is organized. A CNN learns the filters and thus needs little preprocessing. It has the following architecture:
Challenges to Deep Neural Networks
DNNs face two major challenges:
Since a DNN possesses added layers of abstraction, it can model rare dependencies in the training data. To fight this, we can:
- Use regularization methods like Ivakhnenko’s unit pruning, weight decay, or sparsity.
- Using dropout regularization to randomly omit units from hidden layers when training.
- Using methods like cropping and rotating to augment data; to enlarge smaller training sets.
To sweep through the parameter space (size, learning rate, initial weights) may lead to a need for more computational resources and time. To battle this, we can:
- Perform Batching to compute the gradient to multiple training examples at once.
- Use many-core architectures for their large processing capabilities and suitability for matrix and vector computations.
Deep Belief Networks
Before we can proceed to exit, let’s talk about one more thing — Deep Belief Networks. A DBN is a sort of deep neural network that holds multiple layers of latent variables or hidden units. Such a network observes connections between layers rather than between units at these layers.
If we train a DBN on a set of examples without supervision, we can let it learn to reconstruct input probabilistically. You can call the layers feature detectors. After this, we can train it with supervision to carry out classification.
So, this was all in Deep Neural Networks with Python. Hope you like our explanation.
In this Deep Neural Networks article, we looked at Deep Learning, its types, the challenges it faces, and Deep Belief Networks. Leave your suggestions and queries in the comments.
Published at DZone with permission of Rinu Gour. See the original article here.
Opinions expressed by DZone contributors are their own.