Over a million developers have joined DZone.

Artificial Neural Networks: Some Misconceptions (Part 4)

DZone's Guide to

Artificial Neural Networks: Some Misconceptions (Part 4)

Continue learning about some misconceptions around artificial neural networks. These misconceptions have to do with training data and retraining ANNs.

· AI Zone ·
Free Resource

Insight for I&O leaders on deploying AIOps platforms to enhance performance monitoring today. Read the Guide.

Learn about two more major misconceptions and artificial neural networks.

Neural Networks Cannot Be Trained on Any Data

One of the biggest reasons why neural networks may not work is because people do not properly pre-process the data being fed into the neural network. Data normalization and the removal of redundant information and outliers should all be performed to improve the probability of good neural network performance.

Data Normalization

With data normalization, neural networks consist of various layers of perceptrons linked together by weighted connections. Each perceptron contains an activation function, each with an active range (except for radial basis functions). Inputs into the neural network need to be scaled within this range so that the neural network is able to differentiate between different input patterns.

For example, consider a neural network trading system that receives indicators about a set of securities as inputs, and outputs whether each security should be bought or sold. One of the inputs is the price of the security (we are using the Sigmoid activation function). However, most of the securities cost between $5 and $15 per share and the output of the Sigmoid function approaches 1.0. So, the output of the Sigmoid function will be-be 1.0 for all securities, all of the perceptrons will fire, and the neural network will not learn.

Neural networks trained on unprocessed data to produce models where the lights are on but nobody’s home.

Outlier Removal

An outlier is a value that is much smaller or larger than most of the other values in some set of data. Outliers can cause problems with statistical techniques like regression analysis and curve fitting because when the model tries to accommodate the outlier, the performance of the model across all other data deteriorates.

This diagram shows the effect of removing an outlier from the training data for a linear regression. The results are comparable to neural networks. (Image soure here.)

The illustration shows that trying to accommodate an outlier into the linear regression model results in the poor fit of the dataset. The effect of outliers on non-linear regression models, including neural networks, is similar. Therefore, it is good practice is to remove outliers from the training dataset. That said, identifying outliers is a challenge in and of itself. This tutorial and paper discuss existing techniques for outlier detection and removal.

Remove Redundancy

When two or more of the independent variables being fed into the neural network are highly correlated (multiple colinearity), this can negatively affect the neural network's learning ability. Highly correlated inputs also mean that the amount of unique information presented by each variable is small, so the less significant input can be removed. Another benefit to removing redundant variables is faster training times. Adaptive neural networks can be used to prune redundant connections and perceptrons.

Neural Networks May Need to Be Retrained

Given that you were able to train a neural network to trade successfully in and out of an example, this neural network may still stop working over time. This is not a poor reflection on neural networks but rather an accurate reflection of the financial markets. Financial markets are complex adaptive systems, meaning that they are constantly changing. What worked yesterday may not work tomorrow. This characteristic is called a non-stationary or dynamic optimization problem — and neural networks are not particularly good at handling them.

Dynamic environments, such as financial markets, are extremely difficult for neural networks to model. Two approaches are to keep retraining the neural network over time or to use a dynamic neural network. Dynamic neural networks "track" changes to the environment over time and adjust their architecture and weights accordingly. They are adaptive over time. For dynamic problems, multi-solution meta-heuristic optimization algorithms can be used to track changes to local optima over time. One such algorithm is the multi-swarm optimization algorithm, a derivative of the particle swarm optimization. Additionally, genetic algorithms with enhanced diversity or memory have also been shown to be robust in dynamic environments.

The illustration below demonstrates how a genetic algorithm evolves over time to find new optima in a dynamic environment. This illustration also happens to mimic trade crowding, which is when market participants crowd a profitable trading strategy, thereby exhausting trading opportunities, causing the trade to become less profitable.

This animated image shows a dynamic fitness landscape (search space) change over time. (Image source here.)

TrueSight is an AIOps platform, powered by machine learning and analytics, that elevates IT operations to address multi-cloud complexity and the speed of digital transformation.

ai ,neural network ,deep learning ,training data ,data normalization

Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}