Neural Networks - A Brief Description

A human brain continually receives input signals from many sources and processes them to create the appropriate output response. Our brains have billions of neurons that interconnect to create elaborate “neural networks”. These networks execute the millions of necessary functions needed to sustain normal life. For some years now, researchers have been developing models, both in hardware and in software, that mimic a brain’s cerebral activity in an effort to produce an ultimate form of artificial intelligence. Many theoretical models (or paradigms), dating as far back as the 1950’s, have been developed. Most have had limited real-world application potential, and thus, neural networks have remained in relative obscurity for decades. The backpropagation paradigm, however, is largely responsible for changing this trend. It is an extremely effective learning tool that can be applied to a wide variety of problems. Backpropagation related paradigms require supervised training. This means they must be taught using a set of training data where known solutions are supplied.

Backpropagation type neural networks process information in interconnecting processing elements (often termed neurons, units or nodes—we will use “nodes”). These nodes are organized into groups termed layers. There are three distinct types of layers in a backpropagation neural network: the input layer, the hidden layer(s) and the output layer. A network consists of one input layer, one or more hidden layers and one output layer. Connections exist between the nodes of adjacent layers to relay the output signals from one layer to the next. Fully connected networks occur when all nodes in each layer receive connections from all nodes in each preceding layer. Information enters a network through the nodes of the input layer. The input layer nodes are unique in that their sole purpose is to distribute the input information to the next processing layer (i.e., the first hidden layer). The hidden and output layer nodes process all incoming signals by applying factors to them (termed weights). Each layer also has an additional element called a bias node. Bias nodes simply output a bias signal to the nodes of the current layer. Qnet handles these bias nodes automatically. They do not need to be included or specified by the user. All inputs to a node are weighted, combined and then processed through a transfer function that controls the strength of the signal relayed through the node’s output connections. A nodes operation is shown in figure 2 below. The transfer function serves to normalize a node’s output signal strength between 0 and 1. Qnet provides four transfer functions: the sigmoid (default), gaussian, hyperbolic tangent and hyperbolic secant functions. Qnet allows the transfer function to be selected on a layer-by-layer basis to create unique hybrid networks. Network processing continues through each layer until the network’s response is obtained at the output layer.

Figure 5.1 - Neural Network Design Topology.

When a network is used in recall mode, processing ends at the output layer. During training, the network’s response at the output layer is compared to a supplied set of known answers (training targets). The errors are determined and backpropagated though the network in an attempt to improve the network’s response. The nodal weight factors are adjusted by amounts determined by the training algorithm. The iterative procedure of processing inputs through the network, determining the errors and backpropagating the errors through the network to adjust the weights constitutes the learning process. One training iteration is complete when all supplied training cases have been processed through the network. The training algorithms adjust the weights in an attempt to drive the network’s response error to a minimum. Two factors are used to control the training algorithm’s adjustment of the weights. They are the “learning rate coefficient”, eta, and the “momentum factor”, alpha. If the learning rate is too fast (i.e., eta is too large), network training can become unstable. If eta is too small, the network will learn at a very slow pace. The momentum factor has a smaller influence on learning speeds, but it can influence training stability and promote faster learning for most networks. Qnet uses a sophisticated control scheme that adjusts the learning rate coefficient to keep network training proceeding at a near optimal pace.

Figure 5.2 - Node/Neuron Functionality. The dot product is computed between the inputs and the weight vector. That result is passed through the transfer function to obtain the node’s output signal strength. The output is passed through to the next layer until the ouput layer is reached.