Training Divergence

During the training process, the network’s learning pace may accelerate to unstable levels. When this happens, learning instabilities develop. These instabilities show up as oscillations in the training error. If the learning rate is not lowered in response to the instabilities, network divergence is possible. Qnet’s Learn Rate Control helps prevent divergence by automatically lowering eta. If a network does diverge, the training RMS error will normally reach a large constant value (around 0.5 for many models). For multiple output node networks, it is possible to have a divergence at one output node and have continued convergence for others. In these cases, only a minor increase in the training error may be observed, followed by a continued decrease in the training error. Such cases are often difficult to detect by monitoring RMS error alone.

Qnet offers two tools for detecting diverged conditions. From the training menu, selecting "Info/Divergence Checking" will quickly scan all nodes for diverged characteristics. The first output node that appears diverged will be indicated. It is also possible to spot diverged nodes by plotting each output responses with NetGraph (Targets/Outputs vs Pattern Sequence). If an output node's response is constant and completely outside the range of the target data, then that node has diverged. This indicates that the normalized response of the network is all 1's or 0's.

A diverged network must be either reset to a randomized state (select "Options/Initialize-Reset Weights" from the training menu) or returned to an iteration prior to divergence (using the AutoSave feature). Use the AutoSave recovery procedure for cases where a great deal of training would be lost by simply resetting the network. Setting a lower learn rate (eta) value or limiting LRC with a lower maximum eta will help to prevent the divergence from reoccurring.