When is Training Complete?

Determining when training has completed is not always a simple process. Often, determining the optimal training point is possible only after many iterations have been run. For example, if a network is overtraining based on RMS error, we must check to make sure that the minimum test set error is not a local one. If the training error is making little or no progress in the downward direction, the network could be temporarily stalled at a learning “plateau”. Usually, only continued iterations allow the identification of the true training state.

In some instances, the standard RMS error may not be the optimal analysis tool. Some models may be better optimized using correlation or tolerance numbers. Furthermore, overtraining in the test set may not occur in RMS error, correlation and tolerance at the exact same location. You may decide to terminate training when any one of the three test set error tracking items begin overtraining or only after all three of the items reverse and overtrain. The methods to determine when a network has reached its converged or optimal state are:

• RMS test error analysis: Terminate the training process at the point where the test set error is at a minimum. The network is optimized for accuracy on cases available outside the training set. All cases carry equal weight.
• Test set correlation analysis: Terminate the training process at the point where the correlation error is at a maximum. The network is optimized for minimum error by the test set cases when the correlation coefficient is at its maximum. This differs from RMS error in that the outputs and targets are weighted by their distance from the mean, thus, increasing the importance cases at the extreme minimums and maximums. Overtraining in this item occurs when the test set correlation coefficient declines indefinitely.
• Test set tolerance analysis: If there is a known accuracy that needs to be reached, Qnet’s tolerance checking can be used to determine when the network has reached the required level of accuracy. This item measures the percentage of cases where the prediction falls within a desired tolerance. Overtraining in this item occurs when the test set percentage declines indefinitely.
• No further significant decline in the training error: If overtraining does not occur (or no test set is used) and a pre-defined tolerance does not exist, this method will optimize network accuracy for cases contained in the training set.

The appropriate method varies depending on the type of model being developed. Financial forecasting models often produce better results using method 2 to optimize the extreme predictions. These cases have the largest influence on correlation and are the cases that will most likely result in buy/sell decisions. A minimum RMS error optimizes by weighting all cases equally. For all types of models, large, randomly selected test sets will produce the most accurate training/overtraining analysis and, also, increase the likelihood that all methods will produce similar overtraining results.