AutoSave

The AutoSave feature of Qnet allows you to easily recover from overtraining situations and training divergence. The user specifies a rate (or interval) at which the network should be stored. Qnet will store the network state in a temporary file at the selected interval during the training process. An AutoSave rate of 1000 will cause the network state to be stored every 1000th training iteration. The network state is not saved to the current network definition file. Training snap shots are saved in a temporary file and must be retrieved and stored to a permanent network definition file by selecting File/Save AutoSave Network... When selected, you will be prompted for a network definition file name and the iteration number that is to be used in the recovery process. You may use the name of the current network definition file or a new one. The iteration number can be determined by viewing the training or test set error history. For overtraining, return to the first stored iteration prior to the point that the test set error started to increase. To recover from a diverged network, return to the first iteration prior to the divergence. The current training run is unaffected by this recovery process. If you wish to start training the network retrieved from AutoSave, exit the current training session and start a training run using the network definition file specified.

The temporary AutoSave file is destroyed when the current training run is terminated. You must retrieve any desired network states before exiting. Also, if you save a network state to the same file name as the one being used for the current run, do not save the current run prior to exiting or the retrieved network will be overwritten.

When selecting the AutoSave rate, several considerations must be made. Storing the network too often can slow execution speeds and increase disk space demands. Selecting very long intervals between AutoSave stores will mean that it could take considerable training time to get back to the desired network state. Limiting AutoSave about 5 minutes between stores will have the following benefits:

1. You can recover from a training problem in a reasonable period of time.
2. Longer time intervals between AutoSave stores will reduce the amount of disk space required. Small networks will typically require less than a few megabytes of disk space for an overnight run if the saves are at least 5 minutes apart. If very long unattended sessions are planned (i.e., weekends or longer), care should be taken if disk space is limited. A simple calculation determines the space required:
   (bytes of disk space required) = (byte size of a saved network’s .NET file) * (# training iterations per hour) * (hours of unattended training planned) / (AutoSave rate).
   The "iterations per minute" number can be computed by dividing the "Max Iterations" by the "Time Remaining" field (in hours) displayed at the start of any training run.
3. Using longer intervals between stores will reduce the impact of AutoSave on training speed. Disk writes are slow and can significantly impede execution times if they are performed often.