An empirical comparison of back propagation and the RDSE algorithm on continuously valued real world data.

Abstract

The ability of a neural network to generalise is dependent on how representative the training patterns were of the whole data domain, and how smoothly the network has fitted to these patterns [Sethi, I.K. (1990). IEEE International Joint Conference on Neural Networks, Seattle, WA, Vol. 2, pp. 219-224]. In non-scaled continuous data domains, training examples will lie at differing distances from each other, making the fitting problem more difficult and varied. This paper introduces a new neuron with an adaptive steepness parameter, implemented as an extra internal connection, which is altered to better interpolate between the data points that its hyperplane divides. Networks of the new neuronal model are trained using a new paradigm entitled the random directed search by entropy algorithm (RDSE). This involves constructing a network by training one neuron at a time and freezing the weights. Each neuron is trained using directed random search [Baba (1989). Neural Networks, 2, 367-373] to find a hyperplane that separates examples by minimising an entropy measure [Quinlan (1986). Induction of Decision Trees, Machine Learning, Vol. 1, pp. 81-106]. This training paradigm solves the problem of pre-defining a network topology, has few problems with local minima, can handle unscaled continuous input data and can be fully trained in a relatively short time scale when compared with other methods, e.g. back propagation (BP).An example benchmark problem is used to illustrate the effects of the new neuronal model, and results for two real world data domains are given which display an improved classification rate when compared against networks with a constant steepness value for every neuron. An empirical comparison between BP and RDSE for the two data sets are also given. These results display improved training times, robustness and classification rates by RDSE when compared against BP.