Stable behavior in a recurrent neural network for a finite state machine.

Abstract

For the learning of a finite state machine (FSM) by a recurrent neural network (RNN), we think about how to train an RNN so as to stably mimic an FSM even for sequences having a long length. First, we consider the relationship between the stable behavior and the internal representation of states, that is, clusters of the internal units' outputs. As for this relationship, we prove that an RNN can get the stable cluster transitions when a neuron activation parameter is larger than a certain finite value micro0. Secondly, to acquire the stable behavior, we regard the internal representation for the stable behavior as prior knowledge. This produces a new target function of learning with internal representation term. We derive a Bayesian style method to estimate coefficients of the terms in the function, corresponding to hyperparameters. Finally, experiments show that RNNs readily acquire stable behavior by using our proposed method.