of Dynamic Movement Primitives in Neural Space: A FORCE-Learning Approach

(1)

Here, $x_{n},n = 1,\ldots,N$ is the internal state of the n-th neuron, $u_{m},m = 1,\ldots,M$ is the m-th input current, and $z_{l},l = 1,\ldots,L$ is the state of l-th readout neuron determined as

$\displaystyle\begin{array}{rcl} z_{l}(t)& =& \sum _{n=1}^{N}w_{ ln}^{\mathrm{out}}r_{ n}(t),\ r_{n}(t) =\tanh (x_{n}(t)).{}\end{array}$

(2)

From the spirit of RC, in the FORCE-learning process, $w^{\mathrm{rec}} = (w_{nn^{{\prime}}}^{\mathrm{rec}}),w^{\mathrm{in}} = (w_{ nm}^{\mathrm{in}})$ and w ^fb = (w _nl^fb) are fixed and sparsely random matrices, and only the matrix w ^out = (w _ln^out) is incrementally modified according to the recursive least square (RLS) or the gradient methods where the difference between the readout states and teacher signals is used as error signals [6].

In addition to the basic structure of the FORCE-learning, we also incorporate an intermediate layer with K neurons as shown in Fig. 1, in order to (i) reduce the number of essential dimensionality of the dynamics inside the RNN and (ii) compare RNNs with different topologies (different realizations of random numbers for preparing the recurrent connectivity matrix w ^rec). Here, the connections between neurons in RNN and ones in the intermediate layer is incrementally updated through the “generalized Hebbian-learning (GHA)” [7] as

$\displaystyle\begin{array}{rcl} \varDelta w_{kn}^{gH}& =& \eta (r_{ n}(t) -\sum _{k^{{\prime}}=1}^{k}w_{ k^{{\prime}}n}^{gH}y_{ k^{{\prime}}}(t))y_{k}(t){}\end{array}$

(3)

The k-th neurons y _kgives the k-th principal component of the dynamics of RNN. It is expected that the dynamics of two RNNs are statistically equivalent, the readout matrices for these two RNNs are almost same against the same teacher signal. We call neural networks with the structure shown in Fig. 1 the FORCE-GHA networks in the following.

Fig. 1

Incorporation of generalized Hebbian-learning with FORCE-learning

3 Numerical Experiments and Results

As an illustrative example of our approach, we consider generation of reaching movement patterns executed by a two-link arm (Fig. 2a) by the training of the FORCE-GHA networks. Here, the purpose of neural networks is to generate the torques u ₁ and u ₂ that rotate the links of an arm. In actual motor learning, the data of the torques is not directly given as teacher signals, neural networks have to acquire them in an unsupervised way such as reinforcement learning. But, this is not a main topic in this study. So, to skip this problem, we first generate arm movement trajectories using the minimum jerk principle [8] in the joint space, then obtain the torque data using the inverse model of a two-link arm used in [9]. We employ these torque data as teacher signals for the training of the FORCE-GHA networks.