On the Predictive Effects of Markovian and Architectural Factors of Echo State Networks

Echo State Networks (ESNs) represent an emerging paradigm for
modeling Recurrent Neural Networks (RNNs).
In this report we try to identify and investigate some of the main
aspects that can be accounted for the success and limitations of this
class of models.
Independently of the architectural design, we first show the effect on
ESNs behavior due to the contractivity of the state transition
function and the related Markovian bias.
The purpose of our study is also to give an insight on how and why a
larger reservoir may improve the predictive performance.  We identify
four key factors which can influence the performance of ESNs: input
variability, multiple time-scales dynamics, non-linear interactions
among units and regression in a high dimensional state space. Several
variants of the basic ESN model are introduced in order to study these
main factors. The proposed variants are tested on four datasets: the
Mackey-Glass chaotic time series, the 10th order NARMA system, and two
predictive tasks on a symbolic sequence domain with
Markovian/anti-Markovian flavor.  Experimental evidence shows that
all the key identified factors have a major role in determining ESNs
performances.