The recurrent neural network (RNN) is a special type of neural network equipped with additional recurrent connections. Those u- nique recurrent connections enable the RNN to store the processed information and capture the long-range dependencies between input data. Therefore, the RNN has been regarded as an expressive model to process nonlinear sequential processing tasks, such as speech recognition, nature language processing and even video indexing [14]. However, the large computation complexity makes it difficult to effectively train a RNN and therefore significantly limits the research on RNNs in the last 20 years.