power is required to support the training phase. Training phase is the critical operation to obtain a neural network for a specific task. It usually demands a large volume of memory and computing resources, and the time consumption can range from a few seconds to hundreds of hours, depending on the scale of the networks [6]. In addition, recent research has demonstrated that the performance of DNNs, such as the classification accuracy, can be drastically improved by increasing the scale of the network [7]. Therefore, in order to better process big data with large scale neural networks, we need to explore more computing power.