Aiming at problem that recurrent neural network（ RNN） structure has drawbacks of slow convergence and poor training effect in deep network

gated structural characteristics of long short-term memory（ LSTM） and highway network are analyzed

and a cross-layer integration gated unit（ CIGU） which connects information between layers is proposed. Combined with characteristics of time extension of RNN

by designing inter-layer gate structure

let CIGU model can have long and short-term memory abilities when it has backward gradient descent in space

which is similar to LSTM when propagating in time

so as to strengthens deep learning ability of RNN in space. The proposed structure is applied to LSTM

and train and test on different gate structures by PTB language data sets.Results show that with the deepening of model layers

speed of training convergence and testing results of CIGU are significantly improved compared with traditional LSTM and LSTM based on highway network structure.