Long Short Term Memory

Hinton tells us that in RNNs with LSTM, the gates (write, keep and read) of the memory cells are turned on or off by “the rest of the network”. He also tells us that the gates are logistic, which allows the gradient to flow easily as we backpropagate through time. But how exactly is the state of the gates set? And does this occur during the forward pass or during back-propagation?


1 Response to “Long Short Term Memory”

  1. 1 Nicholas LĂ©onard March 1, 2013 at 13:59

    The gates use sigmoid activation functions. When activations is over a threshold, the gate is opened, else it is kept closed. Each gate is associated to one or many weights, i.e. a connection. When the gate is open, spreading activations may flow through the gated connection. Otherwise, the activations is not allowed to flow through. Backpropagation takes the reverse of this flow, i.e. the same path, but from end to start. A gate is set during the feedforward phase, but this also affects the backpropagation phase, since it is limited by the path of the feedforward.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: