1) what is the difference in setting the hyper-parameters between RBM models and DAE models?

2) In mini-batch gradient descent which one is more important, “number of mini-batch updates” or “visiting all of the training data”?

3) Name some of the advantages of the symbolic languages like Theano?


2 Responses to “”

  1. 1 Sina Honari February 7, 2013 at 13:39

    1) In DAE, we can tune the hyper-parameters by using the error of the model. However, in the RBM model the error (or optimization criteria) is not easily measured. So, in DAE by measuring the performance on the out of sample data (like the validation set) we can evaluate it’s performance which is not the case for the RBM models.

    2) If we have a fixed set of training set, the number of mini-batch updates is more important, because that specifies how often we update the parameters of the model before they converge. However, if we have the choice of restricting the learning algorithm to a small data set versus visiting a much bigger training data, it’s always better to use a bigger training set, even if we have the chance of visiting each training example only once. Because that helps our model to generalize better.

  2. 2 Vincent Archambault-B February 4, 2013 at 11:31

    For part 3), the advantages of symbolic languages like Theano are :

    a) The expression can be compiled in any other language and target CPU or GPU architecture.
    b) Auto differentiation. This saves you the effort of computing the gradient.
    c) Many numerical optimizations can be done behind the scene, allowing for a faster and more robust implementation.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: