Weight penalties vs weight contraints

What are the differences between weight penalties and weight constraints? What are the advantages and drawbacks?


2 Responses to “Weight penalties vs weight contraints”

  1. 1 Pierre Luc Carrier March 12, 2013 at 08:40

    When using a weights penalty, you inclure an new term representing the penalty in the cost function you optimize. The cost to obtimize thus becomes : $Cost + \lambda ||\Theta||^2 $.

    When using a weight constraint, you optimize the same cost as before but such that the constraint $ ||\Theta||^2 < C $ is respected.

    Weight penalty and weight constraints are somewhat equivalent because for every $\lambda$ value used with the penalty you can find a value of C in the weight constraint that for which the optimal solution will be the same. However, they differ in the fact that weight penalty starts pushing down on the weight norms as soon as training starts while weight constraints will have no effect on the weights as long as their norm is smaller than C. This sort of gives the weights a 'grace' period where they can evolve more freely.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: