Oleg Zabluda's blog
Thursday, September 22, 2016
 
Understanding Locally Competitive Networks (2014) Rupesh Kumar Srivastava, Jonathan Masci, Faustino Gomez, Jürgen...
Understanding Locally Competitive Networks (2014) Rupesh Kumar Srivastava, Jonathan Masci, Faustino Gomez, Jürgen Schmidhuber
"""
ReLU (Glorot et al., 2011)), maxout (Goodfellow et al., 2013a) and LWTA (Srivastava et al., 2013) are quite unlike sigmoidal activation [...] A common theme [...] is that they are locally competitive. Maxout and LWTA utilize explicit competition between units in small groups within a layer, while in the case of the rectified linear function, the weighted input sum competes with a fixed value of 0. [...] We start from the observation that in locally competitive networks, a subnetwork of units has nonzero activations for each input pattern [OZ: huh]. Instead of treating a neural network as a complex function approximator, the expressive power of the network can be interpreted to be coming from its ability to activate different subsets of linear units for different patterns. We hypothesize that the network acts as a model that can switch between “submodels” (subnetworks) such that similar submodels respond to similar patterns. As evidence of this behavior, we analyze the activated subnetworks for a large subset of a dataset (which is not used for training) and show that the subnetworks activated for different examples exhibit a structure consistent with our hypothesis. These observations provide a unified explanation for improved credit assignment in locally competitive networks during training, which is believed to be the main reason for their success. Our new point of view suggests a link between these networks and competitive learning approaches of the past decades.
"""
https://arxiv.org/abs/1410.1165
https://arxiv.org/abs/1410.1165

Labels:


- posted by Oleg Zabluda @ 10:20 AM | permalink | | | technorati | bloglines | feedster
| |

Home

Powered by Blogger