Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
"""
In this paper, we empirically show that on the ImageNet dataset large minibatches cause optimization difficulties, but when these are addressed the trained networks exhibit good generalization. Specifically, we show no loss of accuracy when training with large minibatch sizes up to 8192 images. To achieve this result, we adopt a linear scaling rule for adjusting learning rates as a function of minibatch size and develop a new warmup scheme that overcomes optimization challenges early in training. With these simple techniques, our Caffe2-based system trains ResNet-50 with a minibatch size of 8192 on 256 GPUs in one hour, while matching small minibatch accuracy. Using commodity hardware, our implementation achieves ∼90% scaling efficiency when moving from 8 to 256 GPUs.
"""
https://research.fb.com/publications/ImageNet1kIn1h/

https://research.fb.com/publications/ImageNet1kIn1h/

Labels: Oleg Zabluda

- posted by Oleg Zabluda @ 7:39 PM | permalink | | | technorati | bloglines | feedster

 
Unsupervised Sentiment Neuron
Unsupervised Sentiment Neuron
"""
We’ve developed an unsupervised system which learns an excellent representation of sentiment, despite being trained only to predict the next character in the text of Amazon reviews.

A linear model using this representation achieves state-of-the-art sentiment analysis accuracy on a small but extensively-studied dataset, the Stanford Sentiment Treebank (we get 91.8% accuracy versus the previous best of 90.2%), and can match the performance of previous supervised systems using 30-100x fewer labeled examples. Our representation also contains a distinct “sentiment neuron” which contains almost all of the sentiment signal.
[...]
We were very surprised that our model learned an interpretable feature, and that simply predicting the next character in Amazon reviews resulted in discovering the concept of sentiment. We believe the phenomenon is not specific to our model, but is instead a general property of certain large neural networks that are trained to predict the next step or dimension in their inputs.

Methodology

We first trained a multiplicative LSTM with 4,096 units on a corpus of 82 million Amazon reviews to predict the next character in a chunk of text. Training took one month across four NVIDIA Pascal GPUs, with our model processing 12,500 characters per second.

These 4,096 units (which are just a vector of floats) can be regarded as a feature vector representing the string read by the model. After training the mLSTM, we turned the model into a sentiment classifier by taking a linear combination of these units, learning the weights of the combination via the available supervised data.

Sentiment neuron

While training the linear model with L1 regularization, we noticed it used surprisingly few of the learned units. Digging in, we realized there actually existed a single “sentiment neuron” that’s highly predictive of the sentiment value.

Just like with similar models, our model can be used to generate text. Unlike those models, we have a direct dial to control the sentiment of the resulting text: we simply overwrite the value of the sentiment neuron.
[...]
Next steps

Our results are a promising step towards general unsupervised representation learning. We found the results by exploring whether we could learn good quality representations as a side effect of language modeling, and scaled up an existing model on a carefully-chosen dataset. Yet the underlying phenomena remain more mysterious than clear.
[...]
Our results suggest that there exist settings where very large next-step-prediction models learn excellent unsupervised representations. Training a large neural network to predict the next frame in a large collection of videos may result in unsupervised representations for object, scene, and action classifiers.
"""
https://blog.openai.com/unsupervised-sentiment-neuron/
https://blog.openai.com/unsupervised-sentiment-neuron/

Labels: Oleg Zabluda

- posted by Oleg Zabluda @ 5:02 PM | permalink | | | technorati | bloglines | feedster

 
Paul Horowitz: "The Search for Extraterrestrial Intelligence" | Talks at Google
Paul Horowitz: "The Search for Extraterrestrial Intelligence" | Talks at Google
"""
Google's office in Cambridge, MA to discuss the SETI (Search for Extraterrestrial Intelligence) project at Harvard University.
"""
https://www.youtube.com/watch?v=sImBlq542TQ
https://www.youtube.com/watch?v=sImBlq542TQ

Labels: Oleg Zabluda

- posted by Oleg Zabluda @ 1:16 PM | permalink | | | technorati | bloglines | feedster