Oleg Zabluda's blog
Thursday, April 27, 2017
 
Training recurrent neural networks (2013) Ilya Sutskever: Ph.D. thesis
Training recurrent neural networks (2013) Ilya Sutskever: Ph.D. thesis
"""
2.5.2 Recurrent Neural Networks as Generative models

Generative models are parameterized families of probability distributions that extrapolate a finite training set to a distribution over the entire space. [...] An RNN defines a generative model over sequences if the loss function satisfies ...
[...]
5.3 The Objective Function

The goal of character-level language modeling is to predict the next c*5.3 The Objective Function*

The language modeling objective is to maximize the total log probability of the training sequence [...] which implies that the RNN learns a probability distribution over sequences

5.4.4 Debagging

It is easy to convert a sentence into a bag of words, but it is much harder to convert a bag of words into a meaningful sentence. We name the latter the debagging problem. We perform an experiment where a character-level language model evaluates every possible ordering of the words in the bag, and returns and the ordering it deems best. To make the experiment tractable, we only considered bags of 7 words, giving a search space of size 5040. For our experiment, we used the MRNN [...] to debag 500 bags of randomly chosen words from “Ana Karenina”. We use 11 words for each bag, where the first two and the last two words are used as context to aid debagging the middle seven words. We say that the model correctly debags a sentence if the correct ordering is assigned the highest log probability. We found that the wikipedia trained MRNN recovered the correct ordering 34% of the time,
"""
http://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf
http://www.cs.utoronto.ca/~ilya/pubs/ilya_sutskever_phd_thesis.pdf

Labels:


- posted by Oleg Zabluda @ 4:17 PM | permalink | | | technorati | bloglines | feedster
| |

Home

Powered by Blogger