Oleg Zabluda's blog
Friday, September 01, 2017
 
Transformer: A Novel Neural Network Architecture for Language Understanding
Transformer: A Novel Neural Network Architecture for Language Understanding
"""
deciding on the most likely meaning and appropriate representation of the word “bank” in the sentence “I arrived at the bank after crossing the…” requires knowing if the sentence ends in “... road.” or “... river.”

RNNs have in recent years become the typical network architecture for translation, processing language sequentially in a left-to-right or right-to-left fashion.
[...]
In contrast, the Transformer only performs a small, constant number of steps (chosen empirically). In each step, it applies a self-attention mechanism which directly models relationships between all words in a sentence, regardless of their respective position. [...] Transformer can learn to immediately attend to the word “river” and make this decision in a single step.
"""
https://research.googleblog.com/2017/08/transformer-novel-neural-network.html

Labels:


- posted by Oleg Zabluda @ 2:59 PM | permalink | | | technorati | bloglines | feedster
| |

Home

Powered by Blogger