Oleg Zabluda's blog
Wednesday, May 16, 2018
 
"""
"""
since 2012, the amount of compute used in the largest AI training runs has been increasing exponentially with a 3.5 month-doubling time [...] has grown by more than 300,000x [...] The trend represents an increase by roughly a factor of 10 each year.
[...]
Eras

Looking at the graph we can roughly see four distinct eras:

Before 2012: It was uncommon to use GPUs for ML, making any of the results in the graph difficult to achieve.

2012 to 2014: Infrastructure to train on many GPUs was uncommon, so most results used 1-8 GPUs rated at 1-2 TFLOPS for a total of 0.001-0.1 pfs-days.

2014 to 2016: Large-scale results used 10-100 GPUs rated at 5-10 TFLOPS, resulting in 0.1-10 pfs-days. Diminishing returns on data parallelism meant that larger training runs had limited value.

2016 to 2017: Approaches that allow greater algorithmic parallelism such as huge batch sizes, architecture search, and expert iteration, along with specialized hardware such as TPU’s and faster interconnects, have greatly increased these limits, at least for some applications. AlphaGoZero/AlphaZero is the most visible public example of massive algorithmic parallelism,
[...]
The world’s total hardware budget is 1 trillion dollars a year (https://www.statista.com/statistics/422802/hardware-spending-forecast-worldwide/)
[...]
Appendix: Recent novel results that used modest amounts of compute
[...]
Here are some examples of [recent noteworthy] results using modest compute
[...]
Attention is all you need: 0.089 pfs-days (6/2017)
Adam Optimizer: less than 0.0007 pfs-days (12/2014)
Learning to Align and Translate: 0.018 pfs-days (09/2014)
GANs: less than 0.006 pfs-days (6/2014)
Word2Vec: less than 0.00045 pfs-days (10/2013)
Variational Auto Encoders: less than 0.0000055 pfs-days (12/2013)
"""
https://blog.openai.com/ai-and-compute/
https://blog.openai.com/ai-and-compute/

Labels:


- posted by Oleg Zabluda @ 5:12 PM | permalink | | | technorati | bloglines | feedster
| |

Home

Powered by Blogger