Oleg Zabluda's blog
Saturday, April 08, 2017
 
"""
"""
Now is the time to submit paper on Fully Convolutional Encoder Decoder BatchNorm ResNet GAN applied to Style Transfer, optimized with Adam. [...] by Adam Bengio
"""
https://medium.com/@karpathy/a-peek-at-trends-in-machine-learning-ab8a1085a106

Labels:


- posted by Oleg Zabluda @ 10:12 PM | permalink | | | technorati | bloglines | feedster
 
Why Momentum Really Works
Why Momentum Really Works
"""
Momentum proposes the following tweak to gradient descent. [...] The new algorithm may seem at first glance like a cheap hack. A simple trick to get around gradient descent's more aberrant behavior -- a smoother for oscillations between steep canyons. But the truth, if anything, is the other way round. It is gradient descent which is the hack. First, momentum gives up to a quadratic speedup on many functions. 1 This is no small matter -- this is similar to the speedup you get from the Fast Fourier Transform, Quicksort, and Grover's Algorithm. When the universe gives you quadratic speedups, you should start to pay attention.
[...]
With barely a modicum of extra effort, we have essentially square rooted the condition number! These gains, in principle, require explicit knowledge of λ​1 and λn. But the formulas reveal a simple guideline. When the problem's conditioning is poor, the optimal α is approximately twice that of gradient descent, and the momentum term is close to 1. So set β as close to 1 as you can, and then find the highest α which still converges. Being at the knife's edge of divergence, like in gradient descent, is a good place to be.
"""
http://distill.pub/2017/momentum/
http://distill.pub/2017/momentum/

Labels:


- posted by Oleg Zabluda @ 1:32 PM | permalink | | | technorati | bloglines | feedster
 
Systolic Array
Systolic Array
"""
A systolic array is composed of matrix-like rows of data processing units called cells. Data processing units (DPUs) are similar to central processing units (CPUs), (except for the usual lack of a program counter, since operation is transport-triggered, i.e., by the arrival of a data object). Each cell shares the information with its neighbors immediately after processing. The systolic array is often rectangular where data flows across the array between neighbour DPUs, often with different data flowing in different directions. The data streams entering and leaving the ports of the array are generated by auto-sequencing memory units, ASMs. Each ASM includes a data counter.
[...]
An example of a systolic algorithm might be designed for matrix multiplication. One matrix is fed in a row at a time from the top of the array and is passed down the array, the other matrix is fed in a column at a time from the left hand side of the array and passes from left to right. Dummy values are then passed in until each processor has seen one whole row and one whole column. At this point, the result of the multiplication is stored in the array and can now be output a row or a column at a time, flowing down or across the array.[3]
"""
https://en.wikipedia.org/wiki/Systolic_array#Detailed_description
https://en.wikipedia.org/wiki/Systolic_array#Detailed_description

Labels:


- posted by Oleg Zabluda @ 11:24 AM | permalink | | | technorati | bloglines | feedster

Powered by Blogger