99% matrix sparsity is generally not enough for things to be faster on GPU than simply doing dense multiplication.
99% matrix sparsity is generally not enough for things to be faster on GPU than simply doing dense multiplication. Easy-to-understand benchmark data is hard to get. For example, typically, NVIDIA will say that their cuSparse is 2x-5x faster than Intel’s MKL, but nowhere would you find how many FLOPS they get:
https://developer.nvidia.com/cusparse

Here is the secret knowledge for you. On slide 29, AMD brags how clSparse is faster than cuSparse
http://www.iwocl.org/wp-content/uploads/iwocl-2016-clsparse-vendor-blas-library.pdf

getting a whopping 40 GFLOP/s (geometric mean across benchmarks) on a GPU (FURY-X) with 8600 GFLOP/s (40/8600=4.5% utilization) vs 10 GFLOP/s on Titan-X Maxwell with 6144 GFLOP/s (10/6144=1.5% utilization)
https://developer.nvidia.com/cusparse

Labels: Oleg Zabluda

- posted by Oleg Zabluda @ 8:26 PM | permalink | | | technorati | bloglines | feedster

 
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation (2014) Emily Denton, Wojciech...
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation (2014) Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, Rob Fergus
"""
3.2.1 Matrix Decomposition

Matrices are 2-tensors which can be linearly compressed using the Singular Value Decomposition. If W ∈ R^m×k is a real matrix, the SVD is defined as W = USV, where U ∈ R^m×m, S ∈ R^m×k, V ∈ R^k×k. S is a diagonal matrix with the singular values on the diagonal, and U, V are orthogonal matrices. If the singular values of W decay rapidly, W can be well approximated by keeping only the t largest entries of S, resulting in the approximation W˜ = U˜S˜V˜, where U˜ ∈ R^m×t, S˜ ∈ R^t×t, V˜ ∈ R^t×k. Then, for I ∈ R^n×m, the approximation error || IW˜ − IW|| ≤ s_{t+1}|| I || , and thus is controlled by the decay along the diagonal of S.Now the computation IW˜ can be done in O(nmt + nt2 + ntk), which, for sufficiently small t is significantly smaller than O(nmk).
"""
https://arxiv.org/abs/1404.0736
https://arxiv.org/abs/1404.0736

Labels: Oleg Zabluda

- posted by Oleg Zabluda @ 7:41 PM | permalink | | | technorati | bloglines | feedster

 
Obama Administration Rushed to Preserve Intelligence of Russian Election Hacking
Obama Administration Rushed to Preserve Intelligence of Russian Election Hacking
"""
In the Obama administration’s last days, some White House officials scrambled to spread information about Russian efforts to undermine the presidential election — and about possible contacts between associates of President-elect Donald J. Trump and Russians — across the government. [...] At the Obama White House, Mr. Trump’s statements stoked fears among some that intelligence could be covered up or destroyed — or its sources exposed — once power changed hands. What followed was a push to preserve the intelligence [...] It also reflected the suspicion among many in the Obama White House that the Trump campaign might have colluded with Russia on election email hacks — a suspicion that American officials say has not been confirmed. Former senior Obama administration officials said that none of the efforts were directed by Mr. Obama.

Sean Spicer, the Trump White House spokesman, said, “The only new piece of information that has come to light is that political appointees in the Obama administration have sought to create a false narrative to make an excuse for their own defeat in the election.”
[...]
At intelligence agencies, there was a push to process as much raw intelligence as possible into analyses, and to keep the reports at a relatively low classification level to ensure as wide a readership as possible across the government — and, in some cases, among European allies. This allowed the upload of as much intelligence as possible to Intellipedia, a secret wiki used by American analysts to share information.
[...]
The opposite happened with the most sensitive intelligence, including the names of sources and the identities of foreigners who were regularly monitored. Officials tightened the already small number of people who could access that information. They knew the information could not be kept from the new president or his top advisers, but wanted to narrow the number of people who might see the information
[...]
More than a half-dozen current and former officials described various aspects of the effort to preserve and distribute the intelligence, and some said they were speaking to draw attention to the material and ensure proper investigation by Congress. All spoke on the condition of anonymity because they were discussing classified information
[...]
Beyond leaving a trail for investigators, the Obama administration also wanted to help European allies combat a threat that had caught the United States off guard. American intelligence agencies made it clear in the declassified version of the intelligence assessment released in January that they believed Russia intended to use its attacks on the United States as a template for more meddling.
"""
https://www.nytimes.com/2017/03/01/us/politics/obama-trump-russia-election-hacking.html
https://www.nytimes.com/2017/03/01/us/politics/obama-trump-russia-election-hacking.html

Labels: Oleg Zabluda

- posted by Oleg Zabluda @ 11:00 AM | permalink | | | technorati | bloglines | feedster