Oleg Zabluda's blog
Monday, October 15, 2012
 
Some experiments into what can be achieved with flat L2 networks, when 10G network is faster then small number of...
Some experiments into what can be achieved with flat L2 networks, when 10G network is faster then small number of local drives.

Currently standard way to scale datacenter-scale networks is with fat trees (edge-aggregation-core). Their limit is achieved due to the cost and technological availability of core switches. A cheaper alternative is a "flat" Clos network made out of commodity switches [1]. Its biggest advantage is cost and same-speed links throughout at the cost of increased latency.

But single-hop latency, currently is on the order of tens of microseconds, while hard-drive latency is on the order of tens of milliseconds. By picking large enough block size (8 MB in the paper), this can be mitigated, as well as the difference between rack-local hops and global hops. 

Then the win comes from the fact that network bandwidth is on the order of 1 GB/s, while spinning hard drive bandwidth is ~100 MB/s. Locality is still very important for flash (~1GB/s, 25 μs). and RAM (~100 GB/s, 25 ns).

"Flat Datacenter Storage", Microsoft Research, 2012 by
Edmund B. Nightingale, Jeremy Elson, Jinliang Fan, Owen Hofmann, Jon Howell, and Yutaka Suzue
http://research.microsoft.com/pubs/170248/fds-final.pdf

"Towards a next generation data center architecture: scalability
and commoditization", Microsoft Research (2008) by A. Greenberg, P. Lahiri, D. A. Maltz, P. Patel, and S. Sengupta.
http://research.microsoft.com/pubs/79348/presto27-greenberg.pdf

"A Scalable, Commodity Data Center Network Architecture",UC San Diego , 2008 by Mohammad Al-Fares, Alexander Loukissas and Amin Vahdat
http://ccr.sigcomm.org/online/files/p63-alfares.pdf

[1] The UCSD paper infuriatingly keeps referring to their Clos topology as "fat tree", although it contradicts established terminology, is not even a tree, and not fat.

http://en.wikipedia.org/wiki/Fat_tree
http://en.wikipedia.org/wiki/Clos_network

"Sun Datacenter Switch 3456 Architecture White Paper" (has diagrams for T=5  and formulas for arbitrary number of tiers T.
http://192.9.172.90/products/networking/datacenter/ds3456/ds3456_wp.pdf

"PortLand: A Scalable Fault-Tolerant Layer 2 Data Center Network Fabric" 
http://cseweb.ucsd.edu/~vahdat/papers/portland-sigcomm09.pdf
""" 
a three-stage fat tree built from k-port switches can support non-blocking communication among k^3/4 end hosts using 5k^2/4 individual k-port switches 
""" 

"Scale-Out Networking in the Data Center" 
http://cseweb.ucsd.edu/~ssradhak/Papers/scaleout-micro10.pdf
http://research.microsoft.com/pubs/79348/presto27-greenberg.pdf

Labels:


- posted by Oleg Zabluda @ 11:28 AM | permalink | | | technorati | bloglines | feedster
| |

Home

Powered by Blogger