2012-044 – The Data Supercell: A Reliable, High-Performance, Scalable, Federated Disk-based Storage System

The Data Supercell enables long-term, reliable, high-performance, low cost, robust data storage. Design goals include high-bandwidth, low-latency access to very large data, cost effectiveness, and reliability on par with magnetic tape. The Data Supercell differs from present technology in applying robust software techniques for maintaining data integrity to cost-effective, commodity disk drives, allowing reliable, long-term access to petascale datasets at unprecedented rates. Distributed storage, is enabled by federating multiple Data Supercells under PSC’s SLASH2 filesystem. It also supports integration with multiple other storage systems of diverse architectures. The problems solved by the Data Supercell are the insufficient bandwidth and excessive latency for tape-based archives and sporadic network performance when dealing with distributed data storage. Tape bandwidth is very expensive to increase and tape latency cannot be reduced because of the time it takes to spool through each tape. Because of these problems, tape storage is often viewed as “write once, read never”. The Data Supercell solves these problems by scaling bandwidth with the storage capacity while retaining very low latency. Reed McManigle reedm@andrew.cmu.edu 412-268-5443