Storage I/O performance is the number one issue in HPC Systems for Big Data

The survey polled a cross-section of 60 DDN HPC end users representing hundreds of petabytes of storage investment, which included individuals responsible for HPC allocations encompassing high availability storage and file systems across a variety of industries. Of those surveyed, 60 percent manage or use at least one petabyte of storage, and over 20 percent manage or use 10 PBs or more.

According to the survey results, more than two-thirds (68%) of those polled agree that data and data storage has become the most strategic part of the HPC datacenter. Moreover, by a margin of two to one respondents, the survey also reveals that today’s storage technologies will need to undergo massive change to hit exascale proportions.

In addressing this challenge, the overwhelming majority of respondents (78%) agree that hybrid storage is the evolutionary next step for HPC storage, combining both the performance and cost efficiency benefits required of storage at exascale.

Future Advances in Storage Performance to Deliver Massive I/O Gains and Reduction in Storage Costs for HPC, Analytics and Cloud Computing Environments
· With a nod to future technology requirements, the survey also shines light on the biggest storage and/or Big Data challenges for HPC users today, with nearly half of respondents citing mixed I/O performance the primary issue (47%), followed by managing rapid data growth (22%), performance (13%) and multi-site replication or collaboration (5%). Thirteen percent of those polled name ‘other’ concerns, including the need to manage the overall total cost of storage without compromising storage scalability and performance.
· With storage performance a critical requirement especially at today’s petascale levels, of those respondents with an opinion, 85 percent agree that site-wide file systems like those at at Oakridge National Laboratory (ORNL), National Energy Research Scientific Computing Center (NERSC) and Texas Advanced Computing Center (TACC) are the next infrastructure trend in HPC environments.
· With HPC no longer limited to data centers at research institutions of the Top500 Supercomputing sites, the survey also examines end users opinions on analytics and integrating computing workloads into large scale HPC cloud environments. More than two-thirds (70%) of those polled point to storage I/O as the biggest bottleneck in analytics workflows, suggesting that compute is no longer the main challenge for analytics platforms. While multi-site HPC collaboration may be on the rise, security and data sharing complexity remain the biggest impediments according to 75 percent of those surveyed.
· Finally, HPC organizations continue to find themselves under increasing funding scrutiny and according to nearly three quarters of the survey respondents, it has become more difficult for HPC sites to obtain funding for compute infrastructure without demonstrating a clear return on investment (ROI).