Bringing compute to the storage layer

Amazon Web Services has introduced AQUA for Amazon Redshift, an innovative new distributed and hardware-accelerated cache that delivers up to ten times better query performance than other enterprise cloud data warehouses. AQUA brings compute to the storage layer, helping customers avoid networking bandwidth limitations by eliminating unnecessary data movement between where data is stored and compute clusters. With AQUA, customers have more up-to-date dashboards, save development time, and their systems are easier to maintain. AQUA is available on Redshift RA3 instances at no additional cost, and customers can take advantage of the AQUA performance improvements without any code changes.

Since its launch in 2012 as the first data warehouse built for the cloud at a cost of 1/10th that of traditional data warehouses, Amazon Redshift has become the most popular cloud data warehouse. Last year, AWS announced the general availability of Amazon Redshift RA3 instances, which allow customers to scale compute and storage separately and deliver up to 3x better price performance than any other cloud data warehouse. However, even as the performance of data warehouses continues to increase, the rapid growth of data that customers need to process in their data warehouses has led to a difficult balancing act between performance and cost-effective scaling. The prevailing approach to data warehousing has been to build out an architecture whereby large amounts of centralised storage data are moved to waiting compute nodes for processing. The challenge with this approach is that there is a lot of data movement between the shared storage and the compute nodes. As data volumes continue to grow at a rapid clip, this data movement saturates available networking bandwidth and slows down performance. In addition to the networking bottleneck, CPUs are not able to keep up with the faster growth in storage capabilities (SSD storage throughput has grown 6x faster than the ability of CPUs to process data from memory), which either creates a new CPU bottleneck of its own or forces more customers to over-provision compute to get their work done more quickly.

AQUA for Amazon Redshift is a distributed and hardware-accelerated cache for Amazon Redshift; an innovation that improves performance for analytics at the new scale of data. AQUA brings compute to the storage layer, so data does not have to move back and forth between the two. This enables Amazon Redshift to run queries up to ten times faster than other enterprise cloud data warehouses. The AQUA cache scales out and processes data in parallel across many nodes. Each node possesses a hardware module composed of AWS-designed analytics processors that dramatically accelerate data compression, encryption, and data processing tasks like scans, aggregates, and filtering. AQUA also gives customers the added benefit of being able to do compute on their raw storage, which saves time that would otherwise be spent moving data around. With this new architecture, and the order-of-magnitude better performance it brings, Redshift customers have more up-to-date dashboards, save development time, and their systems are easier to maintain.

“Existing data warehouse architectures with centralized storage require that data be moved to compute clusters for processing, which creates a bottleneck and slows down performance,” said Rahul Pathak, VP, Analytics, AWS. “By bringing compute to the storage layer, AQUA helps customers eliminate unnecessary data movement to avoid these networking bandwidth limitations, delivering up to an order-of-magnitude query performance improvement over other enterprise cloud data warehouses — and this game-changing leap in performance is available to Amazon Redshift users at no additional cost.” 

AQUA for Amazon Redshift is generally available today to customers running Amazon Redshift RA3 instances in US East (N. Virginia), US West (Oregon), US East (Ohio), Asia Pacific (Tokyo), and Europe (Ireland), with availability in additional regions coming soon.

FOX Corporation produces and distributes content through some of the world’s leading and most valued brands, including: FOX News, FOX Sports, the FOX Network, and the FOX Television Stations. FOX empowers a diverse range of creators to imagine and develop culturally significant content, while building an organisation that thrives on creative ideas, operational expertise, and strategic thinking. “FOX Corporation’s mission is to give millions of viewers the simple pleasure of being transported by a story on a screen. We have global audiences consuming premiere content across News, Sports, and Entertainment, and data is at the center of everything we do,” said Alex Tverdohleb, VP, Data Services, FOX Corporation. “Amazon Redshift empowers us to analyse petabytes of structured and semi-structured data across our data warehouse, operational database, and Amazon S3 data lake to discover, analyse, and activate data-driven decisions and powerful insights. As our petabyte-scale data continues to grow rapidly, we have been testing AQUA for Redshift to get better performance for our analytics queries while keeping our costs flat. We are seeing AQUA for Amazon Redshift improve the performance of some of our analytics queries by an order of magnitude and it is an example of how we are using latest technology to deliver a more personalised, curated, and timely experience to our viewers.”