2015 in Focus: Data Deluge, Privacy and Hybrid Cloud

“Making predictions is difficult, especially about the future”. So said Nils Bohr, among others; but whoever first coined this sentiment, it has never been truer than it is in the technology world today. Technology moves so fast that predicting its short-term evolution has always been fraught with danger. All that said, several clear-cut trends are developing as we approach 2015 that we think deserve close attention from the industry at large; an impending shortage of data storage, the emergence of better online privacy policies, smarter infrastructure for big data applications and a shift towards hybrid cloud models.

Storage: a dwindling resource

It might seem hard to believe that we’re running out of disk space. But that’s the harsh reality in a world where data is no longer just text files and spreadsheets but countless high-definition videos and high-resolution images. In 2013, the world generated around 3.5 zettabytes of data, the equivalent of 120 billion 32-gigabyte smartphones, or 600 billion DVDs (or, put another way, more than 3 billion years of video). By 2020, we will be creating more than ten times that amount of data. Unfortunately for us, we’re going to run into trouble long before then; by 2016, estimates show that we will be producing data faster than we can store it.

One problem facing the storage industry is that the perpendicular magnetic recording (PMR) technology underlying today’s hard drives is showing its age. While we can already squeeze enormous amounts of information into tiny areas of disk space with PMR, there’s no way to increase the density of that data. The good news is that several innovative technologies are in development to help solve this problem. Seagate is investing in heat-assisted magnetic recording (HAMR), for example, which is expected to increase the limits of magnetic recording by a factor of 100 - with an almost incredible theoretical limit 50 terabits per square inch.

Commercial HAMR drives are still some way off and are unlikely to appear before 2016, according to most industry experts. In 2015, then, the focus will be on interim measures designed to improve efficiency of data storage. Smarter data policies in the workplace, better de-duplication methodologies and more enlightened backup strategies will all have their part to play while the world waits for the next generation of disk drives.

Big data: tiers before downtime

For consumers, the predicted data capacity gap is likely to be a nuisance rather than a major catastrophe; less space for their TV shows and photos. But for organisations increasingly dependent on so-called big data to understand their business and their customers, it represents a much greater threat. Today, big data typically resides on traditional disk storage arrays built around standard hardware and software components, often working in a very inefficient and poorly synchronised manner. But as data volumes grow and processing demands increase with them, CIOs will have to think much more strategically about how to manage data in the longer term.

One approach that will gather steam while IT departments wait for new drive technology is a much more efficient tiered model based on existing technology. By intelligently layering conventional hard disk drives, solid state hybrid drives (SSHD) and solid state drives (SSD), CIOs will be able to organise data much more effectively; allowing quick and easy access to the most critical data from solid state devices while ensuring that the less valuable metadata is still available and secure on slower HDD drives, for example. We’ll see more data centres built around this approach in 2015, as everyone from IT directors to data analysts and marketing departments struggle to capture, store and make sense of the flood of data from the explosion of devices on the world’s networks.

Progress in privacy

One inevitable consequence of the data explosion is a growth in consumer concerns about the use to which their data is being put. Until very recently, consumers had little or no control over what data was kept by the organisations whose products and services they used. In 2014, though, Europe’s decades-old data and privacy regulations were significantly overhauled by the European Court of Justice and the European Parliament. Consumers and brands now have a so-called ‘right to be forgotten’ by search engines, for example, a controversial development which has divided opinion across the digital landscape.

Such rulings have so far done little to clarify the complexities of consumer data privacy, but there are other signs that the business world is starting to take the privacy of their customers more seriously. Some enlightened brands are making substantial efforts to explain to their customers exactly what data they hold and how they are using it to their benefit - often strengthening their customer relationships and improving long-term engagement in the process. As the role of data as a marketing tool grows in 2015, expect to see many more organisations become increasingly transparent and open about their data assets - and at least as importantly, about their data intentions.

Cloud: hybrid happenings

Cloud computing may appear to be mainstream today, but we are still in the very early stages of its evolution into a fully-fledged business platform. Many organisations are now discovering that their early private cloud implementations are already reaching their capacity limits, and are being forced to extend them into public resources (so-called cloud-bursting). Others are finding that it makes little sense to devote costly, resource-intensive private clouds to line-of-business applications that can be run more efficiently from public cloud platforms, and that only mission-critical applications make sense in private cloud environments.

These so-called hybrid approaches require a leap of faith from businesses of any size, but as cloud technology and security evolves so too does the level of trust that businesses are prepared to place in them. 2015 will see hybrid cloud implementations and business models emerging more strongly, driven by improvements in cloud storage and bandwidth as well as more robust security measures.