Why effective visibility in the cloud is critical: and how to get there

Many IT and Security leaders are confident they’re sufficiently secure across their entire IT infrastructure, from on-premises to cloud. 

However, the reality is that most organisations lack sufficient visibility into their on-premise network infrastructure, let alone adequate visibility into their cloud-based networks. This lack of visibility - particularly in cloud environments - is creating opportunities for attackers to breach network defences without being detected and giving them ample time to conduct reconnaissance, plan and carry out their attacks.

The stats don’t lie: 45% of breaches are cloud-based (from IBM Cost of a Data Breach Report 2023.) In spite of this, companies continue to move sensitive assets to the cloud: according to the Thales 2023 Cloud Security Study, 75% of respondents report that 40% or more of their data in the cloud is sensitive (up from 49% of respondents in 2021). In the same study, 55% of respondents say it is more complex to manage data in the cloud than it is in on-premises environments. 

Visibility in the cloud is a common issue. Part of the reason why organisations lack visibility into cloud infrastructure is because both NetOps and SecOps teams have often been "late to the party" when shifting workloads to the cloud. Organisations create cloud migration project teams but often neglect to include NetOps and SecOps teams in the initial planning. Cloud architects are also often insufficiently aware of what these teams need to do their jobs effectively.

Once the migration is completed, NetOps and SecOps teams are handed responsibility to monitor and secure the new cloud infrastructure, often without the necessary visibility and monitoring infrastructure required to do this effectively.

Layers of Defence and Visibility are Critical, On-Premise and in Cloud.

Skilled threat actors deploy multiple techniques to remain in stealth mode for as long as possible. Evasion techniques enable attacks to sneak past network and endpoint security devices undetected, threat actors often cover their tracks by disabling logging, deleting evidence on the systems they attack, and sometimes executing vulnerabilities to take down security tools such as EDR, Firewalls, or NDR/IDS tools.  Especially for SecOps teams, multiple layers of defence and visibility are essential to ensure teams are not defending in the dark.

Robust security requires multiple layers of visibility to ensure that we can still defend when the threat actor evades or disables one or more layers of defence. Teams defending on-premise networks typically use a variety of tools and technologies -  including packet capture - to proactively monitor and secure these networks. Packet capture is extremely difficult for an attacker to evade or manipulate, threats always use the network and packet capture is usually invisible to an attacker. Packet evidence has proven invaluable to those defending on-premise networks but how can organisations obtain the same level of visibility in public cloud networks when they don’t “own” the infrastructure. 

Until relatively recently, full packet capture data in the public cloud simply wasn’t an option, leaving teams reliant on flow monitoring and logs as the only cloud network telemetry options. This lack of access to definitive packet-level evidence meant the well-established playbooks and workflows teams have honed to protect their on-premise infrastructure, didn’t translate well into cloud environments. 

Thankfully, times have changed. Customers can now use VPC mirrors, virtual span ports, cloud packet brokers or virtual TAP agents, to enable full packet data in public cloud environments. This means they can now use their proven workflows and tools to protect their cloud network environments too.

Moreover, with a solution that can provide visibility into both on-premise and cloud networks from a single-pane-of-glass, teams gain visibility into network activity across the entire hybrid cloud network. This ubiquitous visibility is a huge advantage. For example, it can enable teams to track attackers that have successfully compromised public cloud infrastructure and then leverage that cloud access to extend their access to on-premise infrastructure. When visibility data is siloed across on-premise and cloud environments this is almost impossible to do - at least impossible to do quickly enough to respond to attacks and shut them down.

However, many organisations are not yet aware that full packet capture in the public cloud is possible. Indeed, many have not yet fully grasped the need for it. One often cited - and probably apocryphal - story has a CIO exclaim confidently “but there are no packets in the cloud!” Needless to say, that’s not true. There are packets in the cloud. And just as with on-premise networks, packet data provides the only truly definitive source of evidence of exactly what happens on cloud networks too. 

What do NetOps and SecOps Teams need for Effective Network Visibility in the Cloud?

Organisations should look to establish the same level of visibility into their public cloud network environments as they have for their on-premise and private cloud environments. Importantly, that would include having always-on packet capture to record as much full packet data as their budget allows. For many organisations this is a mandatory requirement anyway. For example, US Federal agencies are required to collect a minimum of 72 hours of full packet capture data (as per OMB M-21-31) that they can provide to CISA and/or the FBI if required for a breach investigation.

Regardless of regulatory obligations, having full packet data available to NetOps, SecOps and DevOps teams is very valuable. It saves immense amounts of time in the investigation and resolution of network security and performance issues by avoiding the need to correlate multiple telemetry sources. And it gives teams certainty. They no longer need to formulate theories about what took place on the network. The definitive evidence is right there in the packets.

Implementing “always-on” packet capture is essential. Triggered packet capture - which activates capture only in response to a trigger event - is unsafe, because it relies on being able to predict and detect events where packet data is going to be essential for investigations. Packets are most useful when investigating the most serious threats, those that are unknown and not predictable - such as with a Zero Day threat, disabled layers of defence, complex evasion, or a network failure that is out of your control. 

It’s also important not just to record the full packet data, but also to ensure it’s easy for analysts to quickly locate the packets of interest from within what may be petabytes of recorded packet data. Having the ability to quickly search for and find the packet data related to a specific event is key. 

Ideally, organisations should look for a solution that integrates this capability directly into the monitoring tools that NetOps and SecOps teams are already using - so they can go directly from a detected event to the related packets as part of a seamless investigation workflow. Forcing analysts to stop in the middle of an investigation to pull down large capture files and then slice-and-dice them to locate packets of interest is slow and cumbersome. These days, it’s unnecessary too as long as you select a packet capture solution that offers turnkey API integrations with your tools. 

The Additional Benefits of Ubiquitous Hybrid-Cloud Visibility

A recent report from Enterprise Management Associates (EMA) - NetSecOps: Examining How Network and Security Teams Collaborate for a Better Digital Future - outlines the significant benefits that accrue from encouraging Network and Security teams to collaborate more closely - particularly as organisations migrate workloads to the cloud.

EMA's research highlights organisations which share visibility tools across teams see increased collaboration between security and network teams. This helps break down traditional silos and in many cases has resulted in the formation of hybrid NetSecOps teams. Organisations that encourage this closer collaboration reap significant benefits including:

● Reducing security risks (43.1%)

● Improving operational efficiency (39.8%)

● Faster resolution of network/application performance problems (39.5%)

● Faster detection and resolution of security issues (39.1%)

● Cost savings (38.5%) 

● Network resilience (34.5%) 

● Reduced architectural complexity (24.3%)

Collaboration between teams is also facilitating a two-way exchange of knowledge. Network teams can share their expertise in packet analysis, while security teams can provide insights into cyber threats and attack patterns. This gives team members the opportunity to broaden their skills and increases both their capability and value - further helping organisations to battle the skill shortages that the industry faces. 

There are clear advantages to fostering closer collaboration between NetOps and SecOps teams –  to promote the use of shared tools, workflows, and processes. As cloud adoption continues to accelerate, the ability to capture and analyse packet data across hybrid environments is now increasingly crucial. 

Organisations that plan for this need and architect their networks to ensure they can meet it are significantly better prepared to defend their organisations from costly outages, performance issues and cyberattack than those that realise the need too late.