Security is a Big Data Problem
By Jeff Cotrupe, Industry Director – Big Data & Analytics, Stratecast, Frost & Sullivan
So much has been written about, and, frankly, overhyped about big data in recent years that the most important considerations with regard to big data might be what to believe and where to start. Corporate IT and data science teams deploying data management solutions are expressing three primary concerns with regard to big data:
- Ensuring that they are accessing all relevant data from all sources to meet the needs of the organization—driven by the fear that they are not.
- Managing the “fire hose” of data that results from comprehensive data access. Rather than drowning in data, companies want to be able to focus on the data that is most important.
- Empowering users with focused, actionable information by making it simple for business users to query the data to quickly obtain the information they need.
If an organization can step up to these challenges, it can achieve what data management is supposed to be all about: achieving a high degree of accuracy in assessing current conditions, studying past events and outcomes to spot relevant trends, and using insights from current and past occurrences to predict future behaviors. Security is the number one spending priority among IS and IT professionals. Security is also a big data problem. Threats to an organization can come from all external touch points and access points as well as from inside the organization. Maintaining the security of the organization and its data depends on accessing all relevant data about systems, processes, and transactions that could contain or provide an environment for security threats. Processing and querying the data for solutions that either prevent threats from impacting the organization or counteract threats once detected is also essential. Organizations are valiantly attempting to protect against security threats, but they are at a severe disadvantage. The solutions they are deploying—driven by flawed technology and more seriously, flawed logic—cannot hope to keep pace with the vast and growing pool of potential threats. Conventional approaches to security are ineffective at guarding against threats, and the complexity they impose, both technologically and organizationally, render them unsustainable.
Big Data-driven Machine Learning
Machine learning can help IT security experts make sense of the vast amount of threat intelligence their systems generate. Machine learning introduces concepts, processes, and technologies that represent a simplified model of intelligence that ingests data and solves problems. The most successful algorithm for machine learning is based on a process known as reverse propagation, whereby the system:
- Uses grid computing to process data
- Obtains an output from the end of the process
- Propagates a signal back through the data processing layers to modify all parameters
The system has now “learned” about a condition such that the next data inputs that come into the system can receive an enhanced level of treatment that did not exist before the learning occurred. This process, continuously repeated in response to data inputs, is machine learning in action. By applying Big Data and machine learning to security, an organization can:
- Build a security strategy based not on collecting static threat signatures in a library but, instead, on using big data and analytics to learn what normal behavior in the network looks like; to analyze and correlate anomalies in that behavior; and to take corrective action that resolves those anomalies, eliminating security threats
- Combat rapidly evolving threats with a single, unified system that continually learns from all data inputs and therefore is evolving rapidly itself to meet the challenge
- Leverage data feeds and data correlations to identify persistent threats
Using Big Data to equip an organization’s security system with continuous learning, instead of a collection of known threat signatures, illustrates one of the core benefits of any big data implementation: letting the system help discover insights and answers one did not anticipate.