Using Historical Data to Keep Your Network Safe

Using Historical Data to Keep Your Network Safe

Third of a four-part series

After a burglary at your home or office, you’d probably want to strengthen your security appropriately. Thieves came in a window? Equip the windows with alarms. Bad guys pried open the front door? Add more robust locks.

But what if you couldn’t determine how the burglars got in? That’s essentially the problem with many of today’s network-security systems. They use standard learning models, meaning the model is trained not with actual historic data, but instead with collective sample data that’s later distributed to all systems. From the vendor’s point of view, it’s an approach that’s relatively easy. But for the user, security decisions can become difficult. With only collective samples, network and security managers are stuck using sparse data from equally sparse sources.

Actual data matters. That’s why reducing the richness of the raw data also reduces the richness of the detection. Starting with too little data results in both numerous false positives and false negatives. Also, data profiles or culled data are forms of data reduction, meaning they’re useless for both other security learning models and entirely new models.

That’s why Masergy’s Unified Enterprise Security (UES) platform maintains packet headers for at least 14 days. This helps the system maintain a large enough set of historical data to be used for effective security. And for the best possible network protection, UES uses every field in the packet headers, not just a select few.

Masergy UES captures and retains copious amounts of data, allowing it to spot important correlations among seemingly unlikely data sets. The very improbability of correlation among these data sets is precisely what makes them so rich an anomaly-detection environment. And the easiest way to have a large historical data set is to maintain the raw data used in past analysis runs.

More is Better

Masergy’s UES also includes a data-prediction gradient that matches data with learning models to produce stable, predictable patterns. For Masergy’s analysis engine, unlike competing systems, more data is actually better than less.

With more conventional approaches, anomaly detection is often challenged by a small data set that never allows predictability. Conversely, a large data set can overwhelm the system and fail to produce results fast enough; or it can present so much variation, the system fails to detect it. In addition, noisy data sources, bad clusters or transient data can automatically fall to the bottom of the gradient, meaning the model will not use them for detection.

One key to creating a data gradient is regression testing. If the outcome of a prediction of a model suddenly fails when it has rarely done so before, this either means it’s a true anomaly or that the model is no longer valid for the data. To make this determination, Masergy once again uses historical data, this time to perform regression analysis against the current model. Because anomaly-detection suffers from sparse data, UES maintains an abundance of historic data for the local learning models.

In essence, Masergy UES remembers both how the bad guys got in last time and what “normal” looks like. Then it uses these memories—and this data—to keep your networks safe.

Learn about Masergy’s Unified Enterprise Security and big data analytics.

About Mike Stute

Chief Scientist, Masergy
Mike Stute is Chief Scientist at Masergy Communications and is the chief architect of the Unified Enterprise Security network behavioral analysis system. As a data scientist, he is responsible for the research and development of deep analysis methods using machine learning, probability engines, and complex system analysis in big data environments. Mike has over 22 years experience in information systems security and has developed analysis systems in fields such as power generation, educational institutions, biotechnology, and electronic communication networks.