In our previous post we explained how we cheat to achieve optimized performance values. These values show what our Threat Defender is capable of under ideal conditions. But of course, we do not rely on these values to evaluate its performance in daily use. To gain insight into the performance of Threat Defender in real life, we submit it to extensive tests under more realistic conditions.
Keeping It Real
When we create ideal test conditions, we disable as many software features as possible to minimize the number of operations Threat Defender has to perform. We also use optimized test traffic that is tailored to the individual performance indicators we want to measure. However, this easy-to-process test traffic is far from realistic.
In reality, business and enterprise traffic is very complex with varying packet sizes and numerous protocols that require different analyses. The figure shows the varied composition of real-world traffic distributed over a multitude of protocols.
For our performance tests under realistic conditions, we simulate real-world traffic using two traffic mixes: Cisco EMIX 2012 and BreakingPoint (BP) Enterprise Mix. These mixes consist of packets of varying sizes, sent via various protocols to imitate traffic generated by enterprise networks. The results obtained with these mixes differ because they have a different traffic composition. The Cisco EMIX 2012 is older and contains more web protocols than the newer BP Enterprise mix which in turn contains more encrypted traffic. Using these two different traffic mixes allows us to cover a broader range of traffic and to come closer to reality.
We run these traffic mixes through 1000 simulated clients and servers in four networks that generate several 10,000 traffic flows simultaneously. During each test run, we measure all performance indicators.
The following performance measurements were taken on the same reference system and software as in our previous post, i.e. cognitix Threat Defender version 20181206.0 on a single-socket system:
Intel Xeon E5-2690v4; 14 cores / 28 threads
128 GB ECC RAM
Network adapters Intel 82599ES and X710 (two times 2x10Gbit/s, 40Gbit/s connectivity in total)
960 GB Enterprise SSD
To reflect the impact of the enabled software features on the performance, we take the tests using the default feature set and the full feature set.
The default feature set comprises the basic features you need to make good use of Threat Defender: application detection, URL classification, basic intrusion prevention system (IPS), IPFIX reporting, asset tracking, and the policy engine with behavior-based correlation.
The full feature set additionally includes: full IPS and data leakage protection (DLP) analysis, SSL proxy for all connections, and extended behavior-based correlation.
Honest Results – No Cheating
To simulate real-world traffic as it can occur in company networks, we use two different traffic mixes.
The Cisco enterprise traffic mix was published in 2012 to standardize performance measurements and increase their comparability. About one third of the bandwidth of this traffic mix consists of Microsoft Services traffic and approximately one quarter each is consumed by BitTorrent and IMAP traffic. The remaining bandwidth consists of SMTP and FTP traffic.
The BP Enterprise Mix simulates a varied mix of protocols with realistic protocol percentages. About half the bandwidth of the BP Enterprise Mix consists of Microsoft Services traffic; the rest is divided up among FTP, SSH and protocols for VoIP, email, messengers, file sharing, etc.
With these traffic mixes, Threat Defender achieves the following performance values:
Cisco EMIX 2012
BP Enterprise mix
|Default feature set||Full feature set||Default feature set|| |
Full feature set
|6 Gbit/s||5.4 Gbit/s||6.9 Gbit/s||5.85 Gbit/s|
Packets per second
New sessions per second (TCP)
As is to be expected, Threat Defender achieves better performance results with the default feature set because it has to mobilize less processing power than with the full feature set. The only exception is the minimum latency. The reason for this is that almost all processing stages are already active with the default feature set. Therefore, enabling the full feature set has hardly any impact on the measured minimum latency.
When you compare the two traffic mixes, you’ll notice that the BP Enterprise mix achieves significantly higher values regarding the number of processed packets per second and new sessions per second. This difference is due to the composition of the traffic mixes. The BP Enterprise mix uses a greater number of different protocols. For each protocol, new sessions are created and the more sessions you have, the more packets are exchanged. This is why the Cisco EMIX 2012 utilizes the available bandwidth with fewer sessions (and thereby fewer processed packets) than the BP Enterprise mix.
The performance values in the table above are not polished for marketing purposes and give you an impression of the performance you can really expect when you deploy Threat Defender in your network. For comparison, see again the results we obtained under “ideal test conditions” for this system:
Ideal test conditions
Packets per second
New sessions per second (TCP)
You can clearly see the discrepancies between optimized and realistic measurements.
To truly evaluate performance, a neat set of optimized results is completely useless. Instead, you need to look at a range of values that is obtained under realistic conditions and take the enabled feature set into account to get a complete picture and to come to a fact-based decision.
If you’d like to know how the selected hardware impacts performance tests, read our upcoming blog post where we will compare a single-socket and a dual-socket system.