In our previous post we explained how we cheat to achieve optimized performance values. These values show what our Threat Defender is capable of under ideal conditions. But of course, we do not rely on these values to evaluate its performance in daily use. To gain insight into the performance of Threat Defender in real life, we submit it to extensive tests under more realistic conditions.

Keeping It Real

When we create ideal test conditions, we disable as many software features as possible to minimize the number of operations Threat Defender has to perform. We also use optimized test traffic that is tailored to the individual performance indicators we want to measure. However, this easy-to-process test traffic is far from realistic.

In reality, business and enterprise traffic is very complex with varying packet sizes and numerous protocols that require different analyses. The figure shows the varied composition of real-world traffic distributed over a multitude of protocols.

For our performance tests under realistic conditions, we simulate real-world traffic using two traffic mixes: Cisco EMIX 2012 and BreakingPoint (BP) Enterprise Mix. These mixes consist of packets of varying sizes, sent via various protocols to imitate traffic generated by enterprise networks. The results obtained with these mixes differ because they have a different traffic composition. The Cisco EMIX 2012 is older and contains more web protocols than the newer BP Enterprise mix which in turn contains more encrypted traffic. Using these two different traffic mixes allows us to cover a broader range of traffic and to come closer to reality.

We run these traffic mixes through 1000 simulated clients and servers in four networks that generate several 10,000 traffic flows simultaneously. During each test run, we measure all performance indicators.

The following performance measurements were taken on the same reference system and software as in our previous post, i.e. cognitix Threat Defender version 20181206.0 on a single-socket system:

  • Intel Xeon E5-2690v4; 14 cores / 28 threads

  • 128 GB ECC RAM

  • Network adapters Intel 82599ES and X710 (two times 2x10Gbit/s, 40Gbit/s connectivity in total)

  • 960 GB Enterprise SSD

To reflect the impact of the enabled software features on the performance, we take the tests using the default feature set and the full feature set.

The default feature set comprises the basic features you need to make good use of Threat Defender: application detection, URL classification, basic intrusion prevention system (IPS), IPFIX reporting, asset tracking, and the policy engine with behavior-based correlation.

The full feature set additionally includes: full IPS and data leakage protection (DLP) analysis, SSL proxy for all connections, and extended behavior-based correlation.

Honest Results – No Cheating

To simulate real-world traffic as it can occur in company networks, we use two different traffic mixes.

The Cisco enterprise traffic mix was published in 2012 to standardize performance measurements and increase their comparability. About one third of the bandwidth of this traffic mix consists of Microsoft Services traffic and approximately one quarter each is consumed by BitTorrent and IMAP traffic. The remaining bandwidth consists of SMTP and FTP traffic.

The BP Enterprise Mix simulates a varied mix of protocols with realistic protocol percentages. About half the bandwidth of the BP Enterprise Mix consists of Microsoft Services traffic; the rest is divided up among FTP, SSH and protocols for VoIP, email, messengers, file sharing, etc.

With these traffic mixes, Threat Defender achieves the following performance values:

 

Cisco EMIX 2012

       BP Enterprise mix

Default feature setFull feature setDefault feature set 
Full feature set
 

Throughput

6 Gbit/s5.4 Gbit/s6.9 Gbit/s5.85 Gbit/s

Packets per second

760,000

665,000

1,820,000

1,600,000

Minimum latency

7.125 μs

7.625 μs

8.5 μs

8.5 μs

New sessions per second (TCP)

12,800

11,000

71,500

60,000

 

As is to be expected, Threat Defender achieves better performance results with the default feature set because it has to mobilize less processing power than with the full feature set. The only exception is the minimum latency. The reason for this is that almost all processing stages are already active with the default feature set. Therefore, enabling the full feature set has hardly any impact on the measured minimum latency.

When you compare the two traffic mixes, you’ll notice that the BP Enterprise mix achieves significantly higher values regarding the number of processed packets per second and new sessions per second. This difference is due to the composition of the traffic mixes. The BP Enterprise mix uses a greater number of different protocols. For each protocol, new sessions are created and the more sessions you have, the more packets are exchanged. This is why the Cisco EMIX 2012 utilizes the available bandwidth with fewer sessions (and thereby fewer processed packets) than the BP Enterprise mix.

The performance values in the table above are not polished for marketing purposes and give you an impression of the performance you can really expect when you deploy Threat Defender in your network. For comparison, see again the results we obtained under “ideal test conditions” for this system:

 

Ideal test conditions

Throughput

33.4 Gbit/s

Packets per second

4,940,000

Minimum latency

4.125 μs

New sessions per second (TCP)

410,000

 

You can clearly see the discrepancies between optimized and realistic measurements.

To truly evaluate performance, a neat set of optimized results is completely useless. Instead, you need to look at a range of values that is obtained under realistic conditions and take the enabled feature set into account to get a complete picture and to come to a fact-based decision.

If you’d like to know how the selected hardware impacts performance tests, read our upcoming blog post where we will compare a single-socket and a dual-socket system.

 

 

Get our newsletter

Subscribe to our mailing list

* indicates required

cognitix GmbH will use the information you provide on this form to be in touch with you and to provide updates and marketing. Please let us know all the ways you would like to hear from us:

You can change your mind at any time by clicking the unsubscribe link in the footer of any email you receive from us, or by contacting us at noreply@cognitix.de. We will treat your information with respect. For more information about our privacy practices please visit our website. By clicking below, you agree that we may process your information in accordance with these terms.

We use MailChimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to MailChimp for processing. Learn more about MailChimp's privacy practices here.