Thorough, independent tests are a vital resource as cybersecurity leaders and their teams evaluate vendors’ abilities to guard against increasingly sophisticated threats to their organization. And perhaps no assessment is more widely trusted than the annual MITRE Engenuity ATT&CK Evaluations: Enterprise.
This testing is critical for evaluating vendors because it’s virtually impossible to evaluate cybersecurity vendors based on their own performance claims. Along with vendor reference checks and proof of value evaluations (POV) — a live trial — in their environment, the MITRE results add additional objective input to holistically assess cybersecurity vendors.
Let’s dive into the 2023 results. In this blog, we’ll unpack MITRE’s methodology to test security vendors against real-world threats, offer our interpretation of the results and identify top takeaways emerging from Cynet’s evaluation.
How does MITRE Engenuity test vendors during the evaluation?
The MITRE ATT&CK Evaluation is performed by MITRE Engenuity and tests the endpoint protection solutions against a simulated attack sequence based on real-life approaches taken by well-known advanced persistent threat (APT) groups. The MITRE Engenuity ATT&CK Evaluations: Enterprise tested 29 vendor solutions by emulating the attack sequences of Turla, a sophisticated Russia-based threat group known to have infected victims in over 45 countries.
An important caveat is that MITRE Engenuity does not rank or score vendor results. Instead, the raw test data is published along with some basic online comparison tools. Buyers then use that data to evaluate the vendors based on their organization’s unique priorities and needs. The participating vendors’ interpretations of the results are just that — their interpretations.
So, how do you interpret the results?
That’s a great question — one that a lot of people are asking themselves right now. The MITRE Engenuity ATT&CK Evaluations: Enterprise results aren’t presented in a format that many of us are used to digesting (looking at you, magical graph with quadrants).
And independent researchers often declare “winners” to lighten the cognitive load of figuring out which vendors are the top performers. In this case, identifying the “best” vendor is subjective. Which, if you don’t know what to look for, can feel like a hassle if you’re already frustrated with trying to assess which security vendor is the right fit for your organization.
With these disclaimers issued, let’s now review the results themselves to compare and contrast how participating vendors performed against Turla.
MITRE ATT&CK Results Summary
The following table presents Cynet’s analysis and calculation of all vendor MITRE Engenuity ATT&CK Evaluations: Enterprise test results for Overall Detection, Overall Prevention, and a Total Rating.
Cynet’s formula for Detection Rate is the total number of attack steps detected across all 143 sub-steps (MITRE Engenuity defines this metric as “Visibility”). Cynet defines Prevention Rate as the percentage of attack sub-steps that did not execute due to the vendor detecting and blocking the threat early in the attack lifecycle, thus not allowing additional malicious sub-steps to execute.
How’d Cynet do?
Based on Cynet’s analysis, our team is proud of our performance against Turla in this year’s MITRE Engenuity ATT&CK Evaluations: Enterprise, outperforming the most vendors in several key areas. Here are our top takeaways:
- Cynet delivered 100% Detection: (19 of 19 attack steps) with NO CONFIGURATION CHANGES
- Cynet delivered 100% Visibility: (143 of 143 attack sub-steps) with NO CONFIGURATION CHANGES
- Cynet delivered 100% Analytic Coverage: (143 of 143 detections) with NO CONFIGURATION CHANGES
- Cynet delivered 100% Real-time Detections: (0 Delays across all 143 detections)
See the full analysis of Cynet’s performance in the 2023 MITRE ATT&CK Evaluation.
Let’s dive a little deeper into Cynet’s analysis of some of the results.
Cynet was a top performer when evaluating both visibility and detection quality. This analysis illustrates how well a solution does in detecting threats and providing the context necessary to make the detections actionable. Missed detections are an invitation for a breach, while poor quality detections create unnecessary work for security analysts or potentially cause the alert to be ignored, which again, is an invitation for a breach.
Cynet delivered 100% visibility and perfectly detected every one of the 143 attack steps using no configuration changes. The following chart shows the percentage of detections across all 143 attack sub-steps before the vendors implemented configuration changes. Cynet performed as well as two very large, well known, security companies despite being a fraction their size and far better than some of the biggest names in cybersecurity.
Cynet provided analytic coverage for 100% of the 143 attack steps using no configuration changes. The following chart shows the percentage of detections that contained important tactic or technique information across the 143 attack sub-steps, again before configuration changes were implemented. Cynet performed as well as Palo Alto Networks, a $76 billion publicly traded company with 50 times the number of employees and far better than many established, publicly traded brands.
Still have questions?
Understandable.
In this webinar, Cynet CTO Aviad Hasnis and ISMG SVP Editorial Tom Field review the recently released results and share expert advice for cybersecurity leaders to interpret the results to find the vendor that best fits the specific needs of their organization. He’ll also share more details on Cynet’s performance during the tests and how that could translate to your team’s unique goals.
Author: George Tubin, Director of Product Strategy, Cynet
