Learn more about how VB ESA - M365 works.
VB ESA - M365 is Virus Bulletin’s comparative test series for email security products that add protection to Microsoft 365. It is designed to measure how much extra security these products provide beyond Microsoft 365’s native email filtering.
The test focuses on Microsoft 365 add-on products deployed in one of two common ways: as Integrated Cloud Email Security (ICES) products that connect through APIs, or as Secure Email Gateways (SEGs) that sit in front of Microsoft 365. In both cases, the goal is to assess the product’s incremental protection against spam, phishing and malware emails, while maintaining strong false positive control through the use of legitimate emails.
Our testing process uses real-world email streams to evaluate the performance of Microsoft 365 email security add-ons. Each tested product is exposed continuously to wanted and unwanted emails, including ham, newsletters, spam, phishing and malware.
A central feature of the test is the Microsoft 365 comparative base. Any email that Microsoft 365 itself filters is placed into this comparative base and excluded from product scoring. This means tested products are evaluated only on the residual set of emails that passed Microsoft 365 and remained available for the add-on to detect.
To quantify performance, we classify outcomes into true positives, true negatives, false positives and false negatives. The main detection metric is the Incremental Detection Rate (IDR), which measures how many unwanted emails from the residual set were detected by the tested add-on. Products that achieve strong incremental detection while keeping false positives very low can earn VB ESA - M365 certifications and badges.
For more detailed information, refer to the full methodology.
VB ESA - M365 hosts two broad categories of products:
Although these deployment models differ, both are tested against the same core objective: how effectively they add protection on top of Microsoft 365.
The following FAQs are designed to help you interpret VB ESA - M365 reports and understand how the test is set up and run.
VB ESA - M365 provides broad coverage of common unwanted email categories, including assorted spam, phishing and malware, using real-world email samples introduced into the test environment without unnecessary delay. Legitimate mail is also included through ham and newsletter feeds to measure false positives.
The test is focused on email filtering performance. It does not aim to evaluate adjacent capabilities such as data loss prevention, encryption, archiving or broader collaboration security features.
The Incremental Detection Rate (IDR) measures how much additional protection a tested product provides beyond Microsoft 365. Specifically, it is the proportion of unwanted emails detected by the tested add-on among the emails that were not already filtered by Microsoft 365.
This is important because VB ESA - M365 is not a standalone Microsoft 365 test. It is a test of the extra security added by third-party products deployed alongside Microsoft 365.
The VB ESA - M365 award signifies that a product has demonstrated strong incremental detection on emails that passed Microsoft 365, while also maintaining strict control over false positives.
The VB ESA - M365+ award signifies excellence on the same criteria, requiring even stronger detection and no false positives.
Additional badges may also be awarded for exceptional performance in specific areas, such as phishing or malware detection.
ICES products typically integrate directly with Microsoft 365 through APIs and often act on messages after delivery by moving or remediating them in the mailbox.
SEG products are typically deployed in front of Microsoft 365 via MX routing and can block or reject messages earlier in the mail flow.
Both types can improve protection, but they do so through different deployment models and technical mechanisms.
All tested products are assessed against the same principle: their ability to add protection beyond Microsoft 365’s own filtering. To make this possible, VB ESA - M365 excludes emails already filtered by Microsoft 365 and evaluates products only on the residual set.
Because ICES and SEG products differ in how they are deployed and how they act on emails, readers should interpret the results with that context in mind. However, the test normalizes product responses into a binary outcome: the email was either treated as unwanted or it was not.
The Microsoft 365 comparative base is the set of emails that Microsoft 365 itself filtered before the add-on product could meaningfully contribute. These emails are excluded from product scoring.
This creates a stable reference point for measuring the added value of the tested product, rather than conflating its performance with Microsoft 365’s native protection.
Yes. VB ESA - M365 uses in-the-wild emails collected through third-party feeds and Virus Bulletin’s own threat intelligence. Legitimate ham and newsletters are also sourced from real-world traffic.
Some limited modifications are made to support testing and protect operational details, but the test cases are not synthetic examples created purely for the lab.
No. Products participating in a public official test period cannot normally be withdrawn once that period has started. Public interest requires that the report be published regardless of whether the results are favourable to the vendor.
Virus Bulletin may choose not to publish results only in exceptional cases, such as serious technical problems or invalid test data that make publication irrelevant or unfair.
Absolutely, email us at [email protected].