Not VB ESA - M365 certified and want to be? See how your product can be enrolled in the test programme.
This page is designed to give you insight into how the VB ESA - M365 programme is set up and runs, and how your product can be enrolled in the test programme.
VB ESA - M365 is a continuously running performance test programme for products that supplement Microsoft 365’s native email security. The programme is designed to quantify and compare the incremental protection these products provide against spam, phishing and malware emails that were not already filtered by Microsoft 365, while maintaining strict control over false positives.
Our testing process uses real-world email streams to evaluate Microsoft 365 email security add-ons. Each product is exposed continuously to unwanted and legitimate emails, including spam, phishing, malware, ham and newsletters.
A central feature of the test is the Microsoft 365 comparative base. Emails filtered by Microsoft 365 itself are excluded from product scoring, allowing the test to focus only on the residual set that passed Microsoft 365 and remained available for the tested add-on to detect.
To quantify performance, we categorize outcomes into true positives, true negatives, false positives and false negatives. The main detection metric is the Incremental Detection Rate (IDR), which measures how many unwanted emails from the residual set were detected by the tested add-on.
For more detailed information, you can refer to the full methodology.
The following types of testing are available:
VB ESA - M365 is designed for products that add protection to Microsoft 365 in one of two common deployment models:
General
Testing process
Pricing and certification
AMTSO compliance
These are the major steps you can expect:
The feedback we provide is specific to your product and non-comparative. It includes:
Both unwanted and legitimate emails are used in the test.
VB ESA - M365 uses a Microsoft 365 tenant with a licensed user account configured for the tested product. In many cases, the product itself is hosted by the vendor as part of its normal service model. Where relevant, setup is carried out jointly with the vendor to ensure the product is connected correctly to the test tenant.
We work with you during the initial setup and provide opportunities to validate that the tested configuration reflects the intended production use case. For products where this is appropriate, vendors may audit the product configuration, state or logs remotely during the test.
In most cases, yes. The test framework can accommodate common actions such as:
The test ultimately normalizes these product actions into a binary result: the email was treated as unwanted, or it was not.
No. One of the fundamental rules of fair testing is that any test starting out as a private test cannot be made public, and a public test cannot be made private. This avoids cherry-picking of favourable results.
Yes, if we suspect that a technical issue is affecting the results. For example, if we encounter an excessive number of false negatives or false positives, or if the product appears misconfigured or unstable, we will work with you to investigate. The review and dispute phase serves as a final checkpoint before publication of public results.
Both public and private tests are available under a flexible commercial model. To discuss pricing and the most suitable arrangement for your product, please get in touch.
Public testing is conducted on a continuous basis, with official public periods designated within the programme. If you are looking to join with a particular reporting timeline in mind, it is best to start the discussion in advance so setup and validation can be completed before the relevant official period begins.
No. Awards are issued for a specific product edition and tested configuration. They do not automatically extend to derivative editions, OEM-licensed engines or other product variants that were not part of the test.
The AMTSO certification helps ensure that you receive a testing service that operates within established industry expectations for fairness, transparency and relevance. This benefits vendors directly and also increases confidence in the published reports.
These are ultimately defined by the AMTSO Testing Protocol Standard. In practical terms, vendors participating in AMTSO-audited tests are typically asked to register on AMTSO’s contact list and provide feedback before and after the test regarding how the testing was conducted.
AMTSO audits and certifies tests periodically rather than instantly. Because vendor feedback after the test is part of that process, AMTSO compliance can only be confirmed after the public report has already been released.
Ready to get the conversation started? Please email us at [email protected].