VB100 for vendors

This page is designed to give you insight into how the VB100 certification programme is set up and runs, and how your product can be enrolled to the test bench.

Overview

The VB100 certification programme is for vendors of Windows endpoint security products who seek to distinguish their product by subjecting it to independent testing.

Programme participation is available on an annual basis, with quarterly scheduled tests. To earn and retain the VB100 certified status, your product must demonstrate that it is consistently capable of detecting common malware statically, with the following thresholds:

  • False negatives: Missing at most 0.5% of malicious test cases (i.e. 99.5%+ detection rate)
  • False positives: At most 0.01% false positive rate for clean test cases.

Products that meet the criteria receive VB100 Certified status and are featured on our website. The certified status is retained for the duration of the participation, as long as the scheduled tests are successful and the criteria are met (failing one test and recovering in the subsequent one is allowed).

 

Test scope

VB100 covers static detection of common Windows PE-type malware.

For a comprehensive guide on how we test, please consult the VB100 test methodology.

 

Private tests, one-off tests and custom tests

Private tests are typically one-off arrangements that are conducted in the same manner as the public VB100 tests, however the results are shared privately with you. This makes them a great, low-commitment opportunity for a number of cases, such as:

  • receiving an impression of your product's performance prior to joining the public VB100 test programme
  • testing new product editions or builds prior to release
  • when you only need an efficacy report for a specific and limited audience (such as Microsoft MVI membership or investors).

Private tests also receive priority treatment and results can be often delivered in as little as 1-2 weeks, depending on internal schedules.

Should you only need a single, public report of your product's performance as per the VB100 framework, this can also be arranged. Such single tests, however, do not attract a VB100 certified status since the product is not subjected for periodic testing. Rather, a single test report is released for such a test.

Custom tests based on an altered methodology – or even a completely custom one – are also a possibility. Please get in touch with us at vb100@virusbulletin.com to discuss your projects.

 

FAQs

General

Testing process

Pricing and certification

 

General

 

How does VB100 differ from other tests?

A few highlights:

  • Certification vs. comparative: Whereas most endpoint security tests are comparative by nature, VB100 is a certification test series. Comparative tests seek to rank tested products and determine which one performs better, however VB100 focuses on verifying whether the product tested is capable of meeting the VB100 baseline standard.

  • Schedules: VB100 can deliver test results much faster than a comparative test. In such tests, all products need to be tested in parallel, therefore strict schedules are to be followed. VB100 offers greater flexibility – you can essentially commence testing at any time and receive the test results in weeks, instead of the (quite possibly several) months involved with comparative testing.

  • Coverage: By focusing on the static detection layer only, VB100 can evaluate a very large number of test cases, resulting in greater statistical relevance.

 

What are the steps involved in participating in the VB100 certification programme?

These are the major steps you can expect:

  • Discussing your project – what are your objectives, what sort of test arrangement would work the best for you and when would you like to commence testing, etc.

  • Once the outline takes shape, we ask you to sign a Test Agreement, which covers all the relevant T&Cs for participation, and to complete a short product boarding form to submit your product. On initial submission we will perform a preliminary 'smoke test' to verify your product's compatibility with the VB100 testbed. This preliminary test is also a good opportunity to troubleshoot any technical difficulties before the actual test commences.

  • In each test cycle:
    • Your test starts on a scheduled date.
    • We complete the test then collect and review the resultant test data.
    • A preview of the results is made available to you, allowing you to review (and if needed, contest) the results.
    • Test results are finalized and your certified status (if any is applicable) is updated. We will supply you with a certification logo.

 

Testing process

 

How long does it take to receive the test results?

Subject to internal schedules, as little as 1-2 weeks. Please enquire about the current schedules at vb100@virusbulletin.com.

 

What kind of threats do you use for testing?

VB100 covers static detection of common Windows PE-type malware that changes from test to test.

 

What kind of clean files do you use for testing?

Our own collection of legitimate Windows program installers and the files dropped by those installers, including PE and non-PE files.

 

Do you share your test cases with us?

During your review of the test results, you will receive any false negatives and false positives by SHA256 and MD5 hash. Specific samples are available upon request (subject to quantity limits).

 

What do I receive during a test results review (i.e. 'disputes')?

When the actual testing / data collection phase is completed, you will receive preliminary test data for your review, including:

  • Preview test report with metrics
  • List of false positives and negatives by SHA256 and MD5
  • Captured product logs on request.

That feedback marks the beginning of the 'dispute period', during which you can review – and if you deem necessary – contest the test results.

 

Where do I learn more about the precise test process?

The VB100 Methodology is our best source on how the testing is done. We are also happy to answer any specific questions that you may have at vb100@virusbulletin.com.

 

How do I know if my product is compatible with the test?

As part of the boarding process we will do a preliminary 'smoke test' with a limited number of test cases. This is to verify the product's compatibility with the testbed. If any compatibility issues are identified, the test can be postponed or even called off.

 

If my private test works out really well, can I make it public? If my public test works out poorly, can I make it private?

Sorry, neither of these are possible. One of the fundamental rules of fair testing is that any test starting out as a private test cannot be made public, nor can a public test be made private. This is to prevent 'cherry picking' the favourable results.

 

Will you let me know if my product is not performing well during the test?

Yes, if we suspect that a technical issue is affecting the results. For instance, if we encounter an excessive number of false negatives / false positives, or if the results do not appear to make sense, or if the product crashes, etc. Generally, we can spot issues like these quite well, but should something avoid detection, the review ('disputes') phase, during which you get to verify your results, serves as a final checkpoint.

 

My product takes a specific configuration for this test, can you accommodate for that?

Generally speaking, yes we can, subject to certain limitations. For instance, if the configuration would be severely detrimental to the relevance of the report for the average use case, we may not be able to accommodate your request. Any significant deviations from default will be documented in the test report, along with the justification for those changes.

 

Can you test my brand new product that is yet to be integrated with the Windows Security Center?

Yes, we can. We regularly test pre-release products like that, often as part of the Microsoft MVI registration process. Regarding this latter scenario, please consider that Microsoft accepts third-party tests for such purposes at their discretion, so be sure to verify their latest requirements with them.

 

Pricing and certification

 

How much does testing cost?

Both annual VB100 plans and private tests are available in a highly competitive and adaptive pricing model that follows the value generated for your business, so whether you are a startup or an established player in the field we have a plan to suit you. To find out more please get in touch at vb100@virusbulletin.com.

 

How long does it take to get certified?

As soon as your product passes its first public test, your product receives certified status. This can take as little as 1-2 weeks (most commonly 4 weeks), depending on internal schedules.

 

How long do I keep my certified status?

Provided your product keeps meeting the certification criteria consistently, you keep your certified status up to 60 days after your test contract expires. We will invite you to renew your testing arrangements towards the end of your contract for onward participation and certification.

 

Does my certification cover my other product editions?

The certification is issued for a specific product edition and it does not cover any derivative products (other product editions, OEM-licensed engines, etc). This has to do with the framework of fair testing – fundamentally, a test lab may only make statements about its observations and thus extending the coverage to product editions the lab did not test would be a speculative matter.

 

Getting in touch

Ready to get the conversation started? Please email us at vb100@virusbulletin.com.

 

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.