Tips on evaluating/buying security tools

Strategies/approaches for evaluating Security Tools

By Seán Boran

December 20, 1999.  How do you decide which tool to use for a particular problem? Freeware, shareware or commercial? How do you navigate the sea of marketing promises, comparisons, bugs, large number of products, prices and still feel you have chosen the "right one"?

This article presents a few practical tips for buying/ testing/ evaluating security products.

Your feedback is welcome on this article.

Choosing a product or system depends on considering several criteria. We examine the decision in terms of:

  1. Ease of use, cost of use
  2. Features and Functions
  3. Testing
  4. Bugs & limitations
  5. Politics
  6. Relationship with vendors
  7. Price
  8. Maintenance fees
  9. Summary - making a choice


Ease of use, cost of use

Many security tools have suffered from poor user interfaces, others are so flexible that customising it for your environment can takes days or weeks.

Ideally a tool should have a nice GUI for quickly learning how to get to grips with it, a command line interface for expert-level automation and templates configurations for several different typical scenarios/systems.

Features and Functions

The military world has much experience in the rigorous evaluation of security products. The ideas presented here stem from the ITSEC and TCSEC standards (see References for more information).
There are two key areas of evaluation to consider for security products:

Functionality for security products can be divided into categories such as:

ITSEC and TCSEC define functionality classes such as C2 and B1. Most UNIX-like systems and NT can reach C2, but no standard systems reach B1 level (which includes features like data labelling and mandatory access control). Specialised products are available however. We won't go into any further detail here, see the References section for further reading. Suffice to say, when evaluating a critical security component, it is worthwhile comparing your requirements with these standards and checking for approved products. ITSEC is an improvement on TCSEC in several areas such as assurance and secure networking.



Bugs / limitations

What are the known bugs/limitations? If a vendor won't tell you, or says there aren't any, be very sceptical.

Does the vendor have a history of fixing bugs, or just adding new features? Fewer and fewer companies make the effort to fix bugs, since large software houses (we won't mention any here..) are showing the bad example of not fixing bugs, adding gimmicky features and making buckets of money in the purpose.

Your Politics

In buying your product from a particular vendor, your are "voting" in it's favour. So you should basically agree with the idea of that vendor becoming more powerful.

Do you like the idea of Open Source?

Relationships with vendors


Maintenance fees

Summary - making a choice

The final choice will depend on how tempting the product is in several categories and how important (weighting) each of the criteria are for the system in question.


Seán Boran is an IT security consultant based in Switzerland and the author of the online IT Security Cookbook.               Last Update: 05 September, 2001