Tips on evaluating/buying security tools
Strategies/approaches for evaluating Security Tools
By Seán Boran
December 20, 1999. How do you decide which tool to
use for a particular problem? Freeware, shareware or commercial? How do you navigate the
sea of marketing promises, comparisons, bugs, large number of products, prices and still
feel you have chosen the "right one"?
This article presents a few practical tips for buying/ testing/ evaluating
Your feedback is welcome on this article.
Choosing a product or system depends on considering several criteria. We examine the
decision in terms of:
- Ease of use, cost of use
- Features and Functions
- Bugs & limitations
- Relationship with vendors
- Maintenance fees
- Summary - making a choice
Many security tools have suffered from poor user interfaces, others are so flexible
that customising it for your environment can takes days or weeks.
- Consider the time it takes to learn the interface.
- Consider how easy it is to automate the tool (is a command line and GUI interface
- Are GUI and command-line consistent or mutually-exclusive? (it is hard to believe, but
some tools only let you use the GUI or the command line, not both).
- Consider the time to fully configure the tool to optimally work in your environment.
Ideally a tool should have a nice GUI for quickly learning how to get to grips with it,
a command line interface for expert-level automation and templates configurations for
several different typical scenarios/systems.
The military world has much experience in the rigorous evaluation of security products.
The ideas presented here stem from the ITSEC and TCSEC standards (see References for more information).
There are two key areas of evaluation to consider for security products:
- Functionality & features: What the product promises to do.
- Assurance: How sure can you be that the product protects correctly and effectively?
Functionality for security products can be divided into categories such as:
- Identification / Authentication: When users or programs communicate with each other, the
two parties verify each other's identity, so that they know who they are communicating
- Accountability/Audit Trail: The ability to know who did what, when, where. Users are
responsible and accountable for their actions. Automatic audit trail monitoring and
analysis to detect security breaches.
- Access Control: Access to specified resources can be restricted to certain entities.
- Object Reuse: Objects used by one process may not be reused or manipulated by another
process such that security may be violated.
- Accuracy / Integrity: Objects (information and processes) are accurate and complete.
- Secure information exchange: Information transmitted adheres to expected levels of
authenticity, confidentiality, and non-repudiation.
- Reliability / Availability: Information and services are available when needed.
- Security management: Security components need administration, it can be complex
especially for wide-scale installations.
ITSEC and TCSEC define functionality classes such as C2 and B1. Most UNIX-like systems
and NT can reach C2, but no standard systems reach B1 level (which includes features like
data labelling and mandatory access control). Specialised products are available however.
We won't go into any further detail here, see the References
section for further reading. Suffice to say, when evaluating a critical security
component, it is worthwhile comparing your requirements with these standards and checking
for approved products. ITSEC is an improvement on TCSEC in several areas such as assurance
and secure networking.
- Don't expect leading-edge products to work perfectly.
- Reference customers: can be valuable, particularly for expensive, complex systems. Check
how many systems are really installed and productive and what the
production/administrative problems are.
- West Coast Lab's Checkmark is a system which
tests and certifies computer security products.
- ICSA also certify security products www.icsa.net/services/product_cert/products.shtml
- ITSEC: Assurance is defined in terms of correctness (confidence that the product
actually does what you expect it to do) and effectiveness (the product resists attack, is
not easy to misconfigure).
ITSEC defines two levels of assurance that are relevant to commercial products:
E2: Functional testing that the products satisfies its security target. Informal
description of the detailed design. Evidence of functional testing shall be evaluated.
There shall be a configuration control system and an approved distribution procedure.
E3: In addition to the requirements for level E2, the source code and/or hardware drawings
corresponding to the security mechanisms shall be evaluated. Evidence of testing of those
mechanisms shall be evaluated.
E3 does provide a nice level of assurance and some standard products do provide this
"out of the box" (e.g. some Firewalls, UNIX and NT4).
- Don't test until you know what you want to test.
- Be methodical: test in detail, document what works and what doesn't work.
- Search the Internet (websites & newsgroups) for tests made by others. Search
relevant magazines, note their findings, try to confirm their results. Don't blankly
believe what others say, everyone has their own perspectives and prejudices (try to
understand what they are).
- Contact and share test results with peers.
What are the known bugs/limitations? If a vendor won't tell you, or says there aren't
any, be very sceptical.
Does the vendor have a history of fixing bugs, or just adding new features? Fewer and
fewer companies make the effort to fix bugs, since large software houses (we won't mention
any here..) are showing the bad example of not fixing bugs, adding gimmicky features and
making buckets of money in the purpose.
In buying your product from a particular vendor, your are "voting" in it's
favour. So you should basically agree with the idea of that vendor becoming more powerful.
Do you like the idea of Open Source?
- If so, don't wait for it to "happen", make an effort to use Open Source
products. Until recently, Open Source products were only for the tech-savvy, they are now
often easy to use and (optionally) commercially supported.
- Large corporations should consider investing in Open Source relevant to their business.
Companies can benefit significantly from a co-ordinated plan for using, integrating,
improving and contributing to Open Source projects.
- If you find and use free products, consider sponsoring them in some way (buy CD's,
T-Shirts, documents, support, give donations, help out yourself with development or
documentation). The developers of free products do need money to live, even if their focus
is not at all financial. Feel guilty if you just take advantage of others and their good
- Be sceptical. Salesmen are trained to be smooth and tell you what you want to hear! Try
to keep a friendly business relationship (but not too friendly... stay objective).
- But, understand their needs to make sales and earn enough money to survive.
- Try to find a win-win scenario where you test their products, if serious bugs are found,
they fix them and sell the product to you for a reasonable price. You win by getting a
good product and good support.
- Vendors will price products based on what they think they can get away with (this is
- You want to pay a price which corresponds to value of using the product to you, or which
corresponds to your budget, etc.
- Understand that Vendors are also constrained by sales volumes and complexity. PC
databases are cheaper than UNIX & Mainframe databases, not without reason.
- Timing & obsolescence: Consider how soon the product will become obsolete and how
soon you really need it. Try to buy "just-in-time", especially since many
products can be got within minutes or hours over the net.
- Be sceptical. Most vendors are looking for a regular, high-margin revenue stream.
- But understand. They need a regular revenue stream to provide support, fix bugs and make
- Most vendors try to get up to 25% for regular updates and support. If more than a few
hundred dollars are involved, negotiation is needed to get it down to about 15% or so.
- Often you can get some heavyweights in your purchasing department to do these
negotiations, since it's not a technical issue.
- If your don't really need maintenance, don't get it. An "upgrade" two years
down the line may be way cheaper (if you don't need support).
The final choice will depend on how tempting the product is in several categories and
how important (weighting) each of the criteria are for the system in question.
- features, functionality
- ease of use, cost of use
- price & maintenance fees
- relationship with the vendor (and how they reacted to your test report)
- how appropriate the tool is to your environment
- how quickly you expect it to be obsolete
- Your test results and other people's.
Seán Boran is an IT security consultant based
in Switzerland and the author of the online IT Security Cookbook.
Update: 05 September, 2001