A return to traditional methods

Stuart King

Automated test procedures are the lazy, misleading way to test security. Use your loaf too.

Web product vulnerability testing has become too easy. A cookbook approach would be to take one off-the-shelf commercial scanning tool, point it at the application you want to test, hit the Scan button, and finally send the report to anyone who's interested.

While I'm an advocate for making a tough task easier, web products are complex, and the results from running a 'fire and forget' scanning tool can at best be a mixture of false positive reports and some correctly identified vulnerabilities; at worst they may be thoroughly misleading.

Let's take cross-site scripting (XSS) as an example. Many of the security test reports that I have seen identify XSS as high risk vulnerabilities. I'm not saying that XSS is not a vulnerability worth reporting, but it's important to keep things in context. Much of the time the reports show nothing more than an unvalidated parameter into which the scanning software has successfully managed to insert a message box that says 'hello'.

This occurred fairly recently with a vendor producing a report detailing a number of 'high risk' XSS vulnerabilities. My response was to ask for them to show me a workable exploit that would justify their rating. They were unable to. In fact, most reported XSS issues are likely to be nothing more than evidence of sloppy programming rather than a real risk.

Often point-and-click security scanning and reporting is associated with reports that reflect specific legislation and regulations. It is easy to be drawn into an expensive work program to correct reported faults. Mostly, this should be much further down the priority list. Worse still, if the reports are needed to prove compliance with a particular scheme, for instance the Payment Card Industry standards, then a qualified report could result in a non-compliance status and possible financial penalties.

Show me

Of course, another problem is that we might begin to lose confidence completely in both the vendors we use and the reports they produce. My advice is to insist on seeing a thorough analysis performed within the reports so that stated exploits can be reproduced. That doesn't mean reproducing a pop-up window with a 'Vulnerable to XSS' message, but rather some meaningful code that demonstrates exactly how and why the vulnerability is so serious.

The trouble with black-box automated testing is that most of the really serious issues don't get noticed at all. I've yet to encounter vulnerability testing software that will identify a poor password policy, or report on poorly implemented cryptography. In fact, they should also show where weaknesses, if they exist, stem from a lack of understanding or poor training and discipline by the developers.

Root cause

This is a root cause of the problem. Every scripting error that I have ever come across, whether it is SQL injection or an XSS error, is the result of the application source code failing to follow a standard. Usually it's simply the result of a lack of validation; sometimes a careless mistake, and sometimes because the developer does not understand the implications of what has been coded.

It's an old problem. The traditional solution is to ensure that code follows a standard policy and receives sufficient review. Black box testing with automated tools can find the areas where developers have been careless, but it is only a thorough white-box review of the whole system that will tell you where the real problems are. This will enable you to solve them from the top down rather than fix things, one issue at a time.

It may not be sexy to return from modern tools and techniques and to more practised methods. But if you are serious about security, want to protect the business from expensive over-reaction to inaccurate reports, and still produce a superior, more reliable product, then the old ways are the best.

About the author
Stuart King (CISSP) is a senior information security practitioner employed by the Reed Elsevier Group in the UK to address, analyse and describe solutions for risk associated with online products. Stuart is also an expert on compliance and in dealing with the security challenges associated with outsourcing and off-shore vendors. He may be contacted at stuart.king@reedelsevier.com.