On Wed, 21 Jan 2004, Eric Rescorla wrote: > Bugtraq readers might be interested in this paper: > > Is finding security holes a good idea? > > Eric Rescorla > RTFM, Inc. <http://www.rtfm.com/> > > A large amount of effort is expended every year on finding and patching > security holes. The underlying rationale for this activity is that it > increases welfare by decreasing the number of bugs available for > discovery and exploitation by bad guys, thus reducing the total cost of > intrusions. Given the amount of effort expended, we would expect to see > noticeable results in terms of improved software quality. However, our > investigation does not support a substantial quality improvement--the > data does not allow us to exclude the possibility that the rate of bug > finding in any given piece of software is constant over long periods of > time. If there is little or no quality improvement, then we have no > reason to believe that that the disclosure of bugs reduces the overall > cost of intrusions. It is a very weakly justified assumption that the number of Black Hat exploitations following Black Hat Disclosure is 'almost certainly less' than the 'peak rate after (Public) disclosure'. The rest of your paper rests heavily on that assumption - and it in fact nearly pre-determines your conclusion. It only takes _ONE_ more Warhol Worm from non-public security bugs than publically disclosed ones to invalidate your argument (this is obviously true in the _converse_ as well). This is because the _COST_ of such a worm is magnified beyond reason by the network. Each individual worm represents a large fraction of _all_ such intrusions and their cost. This is NOT a large statistical universe. Individual events skew the whole dataset. One major exploit may cost as much as thousands or even millions of smaller exploits. A bug allowing complete remote compromise of a box that is merely _connected_ to the net is of a completely different magnitude than one that causes Mozilla to mis-display a URL. Treating the two bugs as comparable is a significant error. It is not justified to believe that they are _generated_, _discovered_, _exploited_, or _fixed_ at the same rates or that they have even order of magnitude similiar costs. There remain too many large unknowns to justify _any_ good conclusions re bug hunting and disclosure vs the cost of intrusions. -- Benjamin Franz On that of which one cannot speak, one must remain silent. ---Wittgenstein
