Eric, some thoughts (this is not an argument for or against full-disclosure, please take it as constructive criticism): After reading your paper I agree that the data you used may support your arguments, however, you missed some important points. You don't take into account the "type" of vulnerabilities being found in each of the applications that you've analyzed (you address Severity, but thats a seperate variable). I would argue that if you did that, you would come to some different conclusions. I would also argue that the caliber of bugs being found has increased quite substantially. Easy to find, easy to exploit vulnerabilities have for the most part been exhausted in applications that have been "sufficiently" scrutinized. The bugs being found today (in applications that have had a sufficient amount of scrutiny) are significantly more complex than a number of years ago (in the same application). In the early 1990's, you could find a common buffer overflow (using a blatant strcpy/strcat) in many common core Internet applications. Today, the vulnerabilities being discovered in these same applications are more complex off-by-one, signed/unsigned integer, compilar casting, or byte/character processing problems. Some good examples of this behaviour are sendmail and BIND. "Type" also infers the skill level required for a researcher to find a given vulnerability, due to it's difficulty, with some types being easy to find, while others extremely difficult. Entirely new classes of vulnerabilities are rarely discovered (but they still are). There will only be a finite quantity of these, and once those have been identified, the bar can't go any higher. Of course there can only be a finite number of vulnerabilities in any given application as well. Another two metrics that you don't measure is the amount of scrutiny that a particular application has had, or the size of an application. A large application that has had 30 vulnerabilities found in it by one researcher over 10 years cannot be compared to a small application that has had 30 vulnerabilities found in it by 200 researchers in 1 year. There were 30 vulnerabilities found in both, but the latter application will have improved in quality quite significantly, while the former not (assuming that both applications have the same average number of vulnerabilities per line of code). Unfortunately scrutiny is likely something that you cannot measure in many applications. So, on average I would argue that software quality (in terms of vulnerabilities being discovered) has improved for a given application that has, and continues to be, sufficiently scrutinized (not including substantial updates that introduce new bugs). You simply don't have all of the data points to prove it, and therefore may be missing important conclusions. I may be able to take specific applications, where we have sufficient visibility into scrutiny, size, and type/difficulty of vulnerabilities, and prove your theory wrong (sendmail is a possibility). My conclusion isn't based on the numbers, but simply on my experience with researching vulnerabilities since the early 1990's. Oliver Friedrichs Sr. Manager - DeepSight Symantec, Inc. (650) 381-8045 > Bugtraq readers might be interested in this paper: > > Is finding security holes a good idea? > > Eric Rescorla > RTFM, Inc. <http://www.rtfm.com/> > > A large amount of effort is expended every year on finding and patching > security holes. The underlying rationale for this activity is that it > increases welfare by decreasing the number of bugs available for > discovery and exploitation by bad guys, thus reducing the total cost of > intrusions. Given the amount of effort expended, we would expect to see > noticeable results in terms of improved software quality. However, our > investigation does not support a substantial quality improvement--the > data does not allow us to exclude the possibility that the rate of bug > finding in any given piece of software is constant over long periods of > time. If there is little or no quality improvement, then we have no > reason to believe that that the disclosure of bugs reduces the overall > cost of intrusions. > > The paper can be downloaded from: > http://www.rtfm.com/bugrate.pdf > http://www.rtfm.com/bugrate.ps >
