Steven M. Christey wrote: >Besides which packages were found to be vulnerable, it seems like it >would be equally or more informative to know which other packages were >audited and not found to have bugs. The bulk of the "7500 man-hours" >were probably spent *confirming* the security of some of the software, >and some students may have accidentally selected well-written >software. Sadly, that is intractably hard. As Dijkstra would be quick to remind us, testing can prove the existence of bugs, but never their absence. The problem is that confirming the security of any real software package is far too hard to be done in any class project; it is a task that is way beyond what is feasible in the time available. Crispin Cowan tried to put together an effort (Sardonix) to get people to audit software packages and report on which ones were audited and confirmed to be secure. The problem is that this is ridiculously hard, far harder than trolling for a security hole. There is also the sociological problem that we reward people more for finding security holes than for finding none. To give you some idea of the cost of finding security holes in modern software packages, let me relate some experience from the graduate security class I teach. In Fall 2002, I gave the students a homework where I asked them to pick some program of interest to them and spend 3-6 hours auditing its security. About half of the students found at least one security hole in the package they had picked. Because the exercise was so well received by students, I gave the same homework in the class this year, and saw about the same ratio. This is not a very scientific experiment, but it suggests that if we pick a software package at random, point a smart person with some security training (not necessarily a security expert) at it, and let them have a few hours, then about 50% of the time the software package will turn out to have some security hole. Obviously, this is only a lower bound on how many of our applications are insecure. I will leave you to speculate how many software packages might harbor security flaws if we gave the auditor more time, training, and resources. If you're curious, you can see the homework assignments here: http://www.cs.berkeley.edu/~daw/teaching/cs261-f02/homeworks/hw2.html http://www.cs.berkeley.edu/~daw/teaching/cs261-f04/homeworks/hw2.html Also, you can see some remarks I wrote back in Fall 2002 about the audits people did that semester: http://www.cs.berkeley.edu/~daw/teaching/cs261-f02/solns/hw2.html You can see their audits from Fall 2002 on the Sardonix web site: https://www.sardonix.org/Audited.html Anyway, the point is that with 3-6 hours, you can often find a security flaw in a real software package -- but confirming the security of a security package in that amount of time is hopelessly impractical. Coming to any confidence whatsoever that the package is free of bugs would take orders of magnitude more time than available in the aforementioned homework exercises. It is unfortunate that it is not easier to verify the security of the software we rely upon, but these are the facts of life today. You asked about the relative proportion of different kinds of security flaws. Of the audits I saw in my class, most were buffer overruns, and there were a number of access control flaws (either missing or incorrect access control measures). However, I wouldn't read too much into this -- the students only saw in class a few examples of implementation flaws, so they might not have been looking for other kinds of flaws. Also, I asked them to use RATS to help them with part of their audit, and that tends to direct your attention more towards low-level implementation defects that RATS can find with simple lexical analysis rather than higher-level application security issues.
