It would be very interesting to see any results that try to compare the timeliness of vendor response. I attemped to conduct such a study
I would be too.
(...)a year and a half ago, but the study failed due to lack of time and a lot of other factors such as:
Well, it can be difficult to do compare these times without a common criteria. From personal experience, I think it is quite possible to relate some data (date the advisory is release and date the bug was published) and extract some conclusions. You might have issues with some specific bugs but, I think, the effects you described are diminished if you take a bird's view (and cover several years).
- how one "counts" the number of vulnerabilities. I view this as one of the main roles of CVE, however any set of vulnerability data must be normalized in *some* fashion before being compared, otherwise the stats will be biased.
When I have done this kind of analysis and comparisons I've always used CVE as the vulnerability count as well as the way to relate data from different sources (including vulnerability databases such as ICAT and vendor advisories for example).
(...)
I initially tried to cover a 3 month time span, but it really seemed like at least a year's worth was required.
IMHO the more data you use, the less the effects you described affect the statistical values you will get.
You can't simply compare published advisories against each other, because:
After all, if they refer to the same CVE names, why not?
- different vendors have varying criteria for how severe a bug must be before an advisory is published
That's not an issue, that is, after all, a vendor decission. However, bug severities could be determined by independent parties or through a common formula (see the EISPP Common Advisory Format Description I pointed to in a previous mail). The raw "time to fix" data could be partitioned by severity but, still, it could be a useful metric to compare between vendors.
- some advisories report multiple bugs, which could mean multiple disclosure and notification dates, and different times-to-fix
If advisories refer to different CVE names I don't see that as an issue either, it just means that different vulnerabilities were fixed at the same time.
- sometimes an interim patch is provided before the advisory
That is also a non-issue. Many security bugs can be fixed by appropiate configuration (or de-activation) of the operating system (i.e. procedures described in security manuals). Some interim patches just say 'remove the executable' which is not really a fix. If your target is to determine when a bug gets fixed you should only take into account when it does really get fixed.
- sometimes security issues are patched through some mechanism other than an advisory (e.g. Microsoft's service packs, which fix security bugs but don't normally have an associated security bulletin)
Which is not the best way to distribute security patches since researches (or IT managers) cannot really know what security bugs are fixed by what update.
- sometimes there are multiple advisories for the same bugs (SCO and Red Hat immediately come to mind)
Many vendors do this. In many cases they release a new version of the patch fixing issues with it and still refer to the previous vulnerability. I agree this is a difficult problem since the question arises: when was the bug really fixed? (in the first advisory or in the advisories published after it)
You also can't directly compare by "total bugs per OS" because of the variance in packages that may or may not get installed, plus how one defines what is or isn't part of the "operating system" as mentioned previously. One way to normalize such a comparison is to compare "default" installations to each other, and "most secure" installations to each other - although of course the latter is not always available.
That's why should compare "similar" operating systems. Say, different Linux distributions, or different propietary UNIX systems, which provide similar variance of packages and, in many cases, default installations. Comparing, for example, Microsoft and any UNIX/Linux operating system is comparing apples to oranges, it might be a good PR move but it seldoms provide any useful information. For a more thorough discussion I find point 3, section 4 (Security) in David Wheeler's "Why Open Source Software / Free Software (OSS/FS)? Look at the Numbers!" [1], very enlightening.
Fortunately, the percentage of vulnerability reports with disclosure timelines seems to have been increased significantly in the past year, so maybe there is a critical mass of data available.
I think there is sufficient data to do proper analysis of vulnerabilities from many vendors since many already provide accurate CVE mappings. Microsoft, unfortunately, is not one of those.
As a final note, I have the impression that most vendors (open or closed, commercial or freeware) don't track their own speed-to-fix, and *no* vendor that I know of actually *publishes* their speed-to-fix.
That is actually not really true. Debian does this already, read the FAQ item "How much time will it take Debian to fix vulnerability XXXX?" in the Security Manual [2]
I've actually tracked the information for Debian this year and published the results as part of the Debconf3 conference "Security in Debian: Food for thought and Discussion" [3]
Hopefully someday there will be a solid public paper that actually tries to quantify the *real* amount of time it takes to fix bugs, whether on a per-vendor or per-platform basis, and accounts for all the issues that I described above. (I know of a couple private efforts, but they are not public yet.) Of course, one would want to verify the raw data as well.
I also think that this a scientific paper on this issue is long overdue. It could maybe help clarify some misconceptions, which would be helpful considering the way this leads to FUD being published by vendors. The only (outdate) article I've seen which tries to (slightly) address this issue providing, at the same time, the raw data it used is "Linux vs. Microsoft: Who Solves Security Problems Faster?" by Jim Reavis for SecurityPortal [4]. Of course, it takes the Linux vs. Microsoft stanza which is not the best way to do it, but, still, is much better than many other FUD articles I've seen out there.
Best Regards
Javi
[1] http://www.dwheeler.com/oss_fs_why.html#security [2] http://www.debian.org/doc/manuals/securing-debian-howto/ch11.en.html#s-debian-sec- team-faq [3] Raw data and slides available at http://people.debian.org/~jfs/debconf/security/, see specifically slide #8 and #9. [4] http://web.archive.org/web/20010608142954/http://securityportal.com/cover/coversto ry20000117.html
