In answer to your questions about the Equivalent-Packages process:
1) You are right that the tool can get confused when there is little source in the package or if the majority of files include common things like readme/todo/makefile etc. One thing I could do is exclude source files which are excessively common. I do that in other tools where I use package similarity, but tried to keep this package equivalent tool as simple as possible. It is not difficult to implement and it should reduce some of the false positives.
2) When there are multiple possible matches, I simply choose the package with the highest similarity. One thing that I did was to run the tool against one repo only (instead of say the Fedora repo against the Debian repo) to find near duplicate packages. I have done this only so far for Debian https://github.com/silviocesare/Equivalent-Packages/blob/master/Clusters/Debian5Cluster
The first entry in the list is the base package, the remaining entries are the near duplicate packages and their similarities to the base package. An example from the Debian repo -->
libxml-um-perl libapp-control-perl:0.846154 libcrypt-des-ede3-perl:0.846154 libdata-buffer-perl:0.846154 libdb-file-lock-perl:0.846154 libio-tee-perl:0.846154 liblingua-preferred-perl:0.846154 liblingua-pt-stemmer-perl:0.846154 liblog-tracemessages-perl:0.846154 libpdf-reuse-barcode-perl:0.846154 libsort-fields-perl:0.846154 libtemplate-plugin-calendar-simple-perl:0.846154 libxml-filter-detectws-perl:0.846154 libxml-filter-saxt-perl:0.846154 libxml-handler-printevents-perl:0.846154 libxml-handler-trees-perl:0.846154 libxml-regexp-perl:0.846154
One possible method of reducing false positives is to ignore packages which are equivalent to more than one other package. Or perhaps it could require human intervention.
3) I did some trivial testing of unexpected matches. In fact one thing I looked at was when the same package name was in Fedora and Debian but the similarity was so low it didn't match. Suprisingly a not insignificant number of packages were like this. And manual verification showed in the ones I looked at, they were different packages. This demonstrates that if you base equivalence on names only, then you will get false positives.
I could add heuristics based on the package name to request human intervention, ie. if two packages are found similar and if the package names do not have 50% overlap, then request human verification. I am not sure how useful this will be because from experience, package names can sometimes be problematic.
--
Silvio
On Tue, Feb 1, 2011 at 1:13 AM, Tomas Hoger <thoger@xxxxxxxxxx> wrote:
Hi Silvio!
We currently do not use CPE names for security tracking in Fedora, so
On Mon, 31 Jan 2011 19:21:39 +1100 Silvio Cesare wrote:
> Debian maintain a list of CPE inormation for packages on their
> security tracker
> http://svn.debian.org/wsvn/secure-testing/data/CPE/list
I don't see an obvious benefit maintaining such list. Can you explain
briefly how you use it for Debian security tracking and what benefits
it brings?
It's not that uncommon to see new packages added to Fedora repositories
> This makes it relatively static except when packages are added or
> removed from the repository.
even after the release of some Fedora version.
I played a little more with this list and noticed few problems:
> In the past I generated an automatic mapping between packages in
> Debian and Fedora
> https://github.com/silviocesare/Equivalent-Packages/blob/master/NearestNeighbour/Debian5_Fedora13_Matches
- quite a few Debian packages map to Fedora arptools or binclock.
Probably packages with not much sources, where other file (license,
configure) confuse your tool to match unrelated packages
- there does not seem to be a good way to list cases where multiple
components contain the same sources. In Fedora, mingw32-* packages
are a good example, and the list often maps Debian package foo to
Fedora package mingw32-foo, while there is Fedora package foo that
should be similarly good match. Another example is
zlib:arm-gp2x-linux-zlib.
Did you review "unexpected matches" to see if the sources are really
similar, and how the match is picked when there are multiple "good
candidates"?
--
Tomas Hoger / Red Hat Security Response Team
-- security mailing list security@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/security
