Ok, I found out what the problem was. The culprit is not prelink. It is
slocate. It appeared to be prelink because both are run by cron approximately
at the same time. BTW, I do not use swap space.
When /etc/cron.daily/slocate.cron is run, the memory is gradually filled up.
I'm not talking cache here. After killing slocate when memory was about
completely filled up:
[nomis80@dhcp101 nomis80]$ free -m
total used free shared buffers cached
Mem: 755 736 18 0 149 50
-/+ buffers/cache: 537 217
Swap: 0 0 0
As you can see from this output of top, memory usage currently does not
justify 537 MB of used memory:
6816 nomis80 15 0 22720 22M 15712 S 0.0 2.9 0:01 0 firefox-bin
3535 nomis80 16 0 21484 20M 15140 S 0.0 2.7 0:07 0 kmail
3533 nomis80 15 0 17452 17M 13628 S 0.0 2.2 0:00 0 kopete
3556 nomis80 15 0 13280 12M 10880 S 4.3 1.7 0:01 0 kdeinit
3518 nomis80 15 0 13172 12M 10796 S 0.0 1.7 0:01 0 kdeinit
3363 root 15 0 32060 12M 2488 S 1.1 1.6 0:10 0 X
3510 nomis80 16 0 12712 12M 10456 S 0.0 1.6 0:00 0 kdeinit
3493 nomis80 15 0 11828 11M 9984 S 0.0 1.5 0:00 0 kdeinit
5449 nomis80 15 0 11584 11M 9988 S 0.0 1.4 0:00 0 kdeinit
3514 nomis80 16 0 11424 11M 9628 S 0.0 1.4 0:01 0 kdeinit
3516 nomis80 16 0 10900 10M 9340 S 0.0 1.4 0:00 0 kdeinit
3520 nomis80 16 0 10640 10M 9156 S 0.0 1.3 0:00 0 kdeinit
3526 nomis80 15 0 10520 10M 8532 S 0.0 1.3 0:00 0 korgac
3499 nomis80 15 0 10500 10M 9076 S 0.0 1.3 0:00 0 kdeinit
3524 nomis80 15 0 10356 10M 8952 S 0.0 1.3 0:00 0 kdeinit
3936 nomis80 16 0 10308 10M 8044 S 0.0 1.3 0:02 0 kdeinit
3527 nomis80 15 0 10108 9.9M 8096 S 0.0 1.3 0:00 0 kgpg
3538 nomis80 15 0 10020 9.8M 7456 S 0.0 1.2 0:00 0 kalarmd
3513 nomis80 15 0 9544 9540 8300 S 0.0 1.2 0:00 0 kdeinit
3530 nomis80 16 0 9432 9432 8220 S 0.0 1.2 0:00 0 kdeinit
3490 nomis80 16 0 7836 7836 7380 S 0.0 1.0 0:00 0 kdeinit
3508 nomis80 15 0 6996 6996 4180 S 0.0 0.9 0:02 0 artsd
3487 nomis80 16 0 6612 6612 6308 S 0.0 0.8 0:00 0 kdeinit
3484 nomis80 16 0 5728 5728 5568 S 0.0 0.7 0:00 0 kdeinit
3302 xfs 16 0 4004 4004 980 S 0.0 0.5 0:01 0 xfs
Scott Sloan suggested that /dev/shm might get filled up. It does not look like
it does:
[root@dhcp101 shm]# cd /dev/shm
[root@dhcp101 shm]# ls -a
. ..
[root@dhcp101 shm]# du -h
0 .
Unmounting /dev/shm has no effect.
[root@dhcp101 /]# umount /dev/shm
[root@dhcp101 /]# free -m
total used free shared buffers cached
Mem: 755 737 17 0 150 50
-/+ buffers/cache: 536 218
Swap: 0 0 0
I tried allocating memory as Matias Feliciano suggested. After a lot of disk
trashing (is that even normal?), this is what free showed:
[nomis80@dhcp101 nomis80]$ free -m
total used free shared buffers cached
Mem: 755 93 661 0 1 11
-/+ buffers/cache: 80 674
Swap: 0 0 0
So this procedure effectively liberated the memory. This is weird.
Just for fun, I started again updatedb from slocate.cron. Again, the memory
was filled up. And now, /dev/shm was unmounted. I noticed that the filling up
happens gradually, but quite fast. So I suspect that one filesystem is
causing this and should not be scanned. Which filesystems do I have mounted?
[nomis80@dhcp101 nomis80]$ mount
/dev/hda2 on / type ext3 (rw)
none on /proc type proc (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/hda1 on /mnt/windows type ntfs (rw)
nomis80.intranet:/home/public on /mnt/public type nfs (rw,addr=192.168.1.254)
Which filesystems are skipped by updatedb?
/usr/bin/updatedb -f "nfs,smbfs,ncpfs,proc,devpts" -e
"/tmp,/var/tmp,/usr/tmp,/afs,/net"
Maybe it is ntfs eh? So I add ntfs to -f. Liberate with huge malloc'ing. Rerun
slocate.cron.
[nomis80@dhcp101 nomis80]$ free -m
total used free shared buffers cached
Mem: 755 400 354 0 80 54
-/+ buffers/cache: 264 490
Swap: 0 0 0
Excellent! That worked.
So it was all Microsoft's fault. Or was it the fault of unsupported software?
Or maybe it was simply a bug in updatedb, or maybe in the kernel? I don't
care, I fixed it!
--
Simon Perreault <nomis80@xxxxxxxxxxx> -- http://nomis80.org
