Alexander Puchmayr posted on Sat, 04 May 2013 14:24:58 +0200 as excerpted:
Hi there,
How much memory is virtuoso supposed to use? If I got it right, it shall
use no more than the setting in System Settings/Desktop Search/Extended
Settings/Memory usage, which is in my case 128MB.
Now look at the real memory consumtion (from htop)
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
18831 alex 39 19 2633M 1900M 4440 S 0.0 48.1 17:55.39
/usr/bin/virtuoso-t +foreground +configfile /tmp/virtuoso_T18826.ini
+wait
It uses a virtual range of more than 2.5GB and keeps nearly 2GB locked
in memory (which is half of the available RAM). Is this *really*
necessary? What is virtuoso doing with all that memory and why is the
setting in System Settings obviously ignored? And the most important
question is, how to make virtuoso use a reasonable amount of memory?
BTW: I'm using kde-4.10.1 and virtuoso-server-6.1.6 on a gentoo system
(amd64)
Hello fellow gentooer! =:^)
FWIW, I had enough of semantic-desktop including virtuoso/nepomuk/strigi/
akonadi/redland/rasqal/etc. here, and turned off the whole thing.
USE="-semantic-desktop -virtuoso -redland" ...
Of course that means nothing kdepim related, which means no kmail, but
the new akonadi-based kmail wasn't stable here anyway, and I ended up
switching (after 9 years, since kde2 era!) to claws-mail. FWIW that's
what I use (with its feed-reader plugin) in place of akregator, as well.
The conversion wasn't exactly easy, but I've been very happy with the
results! =:^)
So I'm definitely not the most unbiased source of virtuoso information
you can find, but still...
If you google Linux memory, you'll quickly find that it's a very common
source of confusion but that the situation isn't as simple or usage as
bad as it may initially appear.
In particular, the virtual memory number is "virtually" (heh!) useless to
all but the most technical user, as it's EXTREMELY common for
applications to request huge amounts of memory from the allocator that
they never even touch at all. In practice, many apps allocate for worst-
case usage or even more, and actually use hardly any of what they
requested, at all. This practice is not just allowed, but arguably
encouraged by the kernel itself, which has a default over-commit ratio
that allows way more memory to be actually allocated than really exists,
either as physical RAM or as swap, on the system. Google for the OOM-
killer (OOM=out-of-memory) to see what happens when too many apps try to
actually use the memory they've allocated and the system really does run
out of memory.
So unless you have a very specific reason to track virtual memory,
don't. Simply pretend that figure doesn't exist, preferably by hiding
that column in htop entirely -- there's far more practical uses to which
that available display space can be put. =:^) (Of course if you're
curious just /how/ much memory an app is requesting, you can leave the
column visible. Just know what the number in it actually counts for, not
much.)
(I could suggest as a replacement, the "CODE" column, the text-segment or
code size of the process, or possibly the DATA column, the data segment
plus stack usage of the process, which I suspect would prove to be the
outlier here. Column information from the htop manpage.)
The RES aka RSS, resident set size (and the related MEM% column, which is
based on RES), is a somewhat more useful number, but even then, it's not
what it might first appear. In particular, killing that process won't
normally get you back the memory reported as RSS, nor will starting a new
process require the memory RSS reports it as using once started. This is
because a lot of that memory usage is shared with other processes. One
subset of that shared memory (shared library usage) is accounted for in
the shared column, but particularly for X-based apps, there's a whole lot
more than that going on under the hood. As just one example, consider
the number of apps using common fonts -- these must be kept in memory,
but it's a shared resource that AFAIK doesn't show up in the shr column,
as they're not shared libraries, but other shared resources.
That said, RSS does demonstrate that the 128 MB limit you mention
obviously isn't a total memory usage limit. Take this bit with a rather
large grain of salt as it's simply a guess, but I BELIEVE that limit
refers to the actual processed-and-cached (aka "digested") data size --
the size of the amount of data as actually stored in the database, that
it's allowed to keep cached in memory. But the actual runtime memory
usage will be far higher, not only because the app has to have all the
code to process that information (which I suspect is actually relatively
small, check CODE, LIB, SHR), but because the information itself will
need to be scanned in from the various files in ordered to be processed
and indexed in the first place -- THAT's what I suspect to be the
majority of the usage. If so, it should show up in htop's DATA column,
if you enable that. Which is why I said above that this would probably
the interesting outlier column.
All that said, nearly 2 GB of RSS, while /possibly/ /arguable/ /as/
reasonable for an indexer type app, is somewhat excessive compared to
most apps, and could EASILY be considered excessive to the point of not
being worth the hassle.
As a point of comparison, it's worth seeing my top users here, with
semantic-desktop disabled. (MEM% is far lower here as this system is a
16 gigs physical RAM system, while based on your figures, yours is
probably 4 gig RAM. As you can see I have a few more columns enabled,
too, including PGRP and SESN pid columns for reasons unrelated to the
current discussion, but VIRT isn't one of them.)
PID PPID PGRP SESN USER PRI NI S RES CPU% MEM% IO TIME
+ CPU NLWP Command
2639 2533 2521 2521 x 20 0 S 120M 0.0 0.8 0
2:25.17 3 5 /usr/bin/pan
2554 1 2521 2521 x 20 0 S 118M 0.0 0.7 0
0:14.42 0 4 kdeinit4: plasma-desktop [kdeinit]
2474 2473 2474 2474 root 20 0 S 97408 3.0 0.6 0
8:19.18 4 2 /usr/bin/X -nolisten tcp :0 -auth /h/x/.serverauth.2460
2537 2533 2521 2521 x 20 0 S 67148 4.0 0.4 0
9:37.07 0 4 kwin
2545 1 2521 2521 x 20 0 S 59864 0.0 0.4 0
0:48.59 4 3 /usr/bin/knotify4
2552 1 2521 2521 x 20 0 S 50368 0.0 0.3 0
0:07.02 5 15 kdeinit4: krunner [kdeinit]
As you can see, pan (the news client from which I am posting this, via
gmane.org's list2news gateway) is my biggest memory hog here, 120 MB RES/
RSS, with plasma-desktop following at 118 MB, and X behind it and
rounding out the top three at under 100 MB. (Off topic but possibly of
interest none-the-less, you might also notice my non-standard /h instead
of /home, and the simple "x" username...)
120 MB top RSS is quite a contrast to your near 2 GB virtuoso RSS... and
I've a 16-gig machine to play with, too, while it looks like you're
running 4-gig. Yeah, nearly half of that, almost 2-gig, tied up by
virtuoso DOES seem unreasonably bloated, caveats about interpretation of
RSS or no caveats! Get rid of it and you SHOULD see a difference!
Which is pretty much what I decided when I decided to kill the entire
semantic-desktop thing, not just at runtime, but being a gentooer with
the tools to do so thus exposed, at build-time, setting USE=-semantic-
desktop and toggling off related USE flags as well.
There is some functionality cost to that choice, however. In addition to
killing everything akonadi and kdepim related including kmail, dolphin/
konqueror lose their semantic-desktop-required metadata functionality,
including the information found in the info tab of the file properties
dialog. (The tab remains, but it's always blank.)
But the tradeoff was WELL worth it IMO as KDE was **MUCH** faster
afterward, even tho I had as much as possible runtime-disabled before I
toggled off USE=semantic-desktop and friends, so I wasn't even comparing
to the full runtime case, but to the mostly disabled case. To that point
(I disabled semantic-desktop in early kde 4.7, with it fully disabled by
4.7.1) I had always stated that kde4 until late 4.5 was pre-release
quality; that only with 4.5 was a properly release-quality kde4 available
to release a similar quality late kde 3.5. So while I considered say
4.5.4+ roughly EQUAL to say 3.5.8+ (with the exception of a couple 4.6
releases that were very buggy and regressed), it was only when I turned
off semantic-desktop and related options at BUILD time that I really
honestly could say that the kde4 experience was finally BETTER than late
3.5. Which is ironic, given that one of the big bullet points of kde4
was the whole semantic-desktop thing. But only by build-time disabling
semantic-desktop did I finally get a kde4 that not just matched in
experience for me, but finally surpassed, kde3. Runtime disabling
helped, and given that 48% memory figure it'd probably help you even
more, but it wasn't until I exterminated it to the extent possible at
build-time, that I could REALLY say kde4 was now not only equal to kde3
in experience for me, but actually BETTER.
YMMV as they say, but that's what I found, here. =:^)
I'm just glad that I'm a gentooer and thus have that choice exposed to
me. Until the very recent "kde-lite" efforts I've seen coverage of in
the last few weeks, I knew of no binary-based distro offering the same
"low-fat-kde" experience including reduced or killed semantic-desktop,
that gentoo makes possible with its USE flags. It'll be interesting to
watch that low-fat-kde project develop, and see if it gets any decent
user uptake and whether other distros follow suite, once it matures into
a release-quality offering.
