Re: virtuoso-t memory consumption

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Alexander Puchmayr posted on Sat, 04 May 2013 14:24:58 +0200 as excerpted:

> Hi there,
> 
> How much memory is virtuoso supposed to use? If I got it right, it shall
> use no more than the setting in System Settings/Desktop Search/Extended
> Settings/Memory usage, which is in my case 128MB.
> 
> Now look at the real memory consumtion (from htop)
> 
> 
>   PID USER      PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
> 18831 alex       39  19 2633M 1900M  4440 S  0.0 48.1 17:55.39
> /usr/bin/virtuoso-t +foreground +configfile /tmp/virtuoso_T18826.ini
> +wait
> 
> It uses a virtual range of more than 2.5GB  and keeps nearly 2GB locked
> in memory (which is half of the available RAM). Is this *really*
> necessary? What is virtuoso doing with all that memory and why is the
> setting in System Settings obviously ignored? And the most important
> question is, how to make virtuoso use a reasonable amount of memory?
> 
> BTW: I'm using kde-4.10.1 and virtuoso-server-6.1.6 on a gentoo system
> (amd64)

Hello fellow gentooer! =:^)

FWIW, I had enough of semantic-desktop including virtuoso/nepomuk/strigi/
akonadi/redland/rasqal/etc. here, and turned off the whole thing.  

USE="-semantic-desktop -virtuoso -redland" ...

Of course that means nothing kdepim related, which means no kmail, but 
the new akonadi-based kmail wasn't stable here anyway, and I ended up 
switching (after 9 years, since kde2 era!) to claws-mail.  FWIW that's 
what I use (with its feed-reader plugin) in place of akregator, as well.  
The conversion wasn't exactly easy, but I've been very happy with the 
results! =:^)

So I'm definitely not the most unbiased source of virtuoso information 
you can find, but still...

If you google Linux memory, you'll quickly find that it's a very common 
source of confusion but that the situation isn't as simple or usage as 
bad as it may initially appear.

In particular, the virtual memory number is "virtually" (heh!) useless to 
all but the most technical user, as it's EXTREMELY common for 
applications to request huge amounts of memory from the allocator that 
they never even touch at all.  In practice, many apps allocate for worst-
case usage or even more, and actually use hardly any of what they 
requested, at all.  This practice is not just allowed, but arguably 
encouraged by the kernel itself, which has a default over-commit ratio 
that allows way more memory to be actually allocated than really exists, 
either as physical RAM or as swap, on the system.  Google for the OOM-
killer (OOM=out-of-memory) to see what happens when too many apps try to 
actually use the memory they've allocated and the system really does run 
out of memory.

So unless you have a very specific reason to track virtual memory, 
don't.  Simply pretend that figure doesn't exist, preferably by hiding 
that column in htop entirely -- there's far more practical uses to which 
that available display space can be put. =:^)  (Of course if you're 
curious just /how/ much memory an app is requesting, you can leave the 
column visible.  Just know what the number in it actually counts for, not 
much.)

(I could suggest as a replacement, the "CODE" column, the text-segment or 
code size of the process, or possibly the DATA column, the data segment 
plus stack usage of the process, which I suspect would prove to be the 
outlier here.  Column information from the htop manpage.)

The RES aka RSS, resident set size (and the related MEM% column, which is 
based on RES), is a somewhat more useful number, but even then, it's not 
what it might first appear.  In particular, killing that process won't 
normally get you back the memory reported as RSS, nor will starting a new 
process require the memory RSS reports it as using once started.  This is 
because a lot of that memory usage is shared with other processes.  One 
subset of that shared memory (shared library usage) is accounted for in 
the shared column, but particularly for X-based apps, there's a whole lot 
more than that going on under the hood.  As just one example, consider 
the number of apps using common fonts -- these must be kept in memory, 
but it's a shared resource that AFAIK doesn't show up in the shr column, 
as they're not shared libraries, but other shared resources.

That said, RSS does demonstrate that the 128 MB limit you mention 
obviously isn't a total memory usage limit.  Take this bit with a rather 
large grain of salt as it's simply a guess, but I BELIEVE that limit 
refers to the actual processed-and-cached (aka "digested") data size -- 
the size of the amount of data as actually stored in the database, that 
it's allowed to keep cached in memory.  But the actual runtime memory 
usage will be far higher, not only because the app has to have all the 
code to process that information (which I suspect is actually relatively 
small, check CODE, LIB, SHR), but because the information itself will 
need to be scanned in from the various files in ordered to be processed 
and indexed in the first place -- THAT's what I suspect to be the 
majority of the usage.  If so, it should show up in htop's DATA column, 
if you enable that.  Which is why I said above that this would probably 
the interesting outlier column.


All that said, nearly 2 GB of RSS, while /possibly/ /arguable/ /as/ 
reasonable for an indexer type app, is somewhat excessive compared to 
most apps, and could EASILY be considered excessive to the point of not 
being worth the hassle.


As a point of comparison, it's worth seeing my top users here, with 
semantic-desktop disabled.  (MEM% is far lower here as this system is a 
16 gigs physical RAM system, while based on your figures, yours is 
probably 4 gig RAM.  As you can see I have a few more columns enabled, 
too, including PGRP and SESN pid columns for reasons unrelated to the 
current discussion, but VIRT isn't one of them.)

  PID  PPID  PGRP  SESN USER      PRI  NI S   RES CPU% MEM%    IO   TIME
+  CPU NLWP Command

 2639  2533  2521  2521 x          20   0 S  120M  0.0  0.8     0  
2:25.17   3    5 /usr/bin/pan

 2554     1  2521  2521 x          20   0 S  118M  0.0  0.7     0  
0:14.42   0    4 kdeinit4: plasma-desktop [kdeinit]

 2474  2473  2474  2474 root       20   0 S 97408  3.0  0.6     0  
8:19.18   4    2 /usr/bin/X -nolisten tcp :0 -auth /h/x/.serverauth.2460

 2537  2533  2521  2521 x          20   0 S 67148  4.0  0.4     0  
9:37.07   0    4 kwin

 2545     1  2521  2521 x          20   0 S 59864  0.0  0.4     0  
0:48.59   4    3 /usr/bin/knotify4

 2552     1  2521  2521 x          20   0 S 50368  0.0  0.3     0  
0:07.02   5   15 kdeinit4: krunner [kdeinit]


As you can see, pan (the news client from which I am posting this, via 
gmane.org's list2news gateway) is my biggest memory hog here, 120 MB RES/
RSS, with plasma-desktop following at 118 MB, and X behind it and 
rounding out the top three at under 100 MB.  (Off topic but possibly of 
interest none-the-less, you might also notice my non-standard /h instead 
of /home, and the simple "x" username...)

120 MB top RSS is quite a contrast to your near 2 GB virtuoso RSS... and 
I've a 16-gig machine to play with, too, while it looks like you're 
running 4-gig.  Yeah, nearly half of that, almost 2-gig, tied up by 
virtuoso DOES seem unreasonably bloated, caveats about interpretation of 
RSS or no caveats!  Get rid of it and you SHOULD see a difference!

Which is pretty much what I decided when I decided to kill the entire 
semantic-desktop thing, not just at runtime, but being a gentooer with 
the tools to do so thus exposed, at build-time, setting USE=-semantic-
desktop and toggling off related USE flags as well.

There is some functionality cost to that choice, however.  In addition to 
killing everything akonadi and kdepim related including kmail, dolphin/
konqueror lose their semantic-desktop-required metadata functionality, 
including the information found in the info tab of the file properties 
dialog.  (The tab remains, but it's always blank.)

But the tradeoff was WELL worth it IMO as KDE was **MUCH** faster 
afterward, even tho I had as much as possible runtime-disabled before I 
toggled off USE=semantic-desktop and friends, so I wasn't even comparing 
to the full runtime case, but to the mostly disabled case.  To that point 
(I disabled semantic-desktop in early kde 4.7, with it fully disabled by 
4.7.1) I had always stated that kde4 until late 4.5 was pre-release 
quality; that only with 4.5 was a properly release-quality kde4 available 
to release a similar quality late kde 3.5.  So while I considered say 
4.5.4+ roughly EQUAL to say 3.5.8+ (with the exception of a couple 4.6 
releases that were very buggy and regressed), it was only when I turned 
off semantic-desktop and related options at BUILD time that I really 
honestly could say that the kde4 experience was finally BETTER than late 
3.5.  Which is ironic, given that one of the big bullet points of kde4 
was the whole semantic-desktop thing.  But only by build-time disabling 
semantic-desktop did I finally get a kde4 that not just matched in 
experience for me, but finally surpassed, kde3.  Runtime disabling 
helped, and given that 48% memory figure it'd probably help you even 
more, but it wasn't until I exterminated it to the extent possible at 
build-time, that I could REALLY say kde4 was now not only equal to kde3 
in experience for me, but actually BETTER.

YMMV as they say, but that's what I found, here. =:^)

I'm just glad that I'm a gentooer and thus have that choice exposed to 
me.  Until the very recent "kde-lite" efforts I've seen coverage of in 
the last few weeks, I knew of no binary-based distro offering the same 
"low-fat-kde" experience including reduced or killed semantic-desktop, 
that gentoo makes possible with its USE flags.  It'll be interesting to 
watch that low-fat-kde project develop, and see if it gets any decent 
user uptake and whether other distros follow suite, once it matures into 
a release-quality offering.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

___________________________________________________
This message is from the kde mailing list.
Account management:  https://mail.kde.org/mailman/listinfo/kde.
Archives: http://lists.kde.org/.
More info: http://www.kde.org/faq.html.




[Index of Archives]     [Trinity (TDE) Desktop Users]     [Fedora KDE]     [Fedora Desktop]     [Linux Kernel]     [Gimp]     [GIMP for Windows]     [Gnome]     [Yosemite Hiking]
  Powered by Linux