|
On 02/28/2014 05:26 PM, Russell Beall
wrote:
Yes. Many rules are groupdn based, sometimes with multiple rules
where the dn must be in some groups but not in others. And some
of the groups have 50,000-100,000+ members.
Breaking out the groupdn checks into something very simple
did not change the performance at all. Changing all the
targetfilter checks to something simple very significantly
affected performance.
This has led me to finally discover the true bottleneck in
the indexing of one particular attribute. The attribute is a
custom attribute similar to memberOf. When converting from SJES
I took the syntax, which was DN syntax, and modified the
matching rule in the schema into distinguishedNameMatch. So the
index of this attribute was constructed based on that.
Apparently even though the values are DN values (almost all of
them), this particular index type must be incompatible. When I
switched from distinguishedNameMatch to caseExactMatch,
performance improved drastically (I'm now switching to
caseIgnoreMatch to see how that fares).
Are these values DNs or not? If they are DNs, they should use
distinguishedNameMatch, and it looks like the real problem is DN
normalization/comparison/indexing speed.
Note that if these values are really DNs, and you cannot control the
format that the DNs are sent by the clients, you may run into
problems using caseExactMatch.
Now we are on par with the old Sun servers and no longer
clogging up when many threads are running simultaneously.
Thank you so much, Rich and Ludwig, for helping me dig
through this!
Thanks,
Russ.
wrote:
It seems this means that the ACI
processing is not correctly using the indexes, or
I didn't create them properly.
Aci processing is done per entry, so
you already have looked up all the entries (which
could use some indexes) and now perform evaluation
on a single entry. If you have acis with groupdns
and you have very large groups the check if the
bound user is a member of th egroup can become a
severe performance problem
It could mean
1) the aci performance is not related to indexing
2) we don't know what indexes it is using
I have run db2index.pl and it successfully
reprocessed the entire set of indexes. I also
ran dbverify to successful completion. I can
tell that the indexes are being used in simple
searches (based on the fact that I get the
"Administrative Limit Exceeded" error when there
is no index).
Does this information point to anything that
I should look into further?
If the aci processing is doing internal searches
that don't show up in logconv.pl, then turning on
access logging for internal searches should show
those unindexed internal searches, which should show
up using logconv.pl
Thanks,
Russ.
On 02/27/2014
12:49 PM, Russell Beall wrote:
Hi Rich,
Thanks for the data. I've been
continuing to experiment and work on
this and especially making sure that
everything that might be used in the
ACIs is indexed. All the indexes
appear to be in order, but I am
confused by one thing… It looks like
there is no entryDN index and only an
entryrdn index.
Correct.
This new format will be
fully workable for complicated dn
lookups in the ACIs, correct? (We have
a lot of "groupdn=" and "userdn="
restrictions).
Correct. groupdn= and userdn= do not use
the entrydn index.
There is no one single ACI which
degrades performance, but I did notice
that when adding back in certain of
the ACIs, performance does degrade
quicker than should be expected just
for the cost of processing only one
additional ACI. I believe there may
definitely be a problem with the
indexes as you suggested but It is
hiding well...
You could enable access logging of
internal operations.
https://access.redhat.com/site/documentation/en-US/Red_Hat_Directory_Server/9.0/html/Configuration_Command_and_File_Reference/Core_Server_Configuration_Reference.html#cnconfig-nsslapd_accesslog_level
An additional symptom which may
point to a server configuration
problem is a strange inability to
import or reindex quickly. My small
dev VM can import several hundred
entries at a time, but this server
will only import or reindex at a rate
of 18-30 records per second. I've
ensured that there is plenty of import
memory as well as cachememsize which
should enable a very speedy import,
but even though all 32 cores are
burning bright, the import speed seems
incredibly slow. (This is of course
after all indexes are created and it
is trying to index while importing.
Import speed with no indexes is
fairly fast).
Any obvious clues I'm missing?
No, not sure what's going on.
Thanks,
Russ.
On
02/19/2014 04:56 PM, Russell
Beall wrote:
Hi all,
We've just set up
monster-sized server nodes
to run 389 as a replacement
to Sun DS. I've been
running my tests and I am
pleased to report that the
memory issue seems to be in
check with growth only up to
double the initial memory
usage after large quantities
of ldapmodify calls. We
have plenty of room in these
boxes to accommodate caching
the entire database.
The key blocker on this
is still the ACL processing
times for which I have been
unable to find a decent
resolution. We have 135
ACIs at the root of the
suffix. When I comment out
most of them but leave one
service account active,
processing times are very
nicely fast. When I leave
them all on, that same
service account takes 2.5
seconds to respond when only
one request is pending. A
new kink in the puzzle here
which is probably going to
be a deal breaker is that if
I run the same request on
multiple threads, each
thread takes proportionately
longer to respond depending
on the number of threads.
If I have 12 threads going
doing a simple lookup, each
thread responds in 45-55
seconds. If I have 24
threads going, each thread
takes 1m45s - 1m55s to
respond. The box has 32
cores available. While
processing, each thread is
burning 100% of an available
CPU thread for the entire
time. Theoretically when up
to 32 requests are
simultaneously processing,
each thread should return in
2.5 seconds just as if it
were one thread.
Note that the directory server
performance does not scale
linearly with the number of
cores. At some point you will
run into thread contention.
Also things like replication
compete for thread resources.
Since all threads are
burning 100% the entire
time, it doesn't seem like
that would be caused by
simple thread locking where
some threads are waiting for
others.
No, see below.
I'm thinking the system
is not properly configured
in some way and there is a
system bottleneck blocking
the processing. When
burning the CPU there is
very little percentage
allocated to the user
percentage, most of the CPU
usage is listed under the
system CPU usage. Is this
normal, or is this
indicative of some system
layer that is bottlenecking
the processing?
Sounds like the ACI may be doing
some sort of unindexed internal
search.
Have you narrowed it down to a
particular ACI that is causing
the problem?
Another question I posed
earlier is whether or not it
is possible to replicate
three subtrees independently
and then keep the aci entry
at the root suffix
independent so it can be set
separately for multiple
downstream replicants. That
way we could possibly
subdivide the service
accounts across different
nodes. Is that possible?
No.
Thanks,
Russ.
==============================
Russell
Beall
Systems
Programmer IV
Enterprise
Identity
Management
University of Southern
California
==============================
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
|
