Don't you have some exotic type of index activated for uniqueMember (like substring)? The default value is only the equality index in dse.ldif
dn: cn=uniquemember,cn=default indexes,cn=config,cn=ldbm database,cn=plugins,c
n=config
objectClass: top
objectClass: nsIndex
cn: uniquemember
nsSystemIndex: false
nsIndexType: eq
In any case, batch write loads are quite particular. You could try to play with nsslapd-db-checkpoint-interval and nsslapd-db-durable-transaction config attributes while you run your batch uniqueMember modifications. You could also try disabling completely or limiting logging intensity (http://directory.fedoraproject.org/wiki/Named_Pipe_Log_Script) .
In my VM tests (RHEL5, 1 vCPU Xeon E5640 @ 2.67GHz, 1Gb mem, 389DS v1.2.10.6) with our production environment (~20k users, groups of ~6000 members created as in your case with perl scripts using ldapmodify running on the same VM) a group of 6000 uniqueMembers is created in 3 minutes 10 sec (190s) from scratch. Using "dstat" i see that the main problem is disk writes (transaction logs of db4):
----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read writ| recv send| in out | int csw
91 0 0 9 0 0| 0 40M| 120B 978B| 0 0 | 316 325
90 3 0 7 0 0| 0 40M| 60B 310B| 0 0 | 184 294
92 2 0 5 0 1| 0 40M| 60B 310B| 0 0 | 160 297
93 1 0 5 0 1| 0 40M| 60B 310B| 0 0 | 159 312
93 3 1 1 0 2| 0 24M| 60B 310B| 0 0 | 206 372
76 4 2 18 0 0| 0 40M| 60B 310B| 0 0 | 165 265
94 0 0 6 0 0| 0 40M| 60B 310B| 0 0 | 221 275
90 1 0 7 1 1| 0 41M| 60B 310B| 0 0 | 479 313
86 2 0 11 1 0| 0 40M| 60B 310B| 0 0 | 403 306
93 0 0 6 0 1| 0 20M| 120B 364B| 0 0 | 489 298
90 1 0 9 0 0| 0 40M| 60B 310B| 0 0 | 389 296
88 1 0 11 0 0| 0 40M| 60B 310B| 0 0 | 358 319
76 0 0 23 1 0| 0 41M| 60B 310B| 0 0 | 403 303
@+
Le 19 avril 2012 18:50, Russell Beall <beall@xxxxxxx> a écrit :
Thanks for the tips. I scanned the dse.ldif for those plugins and I found definitions for them all, but they all have nsslapd-pluginEnabled: off.There is something special about the uniquemember attribute that requires additional processing different from other attributes... Ldapmodify of other attributes runs pretty quick.Regards,Russ.On Apr 19, 2012, at 2:20 AM, Andrey Ivanov wrote:Hi Russel,Le 18 avril 2012 23:06, Russell Beall <beall@xxxxxxx> a écrit :On Apr 18, 2012, at 11:15 AM, Rich Megginson wrote:Yeah, this particular operation has not been optimized. I believe SunDS added explicit optimizations for this particular case.
It is becoming painfully apparent as I write more detailed tests. 389 takes time to add or delete uniquemember values proportionate to the number of values being operated on and is using about twice as much time to delete as it does to add. Sun DS appears to have perhaps an almost O(1) algorithm in play on both adding and deleting values.
Is there perhaps some kind of referential integrity setting that is being used and forcing some kind of lookup of each value, one that we could perhaps turn off? We wouldn't need such a check because our metadirectory process handles the integrity/consistency checking already.
There is memberOf plugin that maintains the memberOf attribute for groups. I don't know whether it is activated by default or not. You could try to disable it. There is also referential integrity plugin, attribute uniqueness plugin, maybe USN plugin or custom indexes that could consume a lot of CPU. Make sure you've disabled them if you don't need them.
@+--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/389-users
