Re: Experiences with Large Groups (>100k Members)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Thanks very much for your reply Trevor.

Just to expand a bit on my bit about sync’ing – We’ve been running on 389 DS for about 5 years now, and it has been solid for the most part.

Our LDAP cluster (multi-master replicated providers, hubs, and consumers) is sync’d to from our RDBMS based identity service via LSC (lsc-project.org).

LSC is normally trouble free, but it doesn’t do group replication very well due to it treating multi-value attributes in a monolithic manner (so we don’t do group sync yet).

 

Just wondering what others have encountered with large groups and syncing between LDAP <--> RDBMS / LDAP <--> LDAP.

 

Thanks,

Trev

 

From: "Wendt, Trevor" <Trevor.Wendt@xxxxxxxxxxxxxxxxxx>
Reply-To: "389-users@xxxxxxxxxxxxxxxxxxxxxxx" <389-users@xxxxxxxxxxxxxxxxxxxxxxx>
Date: Thursday, April 26, 2018 at 1:35 PM
To: "389-users@xxxxxxxxxxxxxxxxxxxxxxx" <389-users@xxxxxxxxxxxxxxxxxxxxxxx>
Subject: [389-users] Re: Experiences with Large Groups (>100k Members)

 

Trev, Was going to suggest splitting and batching your imports, good call.

We have had over 650k and not had any issues with 389ds. Started back on fedora-ds and migrated along with changes. Replication (master/master) is solid, keeps up with day to day changes fine, very stable. Overall no issues for 10+ years using it with steady growth. We are moving away to another solution but by not because of 389ds.  Good luck. -Trevor

 

 

From: Fong, Trevor [mailto:trevor.fong@xxxxxx]
Sent: Thursday, April 26, 2018 2:08 PM
To: General discussion list for the 389 Directory server project. <389-users@xxxxxxxxxxxxxxxxxxxxxxx>
Subject: [389-users] Re: Experiences with Large Groups (>100k Members)

 

Just an update:

I was successful in loading 532k members into a group, in ~45 mins, via ldapmodify by segmenting the ldif into 5 separate add:member sections, of ~100k each.  I also set nsslapd-db-locks in cn=config,cn=ldbm database,cn=plugins,cn=config to 400000 – not sure which made any difference.

 

Still interested in other people’s experience with large groups.

 

From: "Fong, Trevor" <trevor.fong@xxxxxx>
Reply-To: "389-users@xxxxxxxxxxxxxxxxxxxxxxx" <389-users@xxxxxxxxxxxxxxxxxxxxxxx>
Date: Thursday, April 26, 2018 at 11:06 AM
To: "389-users@xxxxxxxxxxxxxxxxxxxxxxx" <389-users@xxxxxxxxxxxxxxxxxxxxxxx>
Subject: [389-users] Experiences with Large Groups (>100k Members)

 

Hi Everyone,

 

I was wondering what experiences people have had with large groups (> 100k members) in 389 DS?

Particularly interested in loading, managing and syncing them.

WRT syncing – how do people efficiently sync large groups?  Most sync utilities sync at the attribute level; if the changed attribute (eg member) is multivalued, it just replaces all values.  That’s OK if there’s only a few values, but is not efficient when there are a large number of them.  A more efficient way would be to diff the 2 attributes and add/delete the differences; does anyone know of any sync tools that do something like this?

 

Background:

I have a few particularly large groups of > 500k members that are currently handled in a DBMS, but want to migrate them to LDAP instead.

When I try to load them via ldapmodify, doing an add:member per member was going to take more than 24 hrs at rate of processing at the time of abort.

Trying to add all members instead, with a single add:member and listing all members after that instruction, eventually ended with an Operations Error.  Turning on housekeeping error level showed it was getting “Lock table is out of available lock entries” error – I’m in the process of retrying with adjusted nsslapd-db-locks in cn=config,cn=ldbm database,cn=plugins,cn=config.

 

Thanks,

Trev

 

 

_________________________________________________

Trevor Fong

Senior Programmer Analyst

Information Technology | Engage. Envision. Enable.

The University of British Columbia

 

 



This electronic message transmission contains information from Black Hills Corporation, its affiliate or subsidiary, which may be confidential or privileged. The information is intended to be for the use of the individual or entity named above. If you are not the intended recipient, be aware the disclosure, copying, distribution or use of the contents of this information is prohibited. If you received this electronic transmission in error, please reply to sender immediately; then delete this message without copying it or further reading.

_______________________________________________
389-users mailing list -- 389-users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to 389-users-leave@xxxxxxxxxxxxxxxxxxxxxxx

[Index of Archives]     [Fedora User Discussion]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [Fedora News]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Maintainers]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Legacy]     [Fedora Desktop]     [Fedora Fonts]     [ATA RAID]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Centos]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora QA]     [Fedora Triage]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Tux]     [Yosemite News]     [Yosemite Photos]     [Linux Apps]     [Maemo Users]     [Gnome Users]     [KDE Users]     [Fedora Tools]     [Fedora Art]     [Fedora Docs]     [Maemo Users]     [Asterisk PBX]     [Fedora Sparc]     [Fedora Universal Network Connector]     [Fedora ARM]

  Powered by Linux