Re: Initialization with fds 1.2.0 and 389-ds 1.2.10 failure

[Rich Megginson <rmeggins@xxxxxxxxxx>]

On 04/11/2013 07:54 AM, carne_de_passaro wrote:

Replica config from FDS 1.2.0 (Supplier):

dn: cn=mapping tree,cn=config

objectClass: top

objectClass: extensibleObject

cn: mapping tree

dn: cn="dc=foo,dc=gov,dc=br",cn=mapping tree, cn=config

objectClass: top

objectClass: extensibleObject

objectClass: nsMappingTree

nsslapd-state: Backend

cn: "dc=foo,dc=gov,dc=br"

nsslapd-parent-suffix: "dc=gov,dc=br"

nsslapd-backend: foo

dn: cn="dc=gov,dc=br",cn=mapping tree, cn=config

objectClass: top

objectClass: extensibleObject

objectClass: nsMappingTree

nsslapd-state: Backend

cn: "dc=gov,dc=br"

nsslapd-backend: raiz

dn: cn="o=netscaperoot", cn=mapping tree, cn=config

nsslapd-state: backend

objectClass: top

objectClass: extensibleObject

objectClass: nsMappingTree

cn: "o=netscaperoot"

nsslapd-backend: NetscapeRoot

dn: cn=replica,cn="dc=foo,dc=gov,dc=br",cn=mapping tree, cn=config

objectClass: nsDS5Replica

objectClass: top

nsDS5ReplicaRoot: dc=foo,dc=gov,dc=br

nsDS5ReplicaType: 3

nsDS5Flags: 1

nsDS5ReplicaId: 4

nsds5ReplicaPurgeDelay: 604800

nsDS5ReplicaBindDN: uid=replicador,cn=config

cn: replica

nsState:: BAAAAKB8ZVEAAAAAAAAAAAIAAAA=

nsDS5ReplicaName: b8856102-1dd111b2-a0e2d329-568c0000

nsds5ReplicaChangeCount: 0

nsds5replicareapactive: 0

dn: cn=replica,cn="dc=gov,dc=br",cn=mapping tree, cn=config

objectClass: nsDS5Replica

objectClass: top

nsDS5ReplicaRoot: dc=gov,dc=br

nsDS5ReplicaType: 3

nsDS5Flags: 1

nsDS5ReplicaId: 3

nsds5ReplicaPurgeDelay: 604800

nsDS5ReplicaBindDN: uid=replicador,cn=config

cn: replica

nsState:: AwAAAKB8ZVEAAAAAAAAAAAEAAAA=

nsDS5ReplicaName: 70fed503-1dd211b2-a0e1d329-568c0000

nsds5ReplicaChangeCount: 0

nsds5replicareapactive: 0

dn: cn=foo-389ds-lab, cn=replica, cn="dc=foo,dc=gov,dc=br", cn=mapping t

 ree, cn=config

objectClass: top

objectClass: nsDS5ReplicationAgreement

description: lab-master -> 389ds-lab

cn: foo-389ds-lab

nsDS5ReplicaRoot: dc=foo,dc=gov,dc=br

nsDS5ReplicaHost: 389ds-lab.xxx.foo

nsDS5ReplicaPort: 636

nsDS5ReplicaBindDN: uid=replicador,cn=config

nsDS5ReplicaTransportInfo: SSL

nsDS5ReplicaBindMethod: SIMPLE

nsDS5ReplicaCredentials: {DES}l/OdqrRRrvxAR6wIJKgLDA==

nsds50ruv: {replicageneration} 5162dc1a000000080000

nsds50ruv: {replica 8 ldap://389ds-lab.xxx.foo:389}

nsds50ruv: {replica 4 ldap://lab-master:389}

nsruvReplicaLastModified: {replica 8 ldap://389ds-lab.xxx.foo:389} 00000000

nsruvReplicaLastModified: {replica 4 ldap://lab-master:389} 00000000

nsds5replicareapactive: 0

nsds5replicaLastUpdateStart: 0

nsds5replicaLastUpdateEnd: 0

nsds5replicaChangesSentSinceStartup:

nsds5replicaLastUpdateStatus: 0 No replication sessions started since server s

 tartup

nsds5replicaUpdateInProgress: FALSE

nsds5replicaLastInitStart: 20130410212832Z

nsds5replicaLastInitEnd: 20130410212952Z

nsds5replicaLastInitStatus: -2 Total update abortedSystem error

dn: cn=foo-master-xxx-reg, cn=replica, cn="dc=foo,dc=gov,dc=br", cn=mapp

 ing tree, cn=config

objectClass: top

objectClass: nsDS5ReplicationAgreement

description: lab-master -> master-xxx-reg

cn: foo-master-xxx-reg

nsDS5ReplicaRoot: dc=foo,dc=gov,dc=br

nsDS5ReplicaHost: master-xxx-reg.xxx.foo

nsDS5ReplicaPort: 636

nsDS5ReplicaBindDN: uid=replicador,cn=config

nsDS5ReplicaTransportInfo: SSL

nsDS5ReplicaBindMethod: SIMPLE

nsDS5ReplicaCredentials: {DES}l/OdqrRRrvxAR6wIJKgLDA==

nsds50ruv: {replicageneration} 51549725000000020000

nsds50ruv: {replica 2 ldap://master-xxx-reg.xxx.foo:389}

nsds50ruv: {replica 4 ldap://lab-master:389}

nsruvReplicaLastModified: {replica 2 ldap://master-xxx-reg.xxx.foo:389} 000

 00000

nsruvReplicaLastModified: {replica 4 ldap://lab-master:389} 00000000

nsds5replicareapactive: 0

nsds5replicaLastUpdateStart: 0

nsds5replicaLastUpdateEnd: 0

nsds5replicaChangesSentSinceStartup:

nsds5replicaLastUpdateStatus: 0 No replication sessions started since server s

 tartup

nsds5replicaUpdateInProgress: FALSE

nsds5replicaLastInitStart: 20130410202603Z

nsds5replicaLastInitEnd: 20130410202817Z

nsds5replicaLastInitStatus: -2 Total update abortedSystem error

dn: cn=raiz-389ds-lab, cn=replica, cn="dc=gov,dc=br", cn=mapping tree, cn=conf

 ig

objectClass: top

objectClass: nsDS5ReplicationAgreement

description: lab-master -> 389ds-lab

cn: raiz-389ds-lab

nsDS5ReplicaRoot: dc=gov,dc=br

nsDS5ReplicaHost: 389ds-lab.xxx.foo

nsDS5ReplicaPort: 636

nsDS5ReplicaBindDN: uid=replicador,cn=config

nsDS5ReplicaTransportInfo: SSL

nsDS5ReplicaBindMethod: SIMPLE

nsDS5ReplicaCredentials: {DES}l/OdqrRRasdfasdfasdf==

nsds50ruv: {replicageneration} 51535e48000000030000

nsds50ruv: {replica 5 ldap://389ds-lab.xxx.foo:389}

nsds50ruv: {replica 3 ldap://lab-master:389} 515b2f48000000030000 515b2f480000

 00030000

nsruvReplicaLastModified: {replica 5 ldap://389ds-lab.xxx.foo:389} 00000000

nsruvReplicaLastModified: {replica 3 ldap://lab-master:389} 00000000

nsds5replicareapactive: 0

nsds5replicaLastUpdateStart: 0

nsds5replicaLastUpdateEnd: 0

nsds5replicaChangesSentSinceStartup:

nsds5replicaLastUpdateStatus: 0 No replication sessions started since server s

 tartup

nsds5replicaUpdateInProgress: FALSE

nsds5replicaLastInitStart: 20130410205925Z

nsds5replicaLastInitEnd: 20130410205928Z

nsds5replicaLastInitStatus: 0 Total update succeeded

dn: cn=raiz-master-xxx-reg, cn=replica, cn="dc=gov,dc=br", cn=mapping tree, cn

 =config

objectClass: top

objectClass: nsDS5ReplicationAgreement

description: lab-master -> master-xxx-reg

cn: raiz-master-xxx-reg

nsDS5ReplicaRoot: dc=gov,dc=br

nsDS5ReplicaHost: master-xxx-reg.xxx.foo

nsDS5ReplicaPort: 636

nsDS5ReplicaBindDN: uid=replicador,cn=config

nsDS5ReplicaTransportInfo: SSL

nsDS5ReplicaBindMethod: SIMPLE

nsDS5ReplicaCredentials: {DES}l/OdqrRRrvxAR6wasdfasdfasdf==

nsds50ruv: {replicageneration} 515f2cca000000030000

nsds50ruv: {replica 1 ldap://master-xxx-reg.xxx.foo:389}

nsds50ruv: {replica 3 ldap://lab-master:389}

nsruvReplicaLastModified: {replica 1 ldap://master-xxx-reg.xxx.foo:389} 000

 00000

nsruvReplicaLastModified: {replica 3 ldap://lab-master:389} 00000000

nsds5replicareapactive: 0

nsds5replicaLastUpdateStart: 0

nsds5replicaLastUpdateEnd: 0

nsds5replicaChangesSentSinceStartup:

nsds5replicaLastUpdateStatus: 0 No replication sessions started since server s

 tartup

nsds5replicaUpdateInProgress: FALSE

nsds5replicaLastInitStart: 0

nsds5replicaLastInitEnd: 0

Replica config from 389ds 1.3.0.2 (Consumer in multi-master replication):

dn: cn=mapping tree,cn=config

objectClass: top

objectClass: extensibleObject

cn: mapping tree

dn: cn=dc\3Dgov\2Cdc\3Dbr,cn=mapping tree,cn=config

objectClass: top

objectClass: extensibleObject

objectClass: nsMappingTree

nsslapd-state: referral on update

cn: dc=gov,dc=br

nsslapd-backend: raiz

nsslapd-referral: ldap://lab-master:389/dc%3Dgov%2Cdc%3Dbr

dn: cn=dc\3Dfoo\2Cdc\3Dgov\2Cdc\3Dbr,cn=mapping tree,cn=config

objectClass: top

objectClass: extensibleObject

objectClass: nsMappingTree

nsslapd-state: referral on update

cn: dc=foo,dc=gov,dc=br

nsslapd-parent-suffix: dc=gov,dc=br

nsslapd-backend: foo

nsslapd-referral: ldap://lab-master:389/dc%3Dfoo%2Cdc%3Dgov%2Cdc%3Dbr

dn: cn=o\3Dnetscaperoot,cn=mapping tree,cn=config

nsslapd-state: backend

objectClass: top

objectClass: extensibleObject

objectClass: nsMappingTree

cn: "o=netscaperoot"

cn: o=netscaperoot

nsslapd-backend: NetscapeRoot

dn: cn=replica,cn=dc\3Dgov\2Cdc\3Dbr,cn=mapping tree,cn=config

objectClass: nsDS5Replica

objectClass: top

nsDS5ReplicaRoot: dc=gov,dc=br

nsDS5ReplicaType: 2

nsDS5Flags: 0

nsds5ReplicaPurgeDelay: 604800

nsDS5ReplicaBindDN: uid=replicador,cn=config

cn: replica

nsDS5ReplicaId: 65535

nsState:: //8AAAAAAACw0mVRAAAAAAAAAAAAAAAAAQAAAAAAAAABAAAAAAAAAA==

nsDS5ReplicaName: 023279bc-a20311e2-821ec0df-d5dbae74

nsds5ReplicaChangeCount: 0

nsds5replicareapactive: 0

dn: cn=replica,cn=dc\3Dfoo\2Cdc\3Dgov\2Cdc\3Dbr,cn=mapping tree,cn=config

objectClass: nsDS5Replica

objectClass: top

nsDS5ReplicaRoot: dc=foo,dc=gov,dc=br

nsDS5ReplicaType: 2

nsDS5Flags: 0

nsds5ReplicaPurgeDelay: 604800

nsDS5ReplicaBindDN: uid=replicador,cn=config

cn: replica

nsDS5ReplicaId: 65535

nsState:: //8AAAAAAACB2WVRAAAAAAAAAAAAAAAAAQAAAAAAAAAAAAAAAAAAAA==

nsDS5ReplicaName: 25f5bf02-a20311e2-821ec0df-d5dbae74

nsds5ReplicaChangeCount: 0

nsds5replicareapactive: 0

And finally, the replica config from 389ds 1.2.11.15 (Consumer  in a multi-master replication. I updated the 389ds in this server yesterday): 

dn: cn=mapping tree,cn=config

objectClass: top

objectClass: extensibleObject

cn: mapping tree

dn: cn=dc\3Dgov\2Cdc\3Dbr,cn=mapping tree,cn=config

objectClass: top

objectClass: extensibleObject

objectClass: nsMappingTree

nsslapd-state: backend

cn: dc=gov,dc=br

nsslapd-backend: raiz

nsslapd-referral: ldap://lab-master:389/dc%3Dgov%2Cdc%3Dbr

dn: cn=dc\3Dfoo\2Cdc\3Dgov\2Cdc\3Dbr,cn=mapping tree,cn=config

objectClass: top

objectClass: extensibleObject

objectClass: nsMappingTree

nsslapd-state: backend

cn: dc=serpro,dc=gov,dc=br

nsslapd-parent-suffix: dc=gov,dc=br

nsslapd-backend: foo

dn: cn=o\3Dnetscaperoot,cn=mapping tree,cn=config

nsslapd-state: backend

objectClass: top

objectClass: extensibleObject

objectClass: nsMappingTree

cn: "o=netscaperoot"

cn: o=netscaperoot

nsslapd-backend: NetscapeRoot

dn: cn=replica,cn=dc\3Dgov\2Cdc\3Dbr,cn=mapping tree,cn=config

objectClass: nsDS5Replica

objectClass: top

nsDS5ReplicaRoot: dc=gov,dc=br

nsDS5ReplicaType: 3

nsDS5Flags: 1

nsDS5ReplicaId: 1

nsds5ReplicaPurgeDelay: 604800

nsDS5ReplicaBindDN: uid=replicador,cn=config

cn: replica

nsState:: AQAAAAAAAABOu2ZRAAAAAAAAAAAAAAAAAQAAAAAAAAABAAAAAAAAAA==

nsDS5ReplicaName: 588eba82-972111e2-9a86dff3-de3c5ff6

nsds5ReplicaChangeCount: 0

nsds5replicareapactive: 0

dn: cn=replica,cn=dc\3Dfoo\2Cdc\3Dgov\2Cdc\3Dbr,cn=mapping tree,cn=config

objectClass: nsDS5Replica

objectClass: top

nsDS5ReplicaRoot: dc=foo,dc=gov,dc=br

nsDS5ReplicaType: 3

nsDS5Flags: 1

nsDS5ReplicaId: 2

nsds5ReplicaPurgeDelay: 604800

nsDS5ReplicaBindDN: uid=replicador,cn=config

cn: replica

nsState:: AgAAAAAAAABOu2ZRAAAAAAAAAAAAAAAAAQAAAAAAAAABAAAAAAAAAA==

nsDS5ReplicaName: 588eba84-972111e2-9a86dff3-de3c5ff6

nsds5ReplicaChangeCount: 0

nsds5replicareapactive: 0

Is that configuration that you wanted to see?

Yes.  In all cases, the supplier is the FDS 1.2.0?  It appears there could be a problem with one of its entries:
[09/Apr/2013:11:02:43 -0300] - slapi_start_bulk_import: bulk import is not supported by this (default) backend
[09/Apr/2013:11:02:46 -0300] NSMMReplicationPlugin - Error -12: could not import entry dn (null) for total update operation conn=2 op=43381
[09/Apr/2013:11:02:46 -0300] - ERROR bulk import abandoned

It would appear the DN is corrupted.  We might be able to find it.  The entries are sent in roughly entry ID order.  So assuming the first entry with ID 1 is sent in op=4, we should be looking for an entry with ID 43381-4 = 43377

On the supplier, do this

dbscan -K 43377 -f /var/lib/dirsrv/slapd-INST/db/dbname/id2entry.db4

If that one looks ok, then try going up or down from 43377

2013/4/10 Rich Megginson <rmeggins@xxxxxxxxxx>

On 04/10/2013 03:38 PM, carne_de_passaro wrote:

Both consumer.

Can you provide your cn=replica config entry from both consumer and supplier, and the replication agreement entry from your supplier?  Please first obscure any sensitive information.

2013/4/10 Rich Megginson <rmeggins@xxxxxxxxxx>

On 04/10/2013 03:33 PM, carne_de_passaro wrote:

I guess that is not exactly the same problem because I test with a 1.2.11.15

supplier or consumer?

and a 1.3.0.2

supplier or consumer?

(compiled) both in CentOS 6.3 and the the error persist. These two machine is on the same network, there is no firewall between them. I can't figure what could the supplier be disconnected.

2013/4/10 Rich Megginson <rmeggins@xxxxxxxxxx>

On 04/09/2013 08:33 AM, carne_de_passaro wrote:

Hello folks,

I have a test environment with a FDS version 1.2.0 on a Debian 5.0 x86 and a 389-ds 1.2.10.12-1.el6.x86_64 on a CentOS 6.3 x86_64.

I have two suffix on the FDS, a root suffix and a subsuffix. I've configured two replication agreements, one for suffix. When I'm try to initialize the root suffix, which contains a few objects, it's works just fine, but when I try to initialize the subsuffix, which contains about 90.000 objects, it's fails and give me the error "Total update aborted. System error. Error code -2" on the FDS console.

Looking at the 389-ds error log file I gave this:

[09/Apr/2013:11:01:45 -0300] NSMMReplicationPlugin - multimaster_be_state_change: replica dc=foo,dc=gov,dc=br is going offline; disabling replication

[09/Apr/2013:11:01:45 -0300] NSMMReplicationPlugin - conn=0 op=0 repl="dc=foo,dc=gov,dc=br": Replica in use locking_purl=conn=2 id=3

[09/Apr/2013:11:01:45 -0300] NSMMReplicationPlugin - replica_disable_replication: replica dc=foo,dc=gov,dc=br is acquired

[09/Apr/2013:11:01:45 -0300] - WARNING: Import is running with nsslapd-db-private-import-mem on; No other process is allowed to access the database

[09/Apr/2013:11:01:45 -0300] NSMMReplicationPlugin - conn=2 op=3 repl="dc=foo,dc=gov,dc=br": StartNSDS50ReplicationRequest: response=0 rc=0

[09/Apr/2013:11:02:06 -0300] - import foo: Processed 17727 entries -- average rate 886.4/sec, recent rate 886.3/sec, hit ratio 0%

[09/Apr/2013:11:02:29 -0300] - import foo: Processed 32924 entries -- average rate 765.7/sec, recent rate 765.7/sec, hit ratio 98%

[09/Apr/2013:11:02:43 -0300] - slapi_start_bulk_import: bulk import is not supported by this (default) backend

[09/Apr/2013:11:02:46 -0300] NSMMReplicationPlugin - Error -12: could not import entry dn (null) for total update operation conn=2 op=43381

[09/Apr/2013:11:02:46 -0300] - ERROR bulk import abandoned

[09/Apr/2013:11:02:46 -0300] - import foo: Aborting all Import threads...

[09/Apr/2013:11:02:51 -0300] - import foo: Import threads aborted.

[09/Apr/2013:11:02:53 -0300] - import foo: Closing files...

[09/Apr/2013:11:02:57 -0300] - libdb: foo/nsuniqueid.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/objectclass.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/cn.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/mailAlternateAddress.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/uniquemember.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/telephoneNumber.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/parentid.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/mail.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/sn.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/givenName.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/entryrdn.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:57 -0300] - libdb: foo/uid.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:58 -0300] - libdb: foo/id2entry.db4: unable to flush: No such file or directory

[09/Apr/2013:11:02:58 -0300] - import foo: Import failed.

[09/Apr/2013:11:02:58 -0300] NSMMReplicationPlugin - Aborting total update in progress for replicated area dc=foo,dc=gov,dc=br connid=2

[09/Apr/2013:11:02:58 -0300] - process_bulk_import_op: NULL target sdn

[09/Apr/2013:11:02:58 -0300] NSMMReplicationPlugin - conn=2 op=-1 repl="dc=foo,dc=gov,dc=br": Released replica

This looks like https://fedorahosted.org/389/ticket/374 which was fixed in 1.2.11

and at the end of the 389-ds access log I gave this:

[09/Apr/2013:11:02:43 -0300] conn=2 op=43380 EXT oid="2.16.840.1.113730.3.5.6" name="Netscape Replication Total Update Entry"

[09/Apr/2013:11:02:43 -0300] conn=2 op=43380 RESULT err=0 tag=120 nentries=0 etime=0

[09/Apr/2013:11:02:43 -0300] conn=2 op=43381 EXT oid="2.16.840.1.113730.3.5.6" name="Netscape Replication Total Update Entry"

[09/Apr/2013:11:02:46 -0300] conn=2 op=-1 fd=65 closed - B4

What does means that error B4 ?

B4 means the supplier disconnected while the consumer was attempting to send back a reply.

Thanks in advance,

Danilo

--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users

--
389 users mailing list
389-users@xxxxxxxxxxxxxxxxxxxxxxx
https://admin.fedoraproject.org/mailman/listinfo/389-users