Eddie C wrote: > >>Any errors? > No as far as I can tell the import goes smooth. Our upstream > applications seem to have no problem with the data it seems to be all > imported. > > I am really troubleshooting and index and performance issues, I figure > out of order keys could be slowing searches down. This a fairly large > db 160000 entires) some objects have upwards of 20 attributes. > > db_verify and db_2index both run fairly quickly. It just seems like > they cant both agree on what clean data looks like. > > FWI I upgraded to the latest 1.0.4 before all this testing. Just to confirm, was the server running when you ran db_verify? > > [18/Dec/2006:15:29:50 -0500] - WARNING: Import is running with > nsslapd-db-private-import-mem on; No other process is allowed to > access the database > [18/Dec/2006:15:29:50 -0500] - import idsk_services: Beginning import > job... > [18/Dec/2006:15:29:50 -0500] - import idsk_services: Index buffering > enabled with bucket size 15 > [18/Dec/2006:15:29:50 -0500] - import idsk_services: Processing file > "/opt/idsk/downloads/idsk.services.com.ldif" > [18/Dec/2006:15:29:50 -0500] - import idsk_services: Finished scanning > file "/opt/idsk/downloads/idsk.services.com.ldif" (2037 entries) > [18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers finished; > cleaning up... > [18/Dec/2006:15:29:51 -0500] - import idsk_services: Workers cleaned up. > [18/Dec/2006:15:29:51 -0500] - import idsk_services: Cleaning up > producer thread... > [18/Dec/2006:15:29:51 -0500] - import idsk_services: Indexing > complete. Post-processing... > [18/Dec/2006:15:29:51 -0500] - import idsk_services: Flushing caches... > [18/Dec/2006:15:29:51 -0500] - import idsk_services: Closing files... > [18/Dec/2006:15:29:51 -0500] - import idsk_services: Import complete. > Processed 2037 entries in 1 seconds. ( 2037.00 entries/sec) > [18/Dec/2006:15:30:28 -0500] - Bringing idsk_data offline... > [18/Dec/2006:15:30:28 -0500] - WARNING: Import is running with > nsslapd-db-private-import-mem on; No other process is allowed to > access the database > [18/Dec/2006:15:30:28 -0500] - import idsk_data: Beginning import job... > [18/Dec/2006:15:30:28 -0500] - import idsk_data: Index buffering > enabled with bucket size 15 > [18/Dec/2006:15:30:28 -0500] - import idsk_data: Processing file > "/opt/idsk/downloads/idsk.com.ldif" > [18/Dec/2006:15:30:48 -0500] - import idsk_data: Processed 77288 > entries -- average rate 3680.4/sec, recent rate 3680.3/sec, hit ratio 0% > [18/Dec/2006:15:31:13 -0500] - import idsk_data: Processed 141956 > entries -- average rate 3086.0/sec, recent rate 3086.0/sec, hit ratio 100% > [18/Dec/2006:15:31:35 -0500] - import idsk_data: Finished scanning > file "/opt/idsk/downloads/idsk.com.ldif" (163684 entries) > [18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers finished; > cleaning up... > [18/Dec/2006:15:31:49 -0500] - import idsk_data: Workers cleaned up. > [18/Dec/2006:15:31:49 -0500] - import idsk_data: Cleaning up producer > thread... > [18/Dec/2006:15:31:49 -0500] - import idsk_data: Indexing complete. > Post-processing... > [18/Dec/2006:15:32:20 -0500] - import idsk_data: Flushing caches... > [18/Dec/2006:15:32:23 -0500] - import idsk_data: Closing files... > [18/Dec/2006:15:32:24 -0500] - import idsk_data: Import complete. > Processed 163684 entries in 116 seconds. ( 1411.07 entries/sec) > > Edward > > On 12/18/06, *Noriko Hosoi* <nhosoi at redhat.com > <mailto:nhosoi at redhat.com>> wrote: > > Eddie C wrote: > > All, > > > > I tested this from scratch. > > I used the ldif_2db function which did work much faster! 112 > > seconds...rather then about 20 minutes. > > However I think the verify_db and db2_index functions are not in > > agreement. > > > > Create indexes. > > After my initial import. > > [root at ldap3 slapd-ldap3]# more out.after.final.txt > > ***************************************************************** > > verify-db: This tool should only be run if recovery start fails > > and the server is down. If you run this tool while the server is > > running, you may get false reports of corrupted files or other > > false errors. > > ***************************************************************** > > Verify log files in db ... Good > > Verify db/o_idsk_com/id2entry.db4 ... Good > > Verify db/userRoot/ancestorid.db4 ... Good > > Verify db/userRoot/entrydn.db4 ... Good > > Verify db/userRoot/cn.db4 ... Good > > Verify db/userRoot/numsubordinates.db4 ... Good > > Verify db/userRoot/aci.db4 ... Good > > Verify db/userRoot/parentid.db4 ... Good > > Verify db/userRoot/objectclass.db4 ... Good > > Verify db/userRoot/id2entry.db4 ... Good > > Verify db/userRoot/nsUniqueId.db4 ... Good > > Verify db/idsk_services/ancestorid.db4 ... > > DB ERROR: db_verify: Page 4: out-of-order key at entry 252 > > DB ERROR: db_verify: Page 7: out-of-order key at entry 194 > > DB ERROR: db_verify: Page 7: out-of-order key at entry 450 > > DB ERROR: db_verify: Page 11: out-of-order key at entry 69 > > DB ERROR: db_verify: Page 11: out-of-order key at entry 325 > > DB ERROR: db_verify: Page 11: out-of-order key at entry 581 > > DB ERROR: db_verify: Page 12: out-of-order key at entry 22 > > DB ERROR: db_verify: Page 16: out-of-order key at entry 249 > > DB ERROR: db_verify: Page 16: out-of-order key at entry 498 > > DB ERROR: db_verify: Page 16: out-of-order key at entry 754 > > DB ERROR: db_verify: Page 17: out-of-order key at entry 195 > > DB ERROR: db_verify: Page 17: out-of-order key at entry 451 > > DB ERROR: db_verify: Page 17: out-of-order key at entry 707 > > DB ERROR: db_verify: Page 18: out-of-order key at entry 148 > > DB ERROR: db_verify: Page 21: out-of-order key at entry 254 > > DB ERROR: db_verify: Page 21: out-of-order key at entry 510 > > DB ERROR: db_verify: Page 21: out-of-order key at entry 766 > > DB ERROR: db_verify: Page 22: out-of-order key at entry 207 > > DB ERROR: db_verify: Page 22: out-of-order key at entry 463 > > DB ERROR: db_verify: Page 22: out-of-order key at entry 719 > > DB ERROR: db_verify: Page 23: out-of-order key at entry 160 > > DB ERROR: db_verify: DB->verify: db/idsk_services/ancestorid.db4: > > DB_VERIFY_BAD: Database verification failed > > Secondary index file ancestorid.db4 in db/idsk_services is > corrupted. > > Please run db2index(.pl) for reindexing. > > > Is this after you imported your ldif file to the backend > 'idsk_services'? > > When you imported the ldif file, did you get any errors from ldif2db? > This is an sample output when we import Example.ldif. Did you see any > messages other than these? > [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096, > pages: 1009265, procpages: 23476 > [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: > 204800k > [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50, > import_pages: 51200, pagesize: 4096 > [18/Dec/2006:04:00:48 -0800] - WARNING: Import is running with > nsslapd-db-private-import-mem on; No other process is allowed to > access > the database > [18/Dec/2006:04:00:48 -0800] - dblayer_instance_start: pagesize: 4096, > pages: 1009265, procpages: 23476 > [18/Dec/2006:04:00:48 -0800] - cache autosizing: import cache: 204800k > [18/Dec/2006:04:00:48 -0800] - li_import_cache_autosize: 50, > import_pages: 51200, pagesize: 4096 > [18/Dec/2006:04:00:49 -0800] - import example: Beginning import job... > [18/Dec/2006:04:00:49 -0800] - import example: Index buffering enabled > with bucket size 100 > [18/Dec/2006:04:00:49 -0800] - import example: Processing file > "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif" > [18/Dec/2006:04:00:49 -0800] - import example: Finished scanning file > "/export/DS7.2-24471/server/usr/share/fedora-ds/data/Example.ldif" > (160 > entries) > [18/Dec/2006:04:00:50 -0800] - import example: Workers finished; > cleaning up... > [18/Dec/2006:04:00:50 -0800] - import example: Workers cleaned up. > [18/Dec/2006:04:00:50 -0800] - import example: Cleaning up producer > thread... > [18/Dec/2006:04:00:50 -0800] - import example: Indexing complete. > Post-processing... > [18/Dec/2006:04:00:50 -0800] - import example: Flushing caches... > [18/Dec/2006:04:00:50 -0800] - import example: Closing files... > [18/Dec/2006:04:00:51 -0800] - All database threads now stopped > [18/Dec/2006:04:00:51 -0800] - import example: Import complete. > Processed 160 entries in 2 seconds. (80.00 entries/sec) > > Thanks, > --noriko > > So then i reran db2index....verify_db again...same result. > > > > Edward > > > > > > On 12/15/06, *Eddie C* <edlinuxguru at gmail.com > <mailto:edlinuxguru at gmail.com> > > <mailto: edlinuxguru at gmail.com <mailto:edlinuxguru at gmail.com>>> > wrote: > > > > I recently did an ldif backup of our iplanet 52 database. Its > > about an 88 MB ldif file. > > I took this to a new FDS server Dell 850 3 ghz duel core 2 sata > > hard disks. > > I ran an ldapadd the data imported perfectly. > > Then I tried to cutover some systems and give the database > some load. > > > > System went 200% processor > > > > Eventually I realized I was missing indexes so I added them > > through the graphical tool. > > > > The log seemed to do something like this > > generating index 1% > > generating index 2% > > .... > > generating index 49% > > Done > > Seemed weird that they would jump from 49% to Done > > At this point the new system was running at 100% processor > > But the queries are running faster on our old 440 MHZ sparc t1 > > server52 database > > > > I ran > > DB ERROR: db_verify: Page 30: out-of-order key at entry 498 > > DB ERROR: db_verify: DB->verify: > db/o_com/channelcontentowner.db4: > > DB_VERIFY_BAD: Database verification failed > > > > then I tried db2_index. The program seemed to be in a tight loop > > complaining about 1 missing entry. > > > > I do not realize how the data can be so corrupted right after an > > import. > > > > These are someone generic symptoms. Any ideas? Thanks > > > > > > > ------------------------------------------------------------------------ > > > > -- > > Fedora-directory-users mailing list > > Fedora-directory-users at redhat.com > <mailto:Fedora-directory-users at redhat.com> > > https://www.redhat.com/mailman/listinfo/fedora-directory-users > <https://www.redhat.com/mailman/listinfo/fedora-directory-users> > > > > > > -- > Fedora-directory-users mailing list > Fedora-directory-users at redhat.com > <mailto:Fedora-directory-users at redhat.com> > https://www.redhat.com/mailman/listinfo/fedora-directory-users > > > > > ------------------------------------------------------------------------ > > -- > Fedora-directory-users mailing list > Fedora-directory-users at redhat.com > https://www.redhat.com/mailman/listinfo/fedora-directory-users > -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3245 bytes Desc: S/MIME Cryptographic Signature Url : http://lists.fedoraproject.org/pipermail/389-users/attachments/20061221/594179e6/attachment.bin
