On Mon, Jul 22, 2019 at 10:49 AM Ajitha Robert <ajitharobert01@xxxxxxxxx> wrote: > > No error log in rbd-mirroring except some connection timeout came once, > Scenario 1: > when I create a bootable volume of 100 GB with a glance image.Image get downloaded and from cinder, volume log throws with "volume is busy deleting volume that has snapshot" . Image was enabled with exclusive lock, journaling, layering, object-map, fast-diff and deep-flatten > Cinder volume is in error state but the rbd image is created in primary but not in secondary. Any chance you know where in Cinder that error is being thrown? A quick grep of the code doesn't reveal that error message. If the image is being synced to the secondary site when you attempt to delete it, it's possible you could hit this issue. Providing debug log messages from librbd on the Cinder controller might also be helpful for this. > Scenario 2: > but when i create a 50gb volume with another glance image. Volume get created. and in the backend i could see the rbd images both in primary and secondary > > From rbd mirror image status i found secondary cluster starts copying , and syncing was struck at around 14 %... It will be in 14 % .. no progress at all. should I set any parameters for this like timeout?? > > I manually checked rbd --cluster primary object-map check <object-name>.. No results came for the objects and the command was in hanging.. Thats why got worried on the failed to map object key log. I couldnt even rebuild the object map. It sounds like one or more of your primary OSDs are not reachable from the secondary site. If you run w/ "debug rbd-mirror = 20" and "debug rbd = 20", you should be able to see the last object it attempted to copy. From that, you could use "ceph osd map" to figure out the primary OSD for that object. > the image which was in syncing mode, showed read only status in secondary. > > > > On Mon, 22 Jul 2019, 17:36 Jason Dillaman, <jdillama@xxxxxxxxxx> wrote: >> >> On Sun, Jul 21, 2019 at 8:25 PM Ajitha Robert <ajitharobert01@xxxxxxxxx> wrote: >> > >> > I have a rbd mirroring setup with primary and secondary clusters as peers and I have a pool enabled image mode.., In this i created a rbd image , enabled with journaling. >> > >> > But whenever i enable mirroring on the image, I m getting error in osd.log. I couldnt trace it out. please guide me to solve this error. >> > >> > I think initially it worked fine. but after ceph process restart. these error coming >> > >> > >> > Secondary.osd.0.log >> > >> > 2019-07-22 05:36:17.371771 7ffbaa0e9700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:61: failed to get omap key: client_a5c76849-ba16-480a-a96b-ebfdb7f6ac65 >> > 2019-07-22 05:36:17.388552 7ffbaa0e9700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:472: active object set earlier than minimum: 0 < 1 >> > 2019-07-22 05:36:17.413102 7ffbaa0e9700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:61: failed to get omap key: order >> > 2019-07-22 05:36:23.341490 7ffbab8ec700 0 <cls> /build/ceph-12.2.12/src/cls/rbd/cls_rbd.cc:4125: error retrieving image id for global id '9e36b9f8-238e-4a54-a055-19b19447855e': (2) No such file or directory >> > >> > >> > primary-osd.0.log >> > >> > 2019-07-22 05:16:49.287769 7fae12db1700 0 log_channel(cluster) log [DBG] : 1.b deep-scrub ok >> > 2019-07-22 05:16:54.078698 7fae125b0700 0 log_channel(cluster) log [DBG] : 1.1b scrub starts >> > 2019-07-22 05:16:54.293839 7fae125b0700 0 log_channel(cluster) log [DBG] : 1.1b scrub ok >> > 2019-07-22 05:17:04.055277 7fae12db1700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:472: active object set earlier than minimum: 0 < 1 >> > >> > 2019-07-22 05:33:21.540986 7fae135b2700 0 <cls> /build/ceph-12.2.12/src/cls/journal/cls_journal.cc:472: active object set earlier than minimum: 0 < 1 >> > 2019-07-22 05:35:27.447820 7fae12db1700 0 <cls> /build/ceph-12.2.12/src/cls/rbd/cls_rbd.cc:4125: error retrieving image id for global id '8a61f694-f650-4ba1-b768-c5e7629ad2e0': (2) No such file or directory >> >> Those don't look like errors, but the log level should probably be >> reduced for those OSD cls methods. If you look at your rbd-mirror >> daemon log, do you see any errors? That would be the important place >> to look. >> >> > >> > -- >> > Regards, >> > Ajitha R >> > _______________________________________________ >> > ceph-users mailing list >> > ceph-users@xxxxxxxxxxxxxx >> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> >> >> >> -- >> Jason -- Jason _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |