Hi Mandell, I see the bug. I pushed a fix to wip-mon-command-race, 5011485e5e3fc9952ea58cd668e6feefc98024bf, and I believe fixes it, but I wasn't able to easily reproduce it myself so I'm not 100% certain. Can you give it a go? Thanks! sage On Mon, 8 Oct 2012, Mandell Degerness wrote: > osd dump output: > > [root@node-172-20-0-14 ~]# ceph osd dump 2 > dumped osdmap epoch 2 > epoch 2 > fsid d82665b6-3435-44b8-a89e-f7185f78d09d > created 2012-10-08 21:29:52.232400 > modifed 2012-10-08 21:29:57.297479 > flags > > pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num > 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 > pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins > pg_num 64 pgp_num 64 last_change 1 owner 0 > pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 64 > pgp_num 64 last_change 1 owner 0 > > max_osd 1 > osd.0 down out weight 0 up_from 0 up_thru 0 down_at 0 > last_clean_interval [0,0) :/0 :/0 :/0 exists,new > 564d7166-07b7-48cc-9b50-46ef7b260d5c > > > [root@node-172-20-0-14 ~]# ceph osd dump 3 > dumped osdmap epoch 3 > epoch 3 > fsid d82665b6-3435-44b8-a89e-f7185f78d09d > created 2012-10-08 21:29:52.232400 > modifed 2012-10-08 21:29:58.299491 > flags > > pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num > 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 > pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins > pg_num 64 pgp_num 64 last_change 1 owner 0 > pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 64 > pgp_num 64 last_change 1 owner 0 > > max_osd 1 > osd.0 up in weight 1 up_from 3 up_thru 0 down_at 0 > last_clean_interval [0,0) 172.20.0.13:6800/1723 172.20.0.13:6801/1723 > 172.20.0.13:6802/1723 exists,up 564d7166-07b7-48cc-9b50-46ef7b260d5c > > > [root@node-172-20-0-14 ~]# ceph osd dump 4 > dumped osdmap epoch 4 > epoch 4 > fsid d82665b6-3435-44b8-a89e-f7185f78d09d > created 2012-10-08 21:29:52.232400 > modifed 2012-10-08 21:29:59.304087 > flags > > pool 0 'data' rep size 2 crush_ruleset 0 object_hash rjenkins pg_num > 64 pgp_num 64 last_change 1 owner 0 crash_replay_interval 45 > pool 1 'metadata' rep size 2 crush_ruleset 1 object_hash rjenkins > pg_num 64 pgp_num 64 last_change 1 owner 0 > pool 2 'rbd' rep size 2 crush_ruleset 2 object_hash rjenkins pg_num 64 > pgp_num 64 last_change 1 owner 0 > > max_osd 3 > osd.0 up in weight 1 up_from 3 up_thru 0 down_at 0 > last_clean_interval [0,0) 172.20.0.13:6800/1723 172.20.0.13:6801/1723 > 172.20.0.13:6802/1723 exists,up 564d7166-07b7-48cc-9b50-46ef7b260d5c > osd.1 down out weight 0 up_from 0 up_thru 0 down_at 0 > last_clean_interval [0,0) :/0 :/0 :/0 exists,new > 3351a0f0-f6e8-430a-b7a4-ea613a3ddf35 > osd.2 down out weight 0 up_from 0 up_thru 0 down_at 0 > last_clean_interval [0,0) :/0 :/0 :/0 exists,new > 3f04cdbe-a468-42d3-a465-2487cc369d90 > > > > > On Mon, Oct 8, 2012 at 3:49 PM, Sage Weil <sage@xxxxxxxxxxx> wrote: > > On Mon, 8 Oct 2012, Mandell Degerness wrote: > >> Sorry, I should have used the https link: > >> > >> https://gist.github.com/af546ece91be0ba268d3 > > > > What do 'ceph osd dump 2', 'ceph osd dump 3', and 'ceph osd dump 4' say? > > > > thanks! > > sage > > > >> > >> On Mon, Oct 8, 2012 at 3:20 PM, Mandell Degerness > >> <mandell@xxxxxxxxxxxxxxx> wrote: > >> > Here is the log I got when running with the options suggested by sage: > >> > > >> > git@xxxxxxxxxxxxxxx:af546ece91be0ba268d3.git > >> > > >> > On Mon, Oct 8, 2012 at 11:34 AM, Sage Weil <sage@xxxxxxxxxxx> wrote: > >> >> Hi Mandell, > >> >> > >> >> On Mon, 8 Oct 2012, Mandell Degerness wrote: > >> >>> Hi list, > >> >>> > >> >>> I've run into a bit of a weird error and I'm hoping that you can tell > >> >>> me what is going wrong. There seems to be a race condition in the way > >> >>> I am using "ceph osd create <uuid>" and actually creating the OSD's. > >> >>> The log from one of the servers is at: > >> >>> > >> >>> https://gist.github.com/528e347a5c0ffeb30abd > >> >>> > >> >>> The process I am trying to follow (for the OSDs) is: > >> >>> > >> >>> 1) Create XFS file system on disk. > >> >>> 2) Use FS UUID as source to get a new OSD id #. > >> >>> 'ceph', 'osd', 'create', '32895846-ca1c-4265-9ce7-9f2a42b41672' > >> >>> (Returns 2.) > >> >>> 3) Pass the UUID and OSD id to the create osd command > >> >>> > >> >>> ceph-osd -c /etc/ceph/ceph.conf --fsid > >> >>> e61c1b11-4a1c-47aa-868d-7b51b1e610d3 --osd-uuid > >> >>> 32895846-ca1c-4265-9ce7-9f2a42b41672 -i 2 --mkfs --osd-journal-size > >> >>> 8192 > >> >>> 4) Start the OSD, as part of the start process, I verify that the > >> >>> whoami and osd fsid agree (in case this disk came from a previous > >> >>> cluster, somehow) - should be just a sanity check > >> >>> 'ceph', 'osd', 'create', '32895846-ca1c-4265-9ce7-9f2a42b41672' > >> >>> (Returns 1!) > >> >>> > >> >>> This is clearly a race condition because we have several cluster > >> >>> creations without this happening and then this happens about once > >> >>> every 8 times or so. Thoughts? > >> >> > >> >> That definitely sounds like a race. I'm not seeing it by inspection, > >> >> though, and wasn't able to reproduce. Is it possible to capture a monitor > >> >> log (debug ms = 1, debug mon = 20) of this occurring and share that? > >> >> > >> >> Thanks! > >> >> sage > >> -- > >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > >> the body of a message to majordomo@xxxxxxxxxxxxxxx > >> More majordomo info at http://vger.kernel.org/majordomo-info.html > >> > >> > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
