On 11/13/2012 05:54 AM, ruslan usifov wrote: > Hello > > I test ceph cluster on VmWare machines (3 nodes in cluster) to make > rbd scalable block device, and have troubles when try to map rbd image > to device, i got follow message in kernel.log I haven't looked into this really yet, but this is a relatively old version of rbd. Current rbd does not use /dev/rbd0 (which is evidently being reused in this case, or at least that's why the error occurred). I'll look a little closer later. I would be very interested to know if doing the same thing with current code exposes a similar problem, or whether you are hitting a problem that has been fixed. -Alex > > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319319] ------------[ > cut here ]------------ > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319437] kernel BUG at > /build/buildd/linux-3.2.0/fs/sysfs/group.c:65! > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319583] invalid > opcode: 0000 [#1] SMP. > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319756] CPU 0. > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319814] Modules > linked in: rbd libceph ppdev vmw_balloon psmouse vmwgfx serio_raw ttm > drm parport_pc i2c_piix > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.320989]. > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.321063] Pid: 1860, > comm: rbd Tainted: G W 3.2.0-32-generic #51-Ubuntu VMware, > Inc. VMware Virtual P > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.321426] RIP: > 0010:[<ffffffff811efbc8>] [<ffffffff811efbc8>] > internal_create_group+0xf8/0x110 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.321659] RSP: > 0018:ffff8800135f1cc8 EFLAGS: 00010246 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.321785] RAX: > 00000000ffffffef RBX: ffff88001ab8e078 RCX: 0000000000014a7b > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.321937] RDX: > ffffffff81c34b00 RSI: 0000000000000000 RDI: ffff88001ab8e078 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.322090] RBP: > ffff8800135f1d08 R08: ffffea0000718480 R09: ffffffff813f3499 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.322241] R10: > 0000000000000000 R11: 0000000000000000 R12: ffffffff81c34b00 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.322394] R13: > ffff88001ab8e068 R14: 0000000000000000 R15: ffff88001ab8e000 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.323204] FS: > 00007f8bcf3aa780(0000) GS:ffff88001fc00000(0000) > knlGS:0000000000000000 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.323398] CS: 0010 DS: > 0000 ES: 0000 CR0: 0000000080050033 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.323540] CR2: > 00007f084b07f000 CR3: 000000001425c000 CR4: 00000000000006f0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.323720] DR0: > 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.323891] DR3: > 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.324047] Process rbd > (pid: 1860, threadinfo ffff8800135f0000, task ffff880014275c00) > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.324237] Stack: > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.324318] > ffff8800135f1d28 ffffffff813f34a9 0000000000000100 ffff880000000010 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.324629] > ffff88001ab8e000 ffff88001f270eb0 ffff88001ab8e068 ffff88001ab8e00c > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.324941] > ffff8800135f1d18 ffffffff811efc13 ffff8800135f1d28 ffffffff810feaf4 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.325253] Call Trace: > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.325344] > [<ffffffff813f34a9>] ? device_add+0x89/0x3e0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.325476] > [<ffffffff811efc13>] sysfs_create_group+0x13/0x20 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.325621] > [<ffffffff810feaf4>] blk_trace_init_sysfs+0x14/0x20 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.325762] > [<ffffffff812f5660>] blk_register_queue+0x40/0x110 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.325901] > [<ffffffff812fbacc>] add_disk+0xbc/0x230 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.326028] > [<ffffffffa01950a1>] rbd_init_disk+0x1b1/0x1d0 [rbd] > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.326170] > [<ffffffffa0195455>] rbd_add+0x285/0x4e0 [rbd] > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.326304] > [<ffffffff81157ce6>] ? alloc_pages_current+0xb6/0x120 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.326447] > [<ffffffff813f4727>] bus_attr_store+0x27/0x30 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.326579] > [<ffffffff811ec24f>] sysfs_write_file+0xef/0x170 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.326715] > [<ffffffff81178113>] vfs_write+0xb3/0x180 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.326842] > [<ffffffff8117843a>] sys_write+0x4a/0x90 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.326971] > [<ffffffff81663442>] system_call_fastpath+0x16/0x1b > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.327110] Code: c3 66 > 90 89 45 c8 e8 b8 d7 ff ff 8b 45 c8 eb df 0f 1f 00 4c 8b 6b 30 4c 89 > 6d d8 e9 77 ff ff ff > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.329592] RIP > [<ffffffff811efbc8>] internal_create_group+0xf8/0x110 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.329773] RSP <ffff8800135f1cc8> > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.329912]---[ end trace > 2d22165b460e6596 ]--- > > > Also i see different log message in kernel.log > > > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318832] ------------[ > cut here ]------------ > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318862] WARNING: at > /build/buildd/linux-3.2.0/fs/sysfs/dir.c:481 sysfs_add_one+0xc0/0xf0() > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318867] Hardware > name: VMware Virtual Platform > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318868] sysfs: cannot > create duplicate filename '/devices/virtual/block/rbd0' > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318871] Modules > linked in: rbd libceph ppdev vmw_balloon psmouse vmwgfx serio_raw ttm > drm parport_pc i2c_piix > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318895] Pid: 1860, > comm: rbd Not tainted 3.2.0-32-generic #51-Ubuntu > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318897] Call Trace: > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318910] > [<ffffffff81066e2f>] warn_slowpath_common+0x7f/0xc0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318913] > [<ffffffff81066f26>] warn_slowpath_fmt+0x46/0x50 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318915] > [<ffffffff811edf60>] sysfs_add_one+0xc0/0xf0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318917] > [<ffffffff811ee007>] create_dir+0x77/0xd0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318926] > [<ffffffff81192142>] ? inode_init_always+0x102/0x1c0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318928] > [<ffffffff811ee10d>] sysfs_create_dir+0x7d/0xc0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318940] > [<ffffffff8130d7c1>] kobject_add_internal+0xb1/0x200 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318943] > [<ffffffff8130dc87>] kobject_add+0x67/0xc0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318945] > [<ffffffff8130d6fa>] ? kobject_get+0x1a/0x30 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318958] > [<ffffffff813f351d>] device_add+0xfd/0x3e0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318963] > [<ffffffff812fb8d1>] register_disk+0x41/0x180 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318965] > [<ffffffff812fbac4>] add_disk+0xb4/0x230 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318969] > [<ffffffffa01950a1>] rbd_init_disk+0x1b1/0x1d0 [rbd] > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318971] > [<ffffffffa0195455>] rbd_add+0x285/0x4e0 [rbd] > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318980] > [<ffffffff81157ce6>] ? alloc_pages_current+0xb6/0x120 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318983] > [<ffffffff813f4727>] bus_attr_store+0x27/0x30 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318985] > [<ffffffff811ec24f>] sysfs_write_file+0xef/0x170 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318989] > [<ffffffff81178113>] vfs_write+0xb3/0x180 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.318991] > [<ffffffff8117843a>] sys_write+0x4a/0x90 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319002] > [<ffffffff81663442>] system_call_fastpath+0x16/0x1b > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319004] ---[ end > trace 2d22165b460e6595 ]--- > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319006] > kobject_add_internal failed for rbd0 with -EEXIST, don't try to > register things with the same name in > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319260] Pid: 1860, > comm: rbd Tainted: G W 3.2.0-32-generic #51-Ubuntu > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319261] Call Trace: > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319263] > [<ffffffff8130d865>] kobject_add_internal+0x155/0x200 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319266] > [<ffffffff8130dc87>] kobject_add+0x67/0xc0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319268] > [<ffffffff8130d6fa>] ? kobject_get+0x1a/0x30 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319270] > [<ffffffff813f351d>] device_add+0xfd/0x3e0 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319272] > [<ffffffff812fb8d1>] register_disk+0x41/0x180 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319274] > [<ffffffff812fbac4>] add_disk+0xb4/0x230 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319277] > [<ffffffffa01950a1>] rbd_init_disk+0x1b1/0x1d0 [rbd] > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319279] > [<ffffffffa0195455>] rbd_add+0x285/0x4e0 [rbd] > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319281] > [<ffffffff81157ce6>] ? alloc_pages_current+0xb6/0x120 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319283] > [<ffffffff813f4727>] bus_attr_store+0x27/0x30 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319285] > [<ffffffff811ec24f>] sysfs_write_file+0xef/0x170 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319287] > [<ffffffff81178113>] vfs_write+0xb3/0x180 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319289] > [<ffffffff8117843a>] sys_write+0x4a/0x90 > Nov 13 15:32:47 ceph-precie-64-02 kernel: [ 188.319291] > [<ffffffff81663442>] system_call_fastpath+0x16/0x1b > > > I use ubuntu 12.0.4 (Linux ceph-precie-64-02 3.2.0-32-generic > #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64 x86_64 x86_64 > GNU/Linux), and laster stable ceph release from > http://ceph.com/debian/, also i use pacemaker + corosync with follow > conf: > > node ceph-precie-64-01 > node ceph-precie-64-02 > node ceph-precie-64-03 > primitive samba_fs ocf:heartbeat:Filesystem \ > params device="-U cb4d3dda-92e9-4bd8-9fbc-2940c096e8ec" > directory="/mnt" fstype="ext4" > primitive samba_rbd ocf:ceph:rbd \ > params name="samba" > group samba samba_rbd samba_fs > property $id="cib-bootstrap-options" \ > dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \ > cluster-infrastructure="openais" \ > expected-quorum-votes="3" \ > stonith-enabled="false" \ > no-quorum-policy="stop" \ > last-lrm-refresh="1352806660" > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
