Hello everybody,
With newstore (+rocksdb) OSD crashed when i tried to format a device rbd
And when i want to restart :
2015-10-04 17:58:48.445653 7f8f96d43980 0 *ceph version
9.0.3-2146-gd1e6976* (d1e697628931775e7fcdbb79a76ddd8e7dedffd5),
process ceph-osd, pid 18359
2015-10-04 17:58:48.445662 7f8f96d43980 -1 WARNING: experimental
feature 'newstore' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2015-10-04 17:58:48.447938 7f8f96d43980 -1 WARNING: the following
dangerous and experimental features are enabled: newstore,rocksdb
2015-10-04 17:58:48.453264 7f8f96d43980 1
newstore(/var/lib/ceph/osd/ceph-0) _open_path using fs driver 'generic'
2015-10-04 17:58:48.453390 7f8f96d43980 1
newstore(/var/lib/ceph/osd/ceph-0) mount path /var/lib/ceph/osd/ceph-0
2015-10-04 17:58:48.453394 7f8f96d43980 1
newstore(/var/lib/ceph/osd/ceph-0) _open_path using fs driver 'generic'
2015-10-04 17:58:48.453406 7f8f96d43980 -1 WARNING: experimental
feature 'rocksdb' is enabled
Please be aware that this feature is experimental, untested,
unsupported, and may result in data corruption, data loss,
and/or irreparable damage to your cluster. Do not use
feature with important data.
2015-10-04 17:58:48.460232 7f8f96d43980 1
newstore(/var/lib/ceph/osd/ceph-0) _open_db opened rocksdb path
/var/lib/ceph/osd/ceph-0 options
2015-10-04 17:58:48.460483 7f8f96d43980 1
newstore(/var/lib/ceph/osd/ceph-0) _recover_next_fid old fid_max 1/1024
2015-10-04 17:58:48.460518 7f8f96d43980 1
newstore(/var/lib/ceph/osd/ceph-0) _recover_next_nid old nid_max 4096
2015-10-04 17:58:48.469128 7f8f96d43980 -1
newstore(/var/lib/ceph/osd/ceph-0) _do_wal_transaction zero on 21 got:
(95) Operation not supported
2015-10-04 17:58:48.773657 7f8f96d43980 -1 osd.0 0 OSD:init: unable to
mount object store
2015-10-04 17:58:48.773673 7f8f96d43980 -1 ^[[0;31m ** ERROR: osd init
failed: (95) Operation not supported
And with newstore debug
(...)
2015-10-04 18:16:59.243196 7f8b53f8d980 20
newstore(/var/lib/ceph/osd/ceph-0) _open_collections opened 1.fc_head
2015-10-04 18:16:59.243206 7f8b53f8d980 20
newstore(/var/lib/ceph/osd/ceph-0) _open_collections opened 1.fd_head
2015-10-04 18:16:59.243219 7f8b53f8d980 20
newstore(/var/lib/ceph/osd/ceph-0) _open_collections opened 1.fe_head
2015-10-04 18:16:59.243229 7f8b53f8d980 20
newstore(/var/lib/ceph/osd/ceph-0) _open_collections opened 1.ff_head
2015-10-04 18:16:59.243238 7f8b53f8d980 20
newstore(/var/lib/ceph/osd/ceph-0) _open_collections opened meta
2015-10-04 18:16:59.243242 7f8b53f8d980 10
newstore(/var/lib/ceph/osd/ceph-0) _aio_start
2015-10-04 18:16:59.243318 7f8b53f8d980 10
newstore(/var/lib/ceph/osd/ceph-0) _wal_replay start
2015-10-04 18:16:59.243331 7f8b53f8d980 20
newstore(/var/lib/ceph/osd/ceph-0) _wal_replay replay 000000000000003c
2015-10-04 18:16:59.243326 7f8b46495700 10
newstore(/var/lib/ceph/osd/ceph-0) _aio_thread start
2015-10-04 18:16:59.243334 7f8b53f8d980 20
newstore(/var/lib/ceph/osd/ceph-0) _do_wal_transaction zero 1/4~ 0~1048576
2015-10-04 18:16:59.243336 7f8b46495700 40
newstore(/var/lib/ceph/osd/ceph-0) _aio_thread polling
2015-10-04 18:16:59.243342 7f8b53f8d980 30
newstore(/var/lib/ceph/osd/ceph-0) _open_fid 1/4~ = 21 (open by handle)
2015-10-04 18:16:59.243967 7f8b53f8d980 -1
newstore(/var/lib/ceph/osd/ceph-0) _do_wal_transaction zero on 21 got:
(95) Operation not supported
2015-10-04 18:16:59.243979 7f8b53f8d980 10
newstore(/var/lib/ceph/osd/ceph-0) _aio_stop
2015-10-04 18:16:59.493405 7f8b46495700 10
newstore(/var/lib/ceph/osd/ceph-0) _aio_thread end
2015-10-04 18:16:59.583752 7f8b53f8d980 -1 osd.0 0 OSD:init: unable to
mount object store
2015-10-04 18:16:59.583768 7f8b53f8d980 -1 ** ERROR: osd init failed:
(95) Operation not supported
That sounds like an OP_ZERO transaction (file descriptor 21)
strace :
[pid 21569] fallocate(21, 02, 0, 1048576) = -1 EOPNOTSUPP (Operation
not supported)
I have the same error ext4 and xfs
Kernel : 3.18.16-gentoo
EXT4 :
- mount option user_xattr
- tune2fs -l
tune2fs 1.42.13 (17-May-2015)
Filesystem volume name: <none>
Last mounted on: /var/lib/ceph/osd/ceph-0
Filesystem UUID: 6027ad59-ed92-4975-b505-771ef9604fe4
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: ext_attr dir_index sparse_super2 filetype
extent flex_bg large_file huge_file
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: not clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 7640576
Block count: 977974998
Reserved block count: 48898749
Free blocks: 977675828
Free inodes: 7640565
First block: 0
Block size: 4096
Fragment size: 4096
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 256
Inode blocks per group: 8
Flex block group size: 32768
Filesystem created: Sun Oct 4 12:07:15 2015
Last mount time: Sun Oct 4 12:07:22 2015
Last write time: Sun Oct 4 12:07:22 2015
Mount count: 1
Maximum mount count: -1
Last checked: Sun Oct 4 12:07:15 2015
Check interval: 0 (<none>)
Lifetime writes: 1166 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Default directory hash: half_md4
Directory Hash Seed: 42eb03b4-82f8-48c1-9793-0015490c4c4a
Backup block groups: 1 29845
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
