Hi all,
I'm trying to install ceph on a 7-nodes preproduction cluster. Each
node has 24x 4TB SAS disks (2x dell md1400 enclosures) and 6x 800GB
SSDs (for cache tiering, not journals). I'm using Ubuntu 14.04 and
ceph-deploy to install the cluster, I've been trying both Firefly and
Giant and getting the same error. However, the logs I'm reporting are
relative to the Firefly installation.
The installation seems to go fine until I try to install the last 2
OSDs (they are SSD disks) of each host. All the OSDs from 0 to 195 are
UP and IN, but when I try to deploy the next OSD (no matter what host)
ceph-osd daemon won't start. The error I get is:
2015-03-25 17:00:17.130937 7fe231312800 0 ceph version 0.80.9
(b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid
20280
2015-03-25 17:00:17.133601 7fe231312800 10
filestore(/var/lib/ceph/osd/ceph-196) dump_stop
2015-03-25 17:00:17.133694 7fe231312800 5
filestore(/var/lib/ceph/osd/ceph-196) basedir
/var/lib/ceph/osd/ceph-196 journal /var/lib/ceph/osd/ceph-196/journal
2015-03-25 17:00:17.133725 7fe231312800 10
filestore(/var/lib/ceph/osd/ceph-196) mount fsid is
8c2fa707-750a-4773-8918-a368367d9cf5
2015-03-25 17:00:17.133789 7fe231312800 0
filestore(/var/lib/ceph/osd/ceph-196) mount detected xfs (libxfs)
2015-03-25 17:00:17.133810 7fe231312800 1
filestore(/var/lib/ceph/osd/ceph-196) disabling 'filestore replica
fadvise' due to known issues with fadvise(DONTNEED) on xfs
2015-03-25 17:00:17.135882 7fe231312800 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features:
FIEMAP ioctl is supported and appears to work
2015-03-25 17:00:17.135892 7fe231312800 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features:
FIEMAP ioctl is disabled via 'filestore fiemap' config option
2015-03-25 17:00:17.136318 7fe231312800 0
genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features:
syncfs(2) syscall fully supported (by glibc and kernel)
2015-03-25 17:00:17.136373 7fe231312800 0
xfsfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_feature:
extsize is disabled by conf
2015-03-25 17:00:17.136640 7fe231312800 5
filestore(/var/lib/ceph/osd/ceph-196) mount op_seq is 1
2015-03-25 17:00:17.137547 7fe231312800 20 filestore (init)dbobjectmap: seq is 1
2015-03-25 17:00:17.137560 7fe231312800 10
filestore(/var/lib/ceph/osd/ceph-196) open_journal at
/var/lib/ceph/osd/ceph-196/journal
2015-03-25 17:00:17.137575 7fe231312800 0
filestore(/var/lib/ceph/osd/ceph-196) mount: enabling WRITEAHEAD
journal mode: checkpoint is not enabled
2015-03-25 17:00:17.137580 7fe231312800 10
filestore(/var/lib/ceph/osd/ceph-196) list_collections
2015-03-25 17:00:17.137661 7fe231312800 10 journal journal_replay fs op_seq 1
2015-03-25 17:00:17.137668 7fe231312800 2 journal open
/var/lib/ceph/osd/ceph-196/journal fsid
8c2fa707-750a-4773-8918-a368367d9cf5 fs_op_seq 1
2015-03-25 17:00:17.137670 7fe22b8b1700 20
filestore(/var/lib/ceph/osd/ceph-196) sync_entry waiting for
max_interval 5.000000
2015-03-25 17:00:17.137690 7fe231312800 10 journal _open_block_device:
ignoring osd journal size. We'll use the entire block device (size:
5367661056)
2015-03-25 17:00:17.162489 7fe231312800 1 journal _open
/var/lib/ceph/osd/ceph-196/journal fd 20: 5367660544 bytes, block size
4096 bytes, directio = 1, aio = 1
2015-03-25 17:00:17.162502 7fe231312800 10 journal read_header
2015-03-25 17:00:17.172249 7fe231312800 10 journal header: block_size
4096 alignment 4096 max_size 5367660544
2015-03-25 17:00:17.172256 7fe231312800 10 journal header: start 50987008
2015-03-25 17:00:17.172257 7fe231312800 10 journal write_pos 4096
2015-03-25 17:00:17.172259 7fe231312800 10 journal open header.fsid =
942f2d62-dd99-42a8-878a-feea443aaa61
2015-03-25 17:00:17.172264 7fe231312800 -1 journal FileJournal::open:
ondisk fsid 942f2d62-dd99-42a8-878a-feea443aaa61 doesn't match
expected 8c2fa707-750a-4773-8918-a368367d9cf5, invalid (someone
else's?) journal
2015-03-25 17:00:17.172268 7fe231312800 3 journal journal_replay open
failed with (22) Invalid argument
2015-03-25 17:00:17.172284 7fe231312800 -1
filestore(/var/lib/ceph/osd/ceph-196) mount failed to open journal
/var/lib/ceph/osd/ceph-196/journal: (22) Invalid argument
2015-03-25 17:00:17.172304 7fe22b8b1700 20
filestore(/var/lib/ceph/osd/ceph-196) sync_entry woke after 0.034632
2015-03-25 17:00:17.172330 7fe22b8b1700 10 journal commit_start
max_applied_seq 1, open_ops 0
2015-03-25 17:00:17.172333 7fe22b8b1700 10 journal commit_start
blocked, all open_ops have completed
2015-03-25 17:00:17.172334 7fe22b8b1700 10 journal commit_start nothing to do
2015-03-25 17:00:17.172465 7fe231312800 -1 ** ERROR: error converting
store /var/lib/ceph/osd/ceph-196: (22) Invalid argument
I'm attaching the "full" log of "ceph-deploy osd create osd-l2-05:sde"
and the /var/log/ceph/ceph-osd.196.log, after trying to re-start the
osd with increased verbosing, as long as the ceph.conf I'm using.
I've also checked if the "journal" symlinks were correct, and they all
point to different devices:
root@osd-l2-05:~# ls -1 $(readlink -e $(readlink -e
/var/lib/ceph/osd/ceph-*/journal))|sort | uniq -c
1 /dev/sda2
1 /dev/sdaa2
1 /dev/sdab2
1 /dev/sdac2
1 /dev/sdad2
1 /dev/sdae2
1 /dev/sdb2
1 /dev/sdc2
1 /dev/sdd2
1 /dev/sde2
1 /dev/sdf2
1 /dev/sdh2
1 /dev/sdi2
1 /dev/sdj2
1 /dev/sdk2
1 /dev/sdl2
1 /dev/sdm2
1 /dev/sdn2
1 /dev/sdo2
1 /dev/sdp2
1 /dev/sdq2
1 /dev/sdr2
1 /dev/sds2
1 /dev/sdt2
1 /dev/sdu2
1 /dev/sdv2
1 /dev/sdw2
1 /dev/sdx2
1 /dev/sdy2
1 /dev/sdz2
On a side note: if I reboot the node, it might happen that the osd 196
actually starts, but then some other OSD (that was UP before the
reboot), will not start anymore. In this case I get a totally
different error on the OSD that was up before the reboot and down
after:
2015-03-25 17:11:38.112462 7fe6524f5700 -1 os/FileJournal.cc: In
function 'int FileJournal::write_aio_bl(off64_t&, ceph::bufferlist&,
uint64_t)' thread 7fe6524f5700 time 2015-03-25 17:11:38.110877
os/FileJournal.cc: 1337: FAILED assert(0 == "io_submit got unexpected error")
ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047)
1: (FileJournal::write_aio_bl(long&, ceph::buffer::list&, unsigned
long)+0x799) [0x95ae59]
2: (FileJournal::do_aio_write(ceph::buffer::list&)+0x1ef) [0x95b4cf]
3: (FileJournal::write_thread_entry()+0x823) [0x961873]
4: (FileJournal::Writer::entry()+0xd) [0x8913cd]
5: (()+0x8182) [0x7fe657c7b182]
6: (clone()+0x6d) [0x7fe6563ee47d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.
(plus a very long trace that you can find in attach, as 'ceph-osd.35.log')
I think there is some kind of "conflict" between the journal of two
OSDs, but I honestly can't figure out why this is happening and how,
since the journal links are all different. The only think I noticed is
that the problem started only when I tried to create the OSD on the
SSD devices, but this might be only partially relevant, because I
added the SSDs only in the end.
Hardware setup:
7x Dell R630 w/ PERC H730P (for SSDs) and PERC H830, both in "jbod" mode
2x MD1400 w/ 12x SAS Seagate 4TB ST4000NM0023
6x SSD Toshiba 800GB PX03SNF080
Thank you in advance,
Antonio
--
antonio.s.messina@xxxxxxxxx
antonio.messina@xxxxxx +41 (0)44 635 42 22
S3IT: Service and Support for Science IT http://www.s3it.uzh.ch/
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland
[osd-l2-05][DEBUG ] connected to host: osd-l2-05
[osd-l2-05][DEBUG ] detect platform information from remote host
[osd-l2-05][DEBUG ] detect machine type
[ceph_deploy.osd][DEBUG ] Deploying osd to osd-l2-05
[osd-l2-05][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[osd-l2-05][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add
[ceph_deploy.osd][DEBUG ] Preparing host osd-l2-05 disk /dev/sde journal None activate True
[osd-l2-05][INFO ] Running command: ceph-disk -v prepare --fs-type xfs --cluster ceph -- /dev/sde
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
[osd-l2-05][WARNING] INFO:ceph-disk:Will colocate journal with data on /dev/sde
[osd-l2-05][WARNING] DEBUG:ceph-disk:Creating journal partition num 2 size 5120 on /dev/sde
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /sbin/sgdisk --new=2:0:5120M --change-name=2:ceph journal --partition-guid=2:fa830818-d4f7-46f8-8745-64f799f1a1c0 --typecode=2:45b0969e-9b03-4f30-b4c6-b4b80cef
f106 --mbrtogpt -- /dev/sde
[osd-l2-05][DEBUG ] The operation has completed successfully.
[osd-l2-05][WARNING] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/sde
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sde
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /sbin/udevadm settle
[osd-l2-05][WARNING] DEBUG:ceph-disk:Journal is GPT partition /dev/disk/by-partuuid/fa830818-d4f7-46f8-8745-64f799f1a1c0
[osd-l2-05][WARNING] DEBUG:ceph-disk:Creating osd partition on /dev/sde
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:8c2fa707-750a-4773-8918-a368367d9cf5 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2b
e -- /dev/sde
[osd-l2-05][DEBUG ] The operation has completed successfully.
[osd-l2-05][WARNING] DEBUG:ceph-disk:Calling partprobe on created device /dev/sde
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sde
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /sbin/udevadm settle
[osd-l2-05][WARNING] DEBUG:ceph-disk:Creating xfs fs on /dev/sde1
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sde1
[osd-l2-05][DEBUG ] meta-data=/dev/sde1 isize=2048 agcount=32, agsize=6063815 blks
[osd-l2-05][DEBUG ] = sectsz=4096 attr=2, projid32bit=0
[osd-l2-05][DEBUG ] data = bsize=4096 blocks=194042065, imaxpct=25
[osd-l2-05][DEBUG ] = sunit=0 swidth=0 blks
[osd-l2-05][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0
[osd-l2-05][DEBUG ] log =internal log bsize=4096 blocks=94747, version=2
[osd-l2-05][DEBUG ] = sectsz=4096 sunit=1 blks, lazy-count=1
[osd-l2-05][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0
[osd-l2-05][WARNING] DEBUG:ceph-disk:Mounting /dev/sde1 on /var/lib/ceph/tmp/mnt.jb6zEM with options noatime,inode64
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sde1 /var/lib/ceph/tmp/mnt.jb6zEM
[osd-l2-05][WARNING] DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.jb6zEM
[osd-l2-05][WARNING] DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.jb6zEM/journal -> /dev/disk/by-partuuid/fa830818-d4f7-46f8-8745-64f799f1a1c0
[osd-l2-05][WARNING] DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.jb6zEM
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.jb6zEM
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sde
[osd-l2-05][DEBUG ] The operation has completed successfully.
[osd-l2-05][WARNING] DEBUG:ceph-disk:Calling partprobe on prepared device /dev/sde
[osd-l2-05][WARNING] INFO:ceph-disk:Running command: /sbin/partprobe /dev/sde
[osd-l2-05][INFO ] Running command: udevadm trigger --subsystem-match=block --action=add
[osd-l2-05][INFO ] checking OSD status...
[osd-l2-05][INFO ] Running command: ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host osd-l2-05 is now ready for osd use.
2015-03-25 17:00:17.130937 7fe231312800 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid 20280 2015-03-25 17:00:17.133601 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) dump_stop 2015-03-25 17:00:17.133694 7fe231312800 5 filestore(/var/lib/ceph/osd/ceph-196) basedir /var/lib/ceph/osd/ceph-196 journal /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.133725 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) mount fsid is 8c2fa707-750a-4773-8918-a368367d9cf5 2015-03-25 17:00:17.133789 7fe231312800 0 filestore(/var/lib/ceph/osd/ceph-196) mount detected xfs (libxfs) 2015-03-25 17:00:17.133810 7fe231312800 1 filestore(/var/lib/ceph/osd/ceph-196) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-03-25 17:00:17.135882 7fe231312800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is supported and appears to work 2015-03-25 17:00:17.135892 7fe231312800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-25 17:00:17.136318 7fe231312800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-25 17:00:17.136373 7fe231312800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_feature: extsize is disabled by conf 2015-03-25 17:00:17.136640 7fe231312800 5 filestore(/var/lib/ceph/osd/ceph-196) mount op_seq is 1 2015-03-25 17:00:17.137547 7fe231312800 20 filestore (init)dbobjectmap: seq is 1 2015-03-25 17:00:17.137560 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) open_journal at /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.137575 7fe231312800 0 filestore(/var/lib/ceph/osd/ceph-196) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-03-25 17:00:17.137580 7fe231312800 10 filestore(/var/lib/ceph/osd/ceph-196) list_collections 2015-03-25 17:00:17.137661 7fe231312800 10 journal journal_replay fs op_seq 1 2015-03-25 17:00:17.137668 7fe231312800 2 journal open /var/lib/ceph/osd/ceph-196/journal fsid 8c2fa707-750a-4773-8918-a368367d9cf5 fs_op_seq 1 2015-03-25 17:00:17.137670 7fe22b8b1700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry waiting for max_interval 5.000000 2015-03-25 17:00:17.137690 7fe231312800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2015-03-25 17:00:17.162489 7fe231312800 1 journal _open /var/lib/ceph/osd/ceph-196/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-25 17:00:17.162502 7fe231312800 10 journal read_header 2015-03-25 17:00:17.172249 7fe231312800 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2015-03-25 17:00:17.172256 7fe231312800 10 journal header: start 50987008 2015-03-25 17:00:17.172257 7fe231312800 10 journal write_pos 4096 2015-03-25 17:00:17.172259 7fe231312800 10 journal open header.fsid = 942f2d62-dd99-42a8-878a-feea443aaa61 2015-03-25 17:00:17.172264 7fe231312800 -1 journal FileJournal::open: ondisk fsid 942f2d62-dd99-42a8-878a-feea443aaa61 doesn't match expected 8c2fa707-750a-4773-8918-a368367d9cf5, invalid (someone else's?) journal 2015-03-25 17:00:17.172268 7fe231312800 3 journal journal_replay open failed with (22) Invalid argument 2015-03-25 17:00:17.172284 7fe231312800 -1 filestore(/var/lib/ceph/osd/ceph-196) mount failed to open journal /var/lib/ceph/osd/ceph-196/journal: (22) Invalid argument 2015-03-25 17:00:17.172304 7fe22b8b1700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry woke after 0.034632 2015-03-25 17:00:17.172330 7fe22b8b1700 10 journal commit_start max_applied_seq 1, open_ops 0 2015-03-25 17:00:17.172333 7fe22b8b1700 10 journal commit_start blocked, all open_ops have completed 2015-03-25 17:00:17.172334 7fe22b8b1700 10 journal commit_start nothing to do 2015-03-25 17:00:17.172465 7fe231312800 -1 ** ERROR: error converting store /var/lib/ceph/osd/ceph-196: (22) Invalid argument 2015-03-25 17:00:17.333282 7ffb23262800 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid 20340 2015-03-25 17:00:17.335962 7ffb23262800 10 filestore(/var/lib/ceph/osd/ceph-196) dump_stop 2015-03-25 17:00:17.336108 7ffb23262800 5 filestore(/var/lib/ceph/osd/ceph-196) basedir /var/lib/ceph/osd/ceph-196 journal /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.336126 7ffb23262800 10 filestore(/var/lib/ceph/osd/ceph-196) mount fsid is 8c2fa707-750a-4773-8918-a368367d9cf5 2015-03-25 17:00:17.336185 7ffb23262800 0 filestore(/var/lib/ceph/osd/ceph-196) mount detected xfs (libxfs) 2015-03-25 17:00:17.336202 7ffb23262800 1 filestore(/var/lib/ceph/osd/ceph-196) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-03-25 17:00:17.337708 7ffb23262800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is supported and appears to work 2015-03-25 17:00:17.337725 7ffb23262800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-25 17:00:17.338147 7ffb23262800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-25 17:00:17.338209 7ffb23262800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_feature: extsize is disabled by conf 2015-03-25 17:00:17.338445 7ffb23262800 5 filestore(/var/lib/ceph/osd/ceph-196) mount op_seq is 1 2015-03-25 17:00:17.339696 7ffb23262800 20 filestore (init)dbobjectmap: seq is 1 2015-03-25 17:00:17.339708 7ffb23262800 10 filestore(/var/lib/ceph/osd/ceph-196) open_journal at /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.339737 7ffb23262800 0 filestore(/var/lib/ceph/osd/ceph-196) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-03-25 17:00:17.339742 7ffb23262800 10 filestore(/var/lib/ceph/osd/ceph-196) list_collections 2015-03-25 17:00:17.339830 7ffb23262800 10 journal journal_replay fs op_seq 1 2015-03-25 17:00:17.339847 7ffb23262800 2 journal open /var/lib/ceph/osd/ceph-196/journal fsid 8c2fa707-750a-4773-8918-a368367d9cf5 fs_op_seq 1 2015-03-25 17:00:17.339886 7ffb23262800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2015-03-25 17:00:17.339881 7ffb1d801700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry waiting for max_interval 5.000000 2015-03-25 17:00:17.364155 7ffb23262800 1 journal _open /var/lib/ceph/osd/ceph-196/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-25 17:00:17.364167 7ffb23262800 10 journal read_header 2015-03-25 17:00:17.371939 7ffb23262800 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2015-03-25 17:00:17.371946 7ffb23262800 10 journal header: start 50987008 2015-03-25 17:00:17.371947 7ffb23262800 10 journal write_pos 4096 2015-03-25 17:00:17.371949 7ffb23262800 10 journal open header.fsid = 942f2d62-dd99-42a8-878a-feea443aaa61 2015-03-25 17:00:17.371954 7ffb23262800 -1 journal FileJournal::open: ondisk fsid 942f2d62-dd99-42a8-878a-feea443aaa61 doesn't match expected 8c2fa707-750a-4773-8918-a368367d9cf5, invalid (someone else's?) journal 2015-03-25 17:00:17.371958 7ffb23262800 3 journal journal_replay open failed with (22) Invalid argument 2015-03-25 17:00:17.371979 7ffb23262800 -1 filestore(/var/lib/ceph/osd/ceph-196) mount failed to open journal /var/lib/ceph/osd/ceph-196/journal: (22) Invalid argument 2015-03-25 17:00:17.372019 7ffb1d801700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry woke after 0.032137 2015-03-25 17:00:17.372033 7ffb1d801700 10 journal commit_start max_applied_seq 1, open_ops 0 2015-03-25 17:00:17.372035 7ffb1d801700 10 journal commit_start blocked, all open_ops have completed 2015-03-25 17:00:17.372036 7ffb1d801700 10 journal commit_start nothing to do 2015-03-25 17:00:17.372249 7ffb23262800 -1 ** ERROR: error converting store /var/lib/ceph/osd/ceph-196: (22) Invalid argument 2015-03-25 17:00:17.537814 7f9270e89800 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid 20400 2015-03-25 17:00:17.540154 7f9270e89800 10 filestore(/var/lib/ceph/osd/ceph-196) dump_stop 2015-03-25 17:00:17.540370 7f9270e89800 5 filestore(/var/lib/ceph/osd/ceph-196) basedir /var/lib/ceph/osd/ceph-196 journal /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.540415 7f9270e89800 10 filestore(/var/lib/ceph/osd/ceph-196) mount fsid is 8c2fa707-750a-4773-8918-a368367d9cf5 2015-03-25 17:00:17.540494 7f9270e89800 0 filestore(/var/lib/ceph/osd/ceph-196) mount detected xfs (libxfs) 2015-03-25 17:00:17.540502 7f9270e89800 1 filestore(/var/lib/ceph/osd/ceph-196) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-03-25 17:00:17.541816 7f9270e89800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is supported and appears to work 2015-03-25 17:00:17.541826 7f9270e89800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-25 17:00:17.542180 7f9270e89800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-25 17:00:17.542235 7f9270e89800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_feature: extsize is disabled by conf 2015-03-25 17:00:17.542487 7f9270e89800 5 filestore(/var/lib/ceph/osd/ceph-196) mount op_seq is 1 2015-03-25 17:00:17.543488 7f9270e89800 20 filestore (init)dbobjectmap: seq is 1 2015-03-25 17:00:17.543502 7f9270e89800 10 filestore(/var/lib/ceph/osd/ceph-196) open_journal at /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.543532 7f9270e89800 0 filestore(/var/lib/ceph/osd/ceph-196) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-03-25 17:00:17.543537 7f9270e89800 10 filestore(/var/lib/ceph/osd/ceph-196) list_collections 2015-03-25 17:00:17.543631 7f9270e89800 10 journal journal_replay fs op_seq 1 2015-03-25 17:00:17.543652 7f9270e89800 2 journal open /var/lib/ceph/osd/ceph-196/journal fsid 8c2fa707-750a-4773-8918-a368367d9cf5 fs_op_seq 1 2015-03-25 17:00:17.543686 7f9270e89800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2015-03-25 17:00:17.543666 7f926b428700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry waiting for max_interval 5.000000 2015-03-25 17:00:17.568383 7f9270e89800 1 journal _open /var/lib/ceph/osd/ceph-196/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-25 17:00:17.568396 7f9270e89800 10 journal read_header 2015-03-25 17:00:17.577731 7f9270e89800 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2015-03-25 17:00:17.577743 7f9270e89800 10 journal header: start 50987008 2015-03-25 17:00:17.577745 7f9270e89800 10 journal write_pos 4096 2015-03-25 17:00:17.577749 7f9270e89800 10 journal open header.fsid = 942f2d62-dd99-42a8-878a-feea443aaa61 2015-03-25 17:00:17.577759 7f9270e89800 -1 journal FileJournal::open: ondisk fsid 942f2d62-dd99-42a8-878a-feea443aaa61 doesn't match expected 8c2fa707-750a-4773-8918-a368367d9cf5, invalid (someone else's?) journal 2015-03-25 17:00:17.577766 7f9270e89800 3 journal journal_replay open failed with (22) Invalid argument 2015-03-25 17:00:17.577798 7f9270e89800 -1 filestore(/var/lib/ceph/osd/ceph-196) mount failed to open journal /var/lib/ceph/osd/ceph-196/journal: (22) Invalid argument 2015-03-25 17:00:17.577852 7f926b428700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry woke after 0.034184 2015-03-25 17:00:17.577875 7f926b428700 10 journal commit_start max_applied_seq 1, open_ops 0 2015-03-25 17:00:17.577877 7f926b428700 10 journal commit_start blocked, all open_ops have completed 2015-03-25 17:00:17.577879 7f926b428700 10 journal commit_start nothing to do 2015-03-25 17:00:17.578040 7f9270e89800 -1 ** ERROR: error converting store /var/lib/ceph/osd/ceph-196: (22) Invalid argument 2015-03-25 17:00:17.747137 7f1460112800 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid 20462 2015-03-25 17:00:17.749822 7f1460112800 10 filestore(/var/lib/ceph/osd/ceph-196) dump_stop 2015-03-25 17:00:17.749928 7f1460112800 5 filestore(/var/lib/ceph/osd/ceph-196) basedir /var/lib/ceph/osd/ceph-196 journal /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.749948 7f1460112800 10 filestore(/var/lib/ceph/osd/ceph-196) mount fsid is 8c2fa707-750a-4773-8918-a368367d9cf5 2015-03-25 17:00:17.749998 7f1460112800 0 filestore(/var/lib/ceph/osd/ceph-196) mount detected xfs (libxfs) 2015-03-25 17:00:17.750003 7f1460112800 1 filestore(/var/lib/ceph/osd/ceph-196) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-03-25 17:00:17.751478 7f1460112800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is supported and appears to work 2015-03-25 17:00:17.751488 7f1460112800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-25 17:00:17.751875 7f1460112800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-25 17:00:17.751948 7f1460112800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_feature: extsize is disabled by conf 2015-03-25 17:00:17.752183 7f1460112800 5 filestore(/var/lib/ceph/osd/ceph-196) mount op_seq is 1 2015-03-25 17:00:17.753549 7f1460112800 20 filestore (init)dbobjectmap: seq is 1 2015-03-25 17:00:17.753561 7f1460112800 10 filestore(/var/lib/ceph/osd/ceph-196) open_journal at /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.753605 7f1460112800 0 filestore(/var/lib/ceph/osd/ceph-196) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-03-25 17:00:17.753622 7f1460112800 10 filestore(/var/lib/ceph/osd/ceph-196) list_collections 2015-03-25 17:00:17.753752 7f1460112800 10 journal journal_replay fs op_seq 1 2015-03-25 17:00:17.753758 7f1460112800 2 journal open /var/lib/ceph/osd/ceph-196/journal fsid 8c2fa707-750a-4773-8918-a368367d9cf5 fs_op_seq 1 2015-03-25 17:00:17.753758 7f145a6b1700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry waiting for max_interval 5.000000 2015-03-25 17:00:17.753792 7f1460112800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2015-03-25 17:00:17.778002 7f1460112800 1 journal _open /var/lib/ceph/osd/ceph-196/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-25 17:00:17.778016 7f1460112800 10 journal read_header 2015-03-25 17:00:17.785776 7f1460112800 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2015-03-25 17:00:17.785783 7f1460112800 10 journal header: start 50987008 2015-03-25 17:00:17.785784 7f1460112800 10 journal write_pos 4096 2015-03-25 17:00:17.785786 7f1460112800 10 journal open header.fsid = 942f2d62-dd99-42a8-878a-feea443aaa61 2015-03-25 17:00:17.785803 7f1460112800 -1 journal FileJournal::open: ondisk fsid 942f2d62-dd99-42a8-878a-feea443aaa61 doesn't match expected 8c2fa707-750a-4773-8918-a368367d9cf5, invalid (someone else's?) journal 2015-03-25 17:00:17.785807 7f1460112800 3 journal journal_replay open failed with (22) Invalid argument 2015-03-25 17:00:17.785831 7f1460112800 -1 filestore(/var/lib/ceph/osd/ceph-196) mount failed to open journal /var/lib/ceph/osd/ceph-196/journal: (22) Invalid argument 2015-03-25 17:00:17.785896 7f145a6b1700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry woke after 0.032135 2015-03-25 17:00:17.785937 7f145a6b1700 10 journal commit_start max_applied_seq 1, open_ops 0 2015-03-25 17:00:17.785940 7f145a6b1700 10 journal commit_start blocked, all open_ops have completed 2015-03-25 17:00:17.785942 7f145a6b1700 10 journal commit_start nothing to do 2015-03-25 17:00:17.786068 7f1460112800 -1 ** ERROR: error converting store /var/lib/ceph/osd/ceph-196: (22) Invalid argument 2015-03-25 17:00:17.945957 7f8f49f58800 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid 20522 2015-03-25 17:00:17.948518 7f8f49f58800 10 filestore(/var/lib/ceph/osd/ceph-196) dump_stop 2015-03-25 17:00:17.948639 7f8f49f58800 5 filestore(/var/lib/ceph/osd/ceph-196) basedir /var/lib/ceph/osd/ceph-196 journal /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.948656 7f8f49f58800 10 filestore(/var/lib/ceph/osd/ceph-196) mount fsid is 8c2fa707-750a-4773-8918-a368367d9cf5 2015-03-25 17:00:17.948704 7f8f49f58800 0 filestore(/var/lib/ceph/osd/ceph-196) mount detected xfs (libxfs) 2015-03-25 17:00:17.948707 7f8f49f58800 1 filestore(/var/lib/ceph/osd/ceph-196) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-03-25 17:00:17.949973 7f8f49f58800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is supported and appears to work 2015-03-25 17:00:17.949982 7f8f49f58800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-25 17:00:17.950236 7f8f49f58800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-25 17:00:17.950268 7f8f49f58800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_feature: extsize is disabled by conf 2015-03-25 17:00:17.950470 7f8f49f58800 5 filestore(/var/lib/ceph/osd/ceph-196) mount op_seq is 1 2015-03-25 17:00:17.951335 7f8f49f58800 20 filestore (init)dbobjectmap: seq is 1 2015-03-25 17:00:17.951346 7f8f49f58800 10 filestore(/var/lib/ceph/osd/ceph-196) open_journal at /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:17.951371 7f8f49f58800 0 filestore(/var/lib/ceph/osd/ceph-196) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-03-25 17:00:17.951377 7f8f49f58800 10 filestore(/var/lib/ceph/osd/ceph-196) list_collections 2015-03-25 17:00:17.951434 7f8f49f58800 10 journal journal_replay fs op_seq 1 2015-03-25 17:00:17.951438 7f8f49f58800 2 journal open /var/lib/ceph/osd/ceph-196/journal fsid 8c2fa707-750a-4773-8918-a368367d9cf5 fs_op_seq 1 2015-03-25 17:00:17.951441 7f8f444f7700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry waiting for max_interval 5.000000 2015-03-25 17:00:17.951457 7f8f49f58800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2015-03-25 17:00:17.976061 7f8f49f58800 1 journal _open /var/lib/ceph/osd/ceph-196/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-25 17:00:17.976077 7f8f49f58800 10 journal read_header 2015-03-25 17:00:17.983299 7f8f49f58800 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2015-03-25 17:00:17.983312 7f8f49f58800 10 journal header: start 50987008 2015-03-25 17:00:17.983326 7f8f49f58800 10 journal write_pos 4096 2015-03-25 17:00:17.983328 7f8f49f58800 10 journal open header.fsid = 942f2d62-dd99-42a8-878a-feea443aaa61 2015-03-25 17:00:17.983333 7f8f49f58800 -1 journal FileJournal::open: ondisk fsid 942f2d62-dd99-42a8-878a-feea443aaa61 doesn't match expected 8c2fa707-750a-4773-8918-a368367d9cf5, invalid (someone else's?) journal 2015-03-25 17:00:17.983349 7f8f49f58800 3 journal journal_replay open failed with (22) Invalid argument 2015-03-25 17:00:17.983374 7f8f49f58800 -1 filestore(/var/lib/ceph/osd/ceph-196) mount failed to open journal /var/lib/ceph/osd/ceph-196/journal: (22) Invalid argument 2015-03-25 17:00:17.983404 7f8f444f7700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry woke after 0.031962 2015-03-25 17:00:17.983419 7f8f444f7700 10 journal commit_start max_applied_seq 1, open_ops 0 2015-03-25 17:00:17.983421 7f8f444f7700 10 journal commit_start blocked, all open_ops have completed 2015-03-25 17:00:17.983422 7f8f444f7700 10 journal commit_start nothing to do 2015-03-25 17:00:17.983532 7f8f49f58800 -1 ** ERROR: error converting store /var/lib/ceph/osd/ceph-196: (22) Invalid argument 2015-03-25 17:00:18.144794 7f4a702f9800 0 ceph version 0.80.9 (b5a67f0e1d15385bc0d60a6da6e7fc810bde6047), process ceph-osd, pid 20582 2015-03-25 17:00:18.147201 7f4a702f9800 10 filestore(/var/lib/ceph/osd/ceph-196) dump_stop 2015-03-25 17:00:18.147285 7f4a702f9800 5 filestore(/var/lib/ceph/osd/ceph-196) basedir /var/lib/ceph/osd/ceph-196 journal /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:18.147331 7f4a702f9800 10 filestore(/var/lib/ceph/osd/ceph-196) mount fsid is 8c2fa707-750a-4773-8918-a368367d9cf5 2015-03-25 17:00:18.147391 7f4a702f9800 0 filestore(/var/lib/ceph/osd/ceph-196) mount detected xfs (libxfs) 2015-03-25 17:00:18.147396 7f4a702f9800 1 filestore(/var/lib/ceph/osd/ceph-196) disabling 'filestore replica fadvise' due to known issues with fadvise(DONTNEED) on xfs 2015-03-25 17:00:18.148768 7f4a702f9800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is supported and appears to work 2015-03-25 17:00:18.148778 7f4a702f9800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: FIEMAP ioctl is disabled via 'filestore fiemap' config option 2015-03-25 17:00:18.149188 7f4a702f9800 0 genericfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_features: syncfs(2) syscall fully supported (by glibc and kernel) 2015-03-25 17:00:18.149256 7f4a702f9800 0 xfsfilestorebackend(/var/lib/ceph/osd/ceph-196) detect_feature: extsize is disabled by conf 2015-03-25 17:00:18.149508 7f4a702f9800 5 filestore(/var/lib/ceph/osd/ceph-196) mount op_seq is 1 2015-03-25 17:00:18.150737 7f4a702f9800 20 filestore (init)dbobjectmap: seq is 1 2015-03-25 17:00:18.150761 7f4a702f9800 10 filestore(/var/lib/ceph/osd/ceph-196) open_journal at /var/lib/ceph/osd/ceph-196/journal 2015-03-25 17:00:18.150804 7f4a702f9800 0 filestore(/var/lib/ceph/osd/ceph-196) mount: enabling WRITEAHEAD journal mode: checkpoint is not enabled 2015-03-25 17:00:18.150809 7f4a702f9800 10 filestore(/var/lib/ceph/osd/ceph-196) list_collections 2015-03-25 17:00:18.150899 7f4a702f9800 10 journal journal_replay fs op_seq 1 2015-03-25 17:00:18.150918 7f4a702f9800 2 journal open /var/lib/ceph/osd/ceph-196/journal fsid 8c2fa707-750a-4773-8918-a368367d9cf5 fs_op_seq 1 2015-03-25 17:00:18.150942 7f4a702f9800 10 journal _open_block_device: ignoring osd journal size. We'll use the entire block device (size: 5367661056) 2015-03-25 17:00:18.150932 7f4a6a898700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry waiting for max_interval 5.000000 2015-03-25 17:00:18.175799 7f4a702f9800 1 journal _open /var/lib/ceph/osd/ceph-196/journal fd 20: 5367660544 bytes, block size 4096 bytes, directio = 1, aio = 1 2015-03-25 17:00:18.175813 7f4a702f9800 10 journal read_header 2015-03-25 17:00:18.182938 7f4a702f9800 10 journal header: block_size 4096 alignment 4096 max_size 5367660544 2015-03-25 17:00:18.182945 7f4a702f9800 10 journal header: start 50987008 2015-03-25 17:00:18.182946 7f4a702f9800 10 journal write_pos 4096 2015-03-25 17:00:18.182948 7f4a702f9800 10 journal open header.fsid = 942f2d62-dd99-42a8-878a-feea443aaa61 2015-03-25 17:00:18.182954 7f4a702f9800 -1 journal FileJournal::open: ondisk fsid 942f2d62-dd99-42a8-878a-feea443aaa61 doesn't match expected 8c2fa707-750a-4773-8918-a368367d9cf5, invalid (someone else's?) journal 2015-03-25 17:00:18.182958 7f4a702f9800 3 journal journal_replay open failed with (22) Invalid argument 2015-03-25 17:00:18.182993 7f4a702f9800 -1 filestore(/var/lib/ceph/osd/ceph-196) mount failed to open journal /var/lib/ceph/osd/ceph-196/journal: (22) Invalid argument 2015-03-25 17:00:18.183034 7f4a6a898700 20 filestore(/var/lib/ceph/osd/ceph-196) sync_entry woke after 0.032101 2015-03-25 17:00:18.183051 7f4a6a898700 10 journal commit_start max_applied_seq 1, open_ops 0 2015-03-25 17:00:18.183053 7f4a6a898700 10 journal commit_start blocked, all open_ops have completed 2015-03-25 17:00:18.183055 7f4a6a898700 10 journal commit_start nothing to do 2015-03-25 17:00:18.183230 7f4a702f9800 -1 ** ERROR: error converting store /var/lib/ceph/osd/ceph-196: (22) Invalid argument
Attachment:
ceph.conf
Description: Binary data
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
