Any Ceph krbd logs in dmesg? Given that this is a Jewel cluster on an older kernel, my blind guess is that your CRUSH map is using newer features not supported by your kernel. If that is the case, there should be a log message. -- Jason Dillaman ----- Original Message ----- > From: "German Anders" <ganders@xxxxxxxxxxxx> > To: "Jason Dillaman" <dillaman@xxxxxxxxxx> > Cc: "Samuel Just" <sjust@xxxxxxxxxx>, "ceph-users" > <ceph-users@xxxxxxxxxxxxxx> > Sent: Tuesday, March 29, 2016 4:50:14 PM > Subject: Re: Scrubbing a lot > Jason, I try that but the mapping is not working anyway: > # rbd --cluster cephIB map e60host02 --pool cinder-volumes -k > /etc/ceph/cephIB.client.cinder.keyring > rbd: sysfs write failed > rbd: map failed: (5) Input/output error > German > 2016-03-29 17:46 GMT-03:00 Jason Dillaman < dillaman@xxxxxxxxxx > : > > Running the following should fix your image up for krbd usage: > > > # rbd --cluster cephIB feature disable e60host02 > > exclusive-lock,object-map,fast-diff,deep-flatten --pool cinder-volumes > > > In the future, you can create krbd-compatible images by adding > > "--image-feature layering" to the "rbd create" command-line (or by updating > > your config as documented in the release notes). > > > -- > > > Jason Dillaman > > > ----- Original Message ----- > > > > From: "German Anders" < ganders@xxxxxxxxxxxx > > > > > To: "Jason Dillaman" < dillaman@xxxxxxxxxx > > > > > Cc: "Samuel Just" < sjust@xxxxxxxxxx >, "ceph-users" > > > > < ceph-users@xxxxxxxxxxxxxx > > > > > Sent: Tuesday, March 29, 2016 4:38:02 PM > > > > Subject: Re: Scrubbing a lot > > > > # rbd --cluster cephIB info e60host02 --pool cinder-volumes > > > > rbd image 'e60host02': > > > > size 102400 MB in 25600 objects > > > > order 22 (4096 kB objects) > > > > block_name_prefix: rbd_data.5ef1238e1f29 > > > > format: 2 > > > > features: layering, exclusive-lock, object-map, fast-diff, deep-flatten > > > > flags: > > > > German > > > > 2016-03-29 17:36 GMT-03:00 Jason Dillaman < dillaman@xxxxxxxxxx > : > > > > > Under Jewel, newly created images default to features that are not > > > > > currently > > > > > compatible with krbd. If you run 'rbd --cluster cephIB info host01 > > > > --pool > > > > > cinder-volumes', what features do you see? If you see more than > > > > layering, > > > > > you need to disable them via the 'rbd feature disable' command. > > > > > > > > > [1] https://github.com/ceph/ceph/blob/master/doc/release-notes.rst#L302 > > > > > > > > > -- > > > > > > > > > Jason Dillaman > > > > > > > > > ----- Original Message ----- > > > > > > > > > > From: "German Anders" < ganders@xxxxxxxxxxxx > > > > > > > > > > > To: "Samuel Just" < sjust@xxxxxxxxxx > > > > > > > > > > > Cc: "ceph-users" < ceph-users@xxxxxxxxxxxxxx > > > > > > > > > > > Sent: Tuesday, March 29, 2016 4:19:03 PM > > > > > > > > > > Subject: Re: Scrubbing a lot > > > > > > > > > > I've just upgrade to jewel , and the scrubbing seems to been > > > > > corrected... > > > > > > but > > > > > > > > > > now I'm not able to map an rbd on a host (before I was able to), > > > > > > basically > > > > > > > > > > I'm getting this error msg: > > > > > > > > > > rbd: sysfs write failed > > > > > > > > > > rbd: map failed: (5) Input/output error > > > > > > > > > > # rbd --cluster cephIB create host01 --size 102400 --pool > > > > > cinder-volumes > > > > > > -k > > > > > > > > > > /etc/ceph/cephIB.client.cinder.keyring > > > > > > > > > > # rbd --cluster cephIB map host01 --pool cinder-volumes -k > > > > > > > > > > /etc/ceph/cephIB.client.cinder.keyring > > > > > > > > > > rbd: sysfs write failed > > > > > > > > > > rbd: map failed: (5) Input/output error > > > > > > > > > > Any ideas? on the /etc/ceph directory on the host I've: > > > > > > > > > > -rw-r--r-- 1 ceph ceph 92 Nov 17 15:45 rbdmap > > > > > > > > > > -rw-r--r-- 1 ceph ceph 170 Dec 15 14:47 secret.xml > > > > > > > > > > -rw-r--r-- 1 ceph ceph 37 Dec 15 15:12 virsh-secret > > > > > > > > > > -rw-r--r-- 1 ceph ceph 0 Dec 15 15:12 virsh-secret-set > > > > > > > > > > -rw-r--r-- 1 ceph ceph 37 Dec 21 14:53 virsh-secretIB > > > > > > > > > > -rw-r--r-- 1 ceph ceph 0 Dec 21 14:53 virsh-secret-setIB > > > > > > > > > > -rw-r--r-- 1 ceph ceph 173 Dec 22 13:34 secretIB.xml > > > > > > > > > > -rw-r--r-- 1 ceph ceph 619 Dec 22 13:38 ceph.conf > > > > > > > > > > -rw-r--r-- 1 ceph ceph 72 Dec 23 09:51 ceph.client.cinder.keyring > > > > > > > > > > -rw-r--r-- 1 ceph ceph 63 Mar 28 09:03 cephIB.client.cinder.keyring > > > > > > > > > > -rw-r--r-- 1 ceph ceph 526 Mar 28 12:06 cephIB.conf > > > > > > > > > > -rw------- 1 ceph ceph 63 Mar 29 16:11 cephIB.client.admin.keyring > > > > > > > > > > Thanks in advance, > > > > > > > > > > Best, > > > > > > > > > > German > > > > > > > > > > 2016-03-29 14:45 GMT-03:00 German Anders < ganders@xxxxxxxxxxxx > : > > > > > > > > > > > Sure, also the scrubbing is happening on all the osds :S > > > > > > > > > > > > > > > > > > > > > # ceph --cluster cephIB daemon osd.4 config diff > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > > > > > > > > > > > > > "diff": { > > > > > > > > > > > > > > > > > > > > > "current": { > > > > > > > > > > > > > > > > > > > > > "admin_socket": "\/var\/run\/ceph\/cephIB-osd.4.asok", > > > > > > > > > > > > > > > > > > > > > "auth_client_required": "cephx", > > > > > > > > > > > > > > > > > > > > > "filestore_fd_cache_size": "10240", > > > > > > > > > > > > > > > > > > > > > "filestore_journal_writeahead": "true", > > > > > > > > > > > > > > > > > > > > > "filestore_max_sync_interval": "10", > > > > > > > > > > > > > > > > > > > > > "filestore_merge_threshold": "40", > > > > > > > > > > > > > > > > > > > > > "filestore_op_threads": "20", > > > > > > > > > > > > > > > > > > > > > "filestore_queue_max_ops": "100000", > > > > > > > > > > > > > > > > > > > > > "filestore_split_multiple": "8", > > > > > > > > > > > > > > > > > > > > > "fsid": "a4bce51b-4d6b-4394-9737-3e4d9f5efed2", > > > > > > > > > > > > > > > > > > > > > "internal_safe_to_start_threads": "true", > > > > > > > > > > > > > > > > > > > > > "keyring": "\/var\/lib\/ceph\/osd\/cephIB-4\/keyring", > > > > > > > > > > > > > > > > > > > > > "leveldb_log": "", > > > > > > > > > > > > > > > > > > > > > "log_file": "\/var\/log\/ceph\/cephIB-osd.4.log", > > > > > > > > > > > > > > > > > > > > > "log_to_stderr": "false", > > > > > > > > > > > > > > > > > > > > > "mds_data": "\/var\/lib\/ceph\/mds\/cephIB-4", > > > > > > > > > > > > > > > > > > > > > "mon_cluster_log_file": > > > > > > "default=\/var\/log\/ceph\/cephIB.$channel.log > > > > > > > > > > > cluster=\/var\/log\/ceph\/cephIB.log", > > > > > > > > > > > > > > > > > > > > > "mon_data": "\/var\/lib\/ceph\/mon\/cephIB-4", > > > > > > > > > > > > > > > > > > > > > "mon_debug_dump_location": "\/var\/log\/ceph\/cephIB-osd.4.tdump", > > > > > > > > > > > > > > > > > > > > > "mon_host": "172.23.16.1,172.23.16.2,172.23.16.3", > > > > > > > > > > > > > > > > > > > > > "mon_initial_members": "cibm01, cibm02, cibm03", > > > > > > > > > > > > > > > > > > > > > "osd_data": "\/var\/lib\/ceph\/osd\/cephIB-4", > > > > > > > > > > > > > > > > > > > > > "osd_journal": "\/var\/lib\/ceph\/osd\/cephIB-4\/journal", > > > > > > > > > > > > > > > > > > > > > "osd_op_threads": "8", > > > > > > > > > > > > > > > > > > > > > "rgw_data": "\/var\/lib\/ceph\/radosgw\/cephIB-4", > > > > > > > > > > > > > > > > > > > > > "setgroup": "ceph", > > > > > > > > > > > > > > > > > > > > > "setuser": "ceph" > > > > > > > > > > > > > > > > > > > > > }, > > > > > > > > > > > > > > > > > > > > > "defaults": { > > > > > > > > > > > > > > > > > > > > > "admin_socket": "\/var\/run\/ceph\/ceph-osd.4.asok", > > > > > > > > > > > > > > > > > > > > > "auth_client_required": "cephx, none", > > > > > > > > > > > > > > > > > > > > > "filestore_fd_cache_size": "128", > > > > > > > > > > > > > > > > > > > > > "filestore_journal_writeahead": "false", > > > > > > > > > > > > > > > > > > > > > "filestore_max_sync_interval": "5", > > > > > > > > > > > > > > > > > > > > > "filestore_merge_threshold": "10", > > > > > > > > > > > > > > > > > > > > > "filestore_op_threads": "2", > > > > > > > > > > > > > > > > > > > > > "filestore_queue_max_ops": "50", > > > > > > > > > > > > > > > > > > > > > "filestore_split_multiple": "2", > > > > > > > > > > > > > > > > > > > > > "fsid": "00000000-0000-0000-0000-000000000000", > > > > > > > > > > > > > > > > > > > > > "internal_safe_to_start_threads": "false", > > > > > > > > > > > > > > > > > > > > > "keyring": > > > > > > > > > > > "\/etc\/ceph\/ceph.osd.4.keyring,\/etc\/ceph\/ceph.keyring,\/etc\/ceph\/keyring,\/etc\/ceph\/keyring.bin", > > > > > > > > > > > > > > > > > > > > > "leveldb_log": "\/dev\/null", > > > > > > > > > > > > > > > > > > > > > "log_file": "\/var\/log\/ceph\/ceph-osd.4.log", > > > > > > > > > > > > > > > > > > > > > "log_to_stderr": "true", > > > > > > > > > > > > > > > > > > > > > "mds_data": "\/var\/lib\/ceph\/mds\/ceph-4", > > > > > > > > > > > > > > > > > > > > > "mon_cluster_log_file": > > > > > > "default=\/var\/log\/ceph\/ceph.$channel.log > > > > > > > > > > > cluster=\/var\/log\/ceph\/ceph.log", > > > > > > > > > > > > > > > > > > > > > "mon_data": "\/var\/lib\/ceph\/mon\/ceph-4", > > > > > > > > > > > > > > > > > > > > > "mon_debug_dump_location": "\/var\/log\/ceph\/ceph-osd.4.tdump", > > > > > > > > > > > > > > > > > > > > > "mon_host": "", > > > > > > > > > > > > > > > > > > > > > "mon_initial_members": "", > > > > > > > > > > > > > > > > > > > > > "osd_data": "\/var\/lib\/ceph\/osd\/ceph-4", > > > > > > > > > > > > > > > > > > > > > "osd_journal": "\/var\/lib\/ceph\/osd\/ceph-4\/journal", > > > > > > > > > > > > > > > > > > > > > "osd_op_threads": "2", > > > > > > > > > > > > > > > > > > > > > "rgw_data": "\/var\/lib\/ceph\/radosgw\/ceph-4", > > > > > > > > > > > > > > > > > > > > > "setgroup": "", > > > > > > > > > > > > > > > > > > > > > "setuser": "" > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > }, > > > > > > > > > > > > > > > > > > > > > "unknown": [] > > > > > > > > > > > > > > > > > > > > > } > > > > > > > > > > > > > > > > > > > > > Thanks a lot! > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > German > > > > > > > > > > > > > > > > > > > > > 2016-03-29 14:10 GMT-03:00 Samuel Just < sjust@xxxxxxxxxx > : > > > > > > > > > > > > > > > > > > > > > > That seems to be scrubbing pretty often. Can you attach a config > > > > > > > diff > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > from osd.4 (ceph daemon osd.4 config diff)? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -Sam > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Mar 29, 2016 at 9:30 AM, German Anders < > > > > > > > ganders@xxxxxxxxxxxx > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've maybe a simple question, I've setup a new cluster with > > > > > > > > > Infernalis > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > release, there's no IO going on at the cluster level and I'm > > > > > > > > > receiving > > > > > > > > > > > > > a > > > > > > > > > > > > > lot > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > of these messages: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:07.462818 mon.0 [INF] pgmap v158062: 8192 pgs: > > > > > > > > > 8192 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > active+clean; 20617 MB data, 46164 MB used, 52484 GB / 52529 GB > > > > > > > > > avail > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:08.176684 osd.13 [INF] 0.d38 scrub starts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:08.179841 osd.13 [INF] 0.d38 scrub ok > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:21:59.526355 osd.9 [INF] 0.8a6 scrub starts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:21:59.529582 osd.9 [INF] 0.8a6 scrub ok > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:03.004107 osd.4 [INF] 0.38b scrub starts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:03.007220 osd.4 [INF] 0.38b scrub ok > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:03.617706 osd.21 [INF] 0.525 scrub starts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:03.621073 osd.21 [INF] 0.525 scrub ok > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:06.527264 osd.9 [INF] 0.8a6 scrub starts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:06.529150 osd.9 [INF] 0.8a6 scrub ok > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:07.005628 osd.4 [INF] 0.38b scrub starts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:07.009776 osd.4 [INF] 0.38b scrub ok > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:07.618191 osd.21 [INF] 0.525 scrub starts > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 2016-03-29 12:22:07.621363 osd.21 [INF] 0.525 scrub ok > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I mean, all the time, and AFAIK these is because the scrub > > > > > > > > > operation > > > > > > > > > is > > > > > > > > > > > > > like > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > an fsck on the object level, so this make me think that it's > > > > > > > > not > > > > > > > > a > > > > > > > > > > > > > normal > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > situation. Is there any command that I can run in order to > > > > > > > > check > > > > > > > > > this? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > # ceph --cluster cephIB health detail > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > HEALTH_OK > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > German > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ceph-users mailing list > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ceph-users@xxxxxxxxxxxxxx > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > > > ceph-users mailing list > > > > > > > > > > ceph-users@xxxxxxxxxxxxxx > > > > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |