On 13/07/14 18:38, Andrija Panic wrote: > Hi Mark, > actually, CEPH is running fine, and I have deployed NEW host (new > compile libvirt with ceph 0.8 devel, and newer kernel) - and it works... > so migrating some VMs to this new host... > > I have 3 physical hosts, that are both MON and 2x OSD per host, all3 > don't work-cloudstack/libvirt... > > Any suggestion on need to recompile libvirt ? I got info from Wido, that > libvirt does NOT need to be recompiled.... > Looking at the differences between src/include/ceph_features.h in 0.72 and 0.81 [1] (note, not *quite* the same version as you are using), there's erasure codes and other new features that are advertised by the later version that the client will need to match. Now *some* of these (crush tunables) can be switched off via: $ ceph osd crush tunables legacy ...which would have been worth a try, but my guess is would not have worked, as (for example) I *don't* think erasure codes feature can be switched off. Hence, unless I'm mistaken (which is always possible) I think you did in fact need to recompile. regards Mark [1] e.g: --- ceph_features.h.72 2014-07-13 19:00:36.805825203 +1200 +++ ceph_features.h.81 2014-07-13 19:02:22.065826068 +1200 @@ -40,6 +40,18 @@ #define CEPH_FEATURE_MON_SCRUB (1ULL<<33) #define CEPH_FEATURE_OSD_PACKED_RECOVERY (1ULL<<34) #define CEPH_FEATURE_OSD_CACHEPOOL (1ULL<<35) +#define CEPH_FEATURE_CRUSH_V2 (1ULL<<36) /* new indep; SET_* steps */ +#define CEPH_FEATURE_EXPORT_PEER (1ULL<<37) +#define CEPH_FEATURE_OSD_ERASURE_CODES (1ULL<<38) +#define CEPH_FEATURE_OSD_TMAP2OMAP (1ULL<<38) /* overlap with EC */ +/* The process supports new-style OSDMap encoding. Monitors also use + this bit to determine if peers support NAK messages. */ +#define CEPH_FEATURE_OSDMAP_ENC (1ULL<<39) +#define CEPH_FEATURE_MDS_INLINE_DATA (1ULL<<40) +#define CEPH_FEATURE_CRUSH_TUNABLES3 (1ULL<<41) +#define CEPH_FEATURE_OSD_PRIMARY_AFFINITY (1ULL<<41) /* overlap w/ tunables3 */ +#define CEPH_FEATURE_MSGR_KEEPALIVE2 (1ULL<<42) +#define CEPH_FEATURE_OSD_POOLRESEND (1ULL<<43) /* * The introduction of CEPH_FEATURE_OSD_SNAPMAPPER caused the feature @@ -102,7 +114,16 @@ CEPH_FEATURE_OSD_SNAPMAPPER | \ CEPH_FEATURE_MON_SCRUB | \ CEPH_FEATURE_OSD_PACKED_RECOVERY | \ - CEPH_FEATURE_OSD_CACHEPOOL | \ + CEPH_FEATURE_OSD_CACHEPOOL | \ + CEPH_FEATURE_CRUSH_V2 | \ + CEPH_FEATURE_EXPORT_PEER | \ + CEPH_FEATURE_OSD_ERASURE_CODES | \ + CEPH_FEATURE_OSDMAP_ENC | \ + CEPH_FEATURE_MDS_INLINE_DATA | \ + CEPH_FEATURE_CRUSH_TUNABLES3 | \ + CEPH_FEATURE_OSD_PRIMARY_AFFINITY | \ + CEPH_FEATURE_MSGR_KEEPALIVE2 | \ + CEPH_FEATURE_OSD_POOLRESEND | \ 0ULL) #define CEPH_FEATURES_SUPPORTED_DEFAULT CEPH_FEATURES_ALL @@ -112,6 +133,8 @@ */ #define CEPH_FEATURES_CRUSH \ (CEPH_FEATURE_CRUSH_TUNABLES | \ - CEPH_FEATURE_CRUSH_TUNABLES2) + CEPH_FEATURE_CRUSH_TUNABLES2 | \ + CEPH_FEATURE_CRUSH_TUNABLES3 | \ + CEPH_FEATURE_CRUSH_V2) #endif
 |