On Thu, 29 Jun 2017, Sage Weil wrote: > On Thu, 29 Jun 2017, Mark Kirkwood wrote: > > Hi, > > > > I'm running a 4 node test 'cluster' (VMs on my workstation) that I've upgraded > > to Luminous RC. Specifically I wanted to test having each node with 1 spinning > > device and one solid state state so I could try out device classes to create > > fast and slow(er) pools. > > > > I started with 4 filestore osds (comimg from the Jewel pre-upgrade), and added > > 4 more, all of which were Bluestore on the ssds. > > > > I used crushtool to set the device classes (see crush test diff below). > > > > That all went very smoothly, with only a couple of things that seemed weird. > > Firstly the crush/osd tree output is a bit strange (but I could get to the > > point where it make sense): > > > > $ sudo ceph osd tree > > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY > > -15 0.23196 root default~ssd > > -11 0.05699 host ceph1~ssd > > 4 0.05699 osd.4 up 1.00000 1.00000 > > -12 0.05899 host ceph2~ssd > > 5 0.05899 osd.5 up 1.00000 1.00000 > > -13 0.05699 host ceph3~ssd > > 6 0.05699 osd.6 up 1.00000 1.00000 > > -14 0.05899 host ceph4~ssd > > 7 0.05899 osd.7 up 1.00000 1.00000 > > -10 0.07996 root default~hdd > > -6 0.01999 host ceph1~hdd > > 0 0.01999 osd.0 up 1.00000 1.00000 > > -7 0.01999 host ceph2~hdd > > 1 0.01999 osd.1 up 1.00000 1.00000 > > -8 0.01999 host ceph3~hdd > > 2 0.01999 osd.2 up 1.00000 1.00000 > > -9 0.01999 host ceph4~hdd > > 3 0.01999 osd.3 up 1.00000 1.00000 > > -1 0.31198 root default > > -2 0.07700 host ceph1 > > 0 0.01999 osd.0 up 1.00000 1.00000 > > 4 0.05699 osd.4 up 1.00000 1.00000 > > -3 0.07899 host ceph2 > > 1 0.01999 osd.1 up 1.00000 1.00000 > > 5 0.05899 osd.5 up 1.00000 1.00000 > > -4 0.07700 host ceph3 > > 2 0.01999 osd.2 up 1.00000 1.00000 > > 6 0.05699 osd.6 up 1.00000 1.00000 > > -5 0.07899 host ceph4 > > 3 0.01999 osd.3 up 1.00000 1.00000 > > 7 0.05899 osd.7 up 1.00000 1.00000 > > I was a bit divided when we were doing this about whether the unfiltered > (above) output or a view that hides the per-class view is better. Maybe > > ceph osd tree > > would show the traditional view (with a device class column) and > > ceph osd class-tree <class> > > would show a single class? For now, see https://github.com/ceph/ceph/pull/16016 - Do not show each per-class variation of the hieararchy - Do include a CLASS column in the tree view This is still somewhat incomplete in that the MIN/MAX/VAR values are overall and not per-class, which makes them less useful if you are actually using the classes. sage > > > But the osd df output is baffling, I've got two identical lines for each osd > > (hard to see immediately - sorting by osd id would make it easier). This is > > not ideal, particularly as for the bluestore guys there is no other way to > > work out utilization. Any ideas - have I done something obviously wrong here > > that is triggering the 2 lines? > > > > $ sudo ceph osd df > > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR PGS > > 4 0.05699 1.00000 60314M 1093M 59221M 1.81 1.27 0 > > 5 0.05899 1.00000 61586M 1234M 60351M 2.00 1.40 0 > > 6 0.05699 1.00000 60314M 1248M 59066M 2.07 1.45 0 > > 7 0.05899 1.00000 61586M 1209M 60376M 1.96 1.37 0 > > 0 0.01999 1.00000 25586M 43812k 25543M 0.17 0.12 45 > > 1 0.01999 1.00000 25586M 42636k 25544M 0.16 0.11 37 > > 2 0.01999 1.00000 25586M 44336k 25543M 0.17 0.12 53 > > 3 0.01999 1.00000 25586M 42716k 25544M 0.16 0.11 57 > > 0 0.01999 1.00000 25586M 43812k 25543M 0.17 0.12 45 > > 4 0.05699 1.00000 60314M 1093M 59221M 1.81 1.27 0 > > 1 0.01999 1.00000 25586M 42636k 25544M 0.16 0.11 37 > > 5 0.05899 1.00000 61586M 1234M 60351M 2.00 1.40 0 > > 2 0.01999 1.00000 25586M 44336k 25543M 0.17 0.12 53 > > 6 0.05699 1.00000 60314M 1248M 59066M 2.07 1.45 0 > > 3 0.01999 1.00000 25586M 42716k 25544M 0.16 0.11 57 > > 7 0.05899 1.00000 61586M 1209M 60376M 1.96 1.37 0 > > TOTAL 338G 4955M 333G 1.43 > > MIN/MAX VAR: 0.11/1.45 STDDEV: 0.97 > > This is just a bug, fixing. > > Thanks! > sage > > > > > > > > > The modifications to crush map > > --- crush.txt.orig 2017-06-28 14:38:38.067669000 +1200 > > +++ crush.txt 2017-06-28 14:41:22.071669000 +1200 > > @@ -8,14 +8,14 @@ > > tunable allowed_bucket_algs 54 > > > > # devices > > -device 0 osd.0 > > -device 1 osd.1 > > -device 2 osd.2 > > -device 3 osd.3 > > -device 4 osd.4 > > -device 5 osd.5 > > -device 6 osd.6 > > -device 7 osd.7 > > +device 0 osd.0 class hdd > > +device 1 osd.1 class hdd > > +device 2 osd.2 class hdd > > +device 3 osd.3 class hdd > > +device 4 osd.4 class ssd > > +device 5 osd.5 class ssd > > +device 6 osd.6 class ssd > > +device 7 osd.7 class ssd > > > > # types > > type 0 osd > > @@ -80,7 +80,7 @@ > > type replicated > > min_size 1 > > max_size 10 > > - step take default > > + step take default class hdd > > step chooseleaf firstn 0 type host > > step emit > > } > > > > -- > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > the body of a message to majordomo@xxxxxxxxxxxxxxx > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
