This is pretty confusing. I found the issue that's causing this in 13.2.0 and found that a fix for it was already pushed to master. https://github.com/ceph/ceph/commit/35c0d1f45cd676f201d4031cb8f447f7ea6aee0e and this appears to have been backported to Mimic and Luminous. Is it possible that when you ran `ceph osd crush tree --format=json-pretty` on v14.0.1 that it was only your client that was v14 and it was talking to an older monitor? On Thu, Jan 24, 2019 at 9:08 PM Noah Watkins <noahwatkins@xxxxxxxxx> wrote: > > On Thu, Jan 24, 2019 at 9:07 PM Noah Watkins <noahwatkins@xxxxxxxxx> wrote: > > > > Wow! Sorry, I must have missed something when I was reading through > > the code. It seems like it has to do with the crush configuration. How > > are you setting up the cluster? Maybe you could share the crush map > > Err, ignore! I forgot I was able to reproduce in the other version of > Ceph so it's probably the same issue :) I'll investigate it in more > detail tomorrow! > > - Noah > > > > > On Thu, Jan 24, 2019 at 9:03 PM Liu, Changcheng > > <changcheng.liu@xxxxxxxxx> wrote: > > > > > > Hi Watkins, > > > It could be hit on ceph v14.0.1-2574-gc55dc78903: > > > root@sceph7:~# ceph --version > > > ceph version 14.0.1-2574-gc55dc78903 (c55dc7890316fa1061ce2d34c80270aaa37073c9) nautilus (dev) > > > > > > root@sceph7:~# ceph osd crush tree --format=json-pretty > > > 2019-01-25 20:33:57.630 7f9dc3153700 -1 WARNING: all dangerous and experimental features are enabled. > > > 2019-01-25 20:33:57.650 7f9dc3153700 -1 WARNING: all dangerous and experimental features are enabled. > > > > > > [ > > > { > > > "id": -1, > > > "name": "default", > > > "type": "root", > > > "type_id": 10, > > > "children": [ > > > -2 > > > ] > > > }, > > > { > > > "id": -2, > > > "name": "sceph7", > > > "type": "host", > > > "type_id": 1, > > > "pool_weights": {}, > > > "children": [ > > > 0 > > > ] > > > }, > > > { > > > "id": 0, > > > "device_class": "ssd", > > > "name": "osd.0", > > > "type": "osd", > > > "type_id": 0, > > > "crush_weight": 1.000000, > > > "depth": 2, > > > "pool_weights": {} > > > } > > > ] > > > [] > > > > > > B.R. > > > Changcheng > > > > > > On 19:59 Thu 24 Jan, Noah Watkins wrote: > > > > The output I posted was from v13.2.0. Sorry that wasn't clear. It > > > > should be fixed in 13.2.4. Please let me know if you find otherwise. > > > > > > > > On Thu, Jan 24, 2019 at 5:28 PM Liu, Changcheng > > > > <changcheng.liu@xxxxxxxxx> wrote: > > > > > > > > > > Hi Watkins, > > > > > You can use json-pretty format to show "ceph osd crush tree" result. > > > > > According to your result on 13.2.4, it's not fixed yet. > > > > > Re-format your result below. > > > > > > > > > > On 10:39 Thu 24 Jan, Noah Watkins wrote: > > > > > > This seems to have been fixed in at least v13.2.4. I was able to > > > > > > reproduce it in 13.2.0. > > > > > > > > > > > > [root@minikube /]# ceph osd crush tree --format=json > > > > > > [{"id":-1,"name":"default","type":"root","type_id":10,"children":[-2]},{"id":-2,"name":"minikube","type":"host","type_id":1,"pool_weights":{},"children":[0]},{"id":0,"device_class":"hdd","name":"osd.0","type":"osd","type_id":0,"crush_weight":0.014999,"depth":2,"pool_weights":{}}][][root@minikube > > > > > > /] > > > > > > > > > > [ > > > > > {"id":-1,"name":"default","type":"root","type_id":10,"children":[-2]}, > > > > > {"id":-2,"name":"minikube","type":"host","type_id":1,"pool_weights":{},"children":[0]}, > > > > > {"id":0,"device_class":"hdd","name":"osd.0","type":"osd","type_id":0,"crush_weight":0.014999,"depth":2,"pool_weights":{}} > > > > > ] > > > > > [] > > > > > ==> buggy null list in the end > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 24, 2019 at 5:16 AM Alfredo Deza <adeza@xxxxxxxxxx> wrote: > > > > > > > > > > > > > > On Thu, Jan 24, 2019 at 7:27 AM Liu, Changcheng > > > > > > > <changcheng.liu@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > Hi Watkins, > > > > > > > > controller-0:~$ ceph --version > > > > > > > > ceph version 13.2.2 (02899bfda814146b021136e9d8e80eba494e1126) mimic (stable) > > > > > > > > controller-0:~$ ceph osd crush tree --format=json-pretty > > > > > > > > > > > > > > > > [ > > > > > > > > { > > > > > > > > "id": -1, > > > > > > > > "name": "storage-tier", > > > > > > > > "type": "root", > > > > > > > > "type_id": 10, > > > > > > > > "children": [ > > > > > > > > -3 > > > > > > > > ] > > > > > > > > }, > > > > > > > > { > > > > > > > > "id": -3, > > > > > > > > "name": "group-0", > > > > > > > > "type": "chassis", > > > > > > > > "type_id": 2, > > > > > > > > "pool_weights": {}, > > > > > > > > "children": [ > > > > > > > > -5, > > > > > > > > -4 > > > > > > > > ] > > > > > > > > }, > > > > > > > > { > > > > > > > > "id": -4, > > > > > > > > "name": "storage-0", > > > > > > > > "type": "host", > > > > > > > > "type_id": 1, > > > > > > > > "pool_weights": {}, > > > > > > > > "children": [] > > > > > > > > }, > > > > > > > > { > > > > > > > > "id": -5, > > > > > > > > "name": "storage-1", > > > > > > > > "type": "host", > > > > > > > > "type_id": 1, > > > > > > > > "pool_weights": {}, > > > > > > > > "children": [] > > > > > > > > } > > > > > > > > ] > > > > > > > > [] > > > > > > > > > > > > > > > > B.R. > > > > > > > > Changcheng > > > > > > > > > > > > > > > > On 16:27 Wed 23 Jan, Noah Watkins wrote: > > > > > > > > > On Tue, Jan 22, 2019 at 8:54 PM Liu, Changcheng > > > > > > > > > <changcheng.liu@xxxxxxxxx> wrote: > > > > > > > > > > > > > > > > > > > > 2) result: > > > > > > > > > > [{"id":-1,"name":"default","type":"root","type_id":10,"children":[]}][] > > > > > > > > > > > > > > > > > > What version of Ceph are you using? Also.. is that the entire command > > > > > > > > > output? For example, on the latest version I would expect there to be > > > > > > > > > a "nodes" item: > > > > > > > > > > > > > > Seems like this has been the case for a while, I see it too on an > > > > > > > older 13.2.0 version. > > > > > > > > > > > > > > > > > > > > > > > > > { > > > > > > > > > "nodes": [ > > > > > > > > > { > > > > > > > > > "id": -1, > > > > > > > > > "name": "default", > > > > > > > > > "type": "root", > > > > > > > > > "type_id": 10, > > > > > > > > > "children": [ > > > > > > > > > -3 > > > > > > > > > ] > > > > > > > > > ... > > > > > > > > > ], > > > > > > > > > "stray": [] > > > > > > > > > }
