On Fri, 16 Oct 2015, Robert LeBlanc wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> I started another fio test to one of the same RBDs (leaving the hung
> ones still hung) and it is working OK, but the hungs ones are still
> just hung.
There is a full-disk failsafe that is still somewhat buggy that could
explain the hung requests (if they were writes and submitted while the
osd(s) were near full).
sage
> - ----------------
> Robert LeBlanc
> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
>
>
> On Fri, Oct 16, 2015 at 10:00 AM, Robert LeBlanc wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA256
> >
> > OK, I've set this up and now all I/O is locked up. I've reduced
> > target_max_bytes because one OSD was reporting 97% usage, there was
> > some I/O for a few seconds as things flushed, but client I/O is still
> > blocked. Anyone have some thoughts?
> >
> > ceph osd crush rule create-simple ssd-tier ssd host firstn
> > ceph osd pool create ssd-pool 128 replicated ssd-tier
> > ceph osd tier add rbd ssd-pool
> > ceph osd tier cache-mode ssd-pool writeback
> > ceph osd tier set-overlay rbd ssd-pool
> > ceph osd pool set ssd-pool hit_set_type bloom
> > ceph osd pool set ssd-pool hit_set_count 6
> > ceph osd pool set ssd-pool hit_set_period 600
> > ceph osd pool set ssd-pool min_read_recency_for_promote 6
> > ceph osd pool set ssd-pool cache_target_dirty_ratio 0.4
> > ceph osd pool set ssd-pool cache_target_full_ratio 0.8
> > ceph osd pool set ssd-pool target_max_bytes 795642691584
> >
> > ceph version 0.94.3-252-g629b631 (629b631488f044150422371ac77dfc005f3de1bc)
> >
> > # ceph status
> > cluster 050309fd-723e-42aa-9624-3b3e033ab359
> > health HEALTH_OK
> > monmap e1: 1 mons at {nodez=192.168.55.15:6789/0}
> > election epoch 2, quorum 0 nodez
> > osdmap e1333: 18 osds: 18 up, 18 in
> > pgmap v87157: 384 pgs, 2 pools, 3326 GB data, 1368 kobjects
> > 3010 GB used, 20262 GB / 24518 GB avail
> > 384 active+clean
> >
> > # ceph osd df
> > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR
> > 18 0.20000 1.00000 208G 135G 64764M 64.63 4.77
> > 5 0.20999 1.00000 210G 181G 18392M 86.36 6.38
> > 19 0.21999 1.00000 208G 161G 37941M 77.17 5.70
> > 10 0.18999 1.00000 210G 167G 32712M 79.70 5.89
> > 7 0.20999 1.00000 210G 181G 18405M 86.35 6.38
> > 20 0.20000 1.00000 208G 119G 80247M 57.39 4.24
> > 22 0.20000 1.00000 208G 87596M 112G 40.95 3.02
> > 8 0.20999 1.00000 210G 170G 29422M 81.23 6.00
> > 23 0.20999 1.00000 208G 151G 47404M 72.75 5.37
> > 1 0.20999 1.00000 210G 105G 96245M 50.17 3.71
> > 6 0.20999 1.00000 210G 131G 69937M 62.40 4.61
> > 21 0.20000 1.00000 208G 192G 5667M 92.26 6.81
> > 0 3.64000 1.00000 3667G 231G 3249G 6.32 0.47
> > 9 3.57999 1.00000 3667G 262G 3219G 7.15 0.53
> > 2 3.64000 1.00000 3667G 273G 3207G 7.47 0.55
> > 3 3.64000 1.00000 3667G 256G 3224G 6.99 0.52
> > 4 3.64000 1.00000 3667G 239G 3241G 6.54 0.48
> > 24 3.57999 1.00000 3667G 272G 3208G 7.42 0.55
> > TOTAL 24518G 3320G 19952G 13.54
> > MIN/MAX VAR: 0.47/6.81 STDDEV: 48.64
> >
> > After dropping target_max_bytes to 644470580183:
> > # ceph df
> > GLOBAL:
> > SIZE AVAIL RAW USED %RAW USED
> > 24518G 20241G 3031G 12.36
> > POOLS:
> > NAME ID USED %USED MAX AVAIL OBJECTS
> > rbd 0 2856G 11.65 6379G 1158862
> > ssd-pool 3 470G 1.92 162G 242140
> > # ceph osd df
> > ID WEIGHT REWEIGHT SIZE USE AVAIL %USE VAR
> > 18 0.20000 1.00000 208G 116G 83987M 55.64 4.50
> > 5 0.20999 1.00000 210G 151G 49392M 71.95 5.82
> > 19 0.21999 1.00000 208G 134G 65792M 64.15 5.19
> > 10 0.18999 1.00000 210G 138G 61961M 66.11 5.35
> > 7 0.20999 1.00000 210G 149G 50672M 71.36 5.77
> > 20 0.20000 1.00000 208G 101842M 101167M 47.61 3.85
> > 22 0.20000 1.00000 208G 72511M 127G 33.90 2.74
> > 8 0.20999 1.00000 210G 145G 55381M 69.17 5.59
> > 23 0.20999 1.00000 208G 127G 72305M 61.11 4.94
> > 1 0.20999 1.00000 210G 95656M 105G 44.46 3.60
> > 6 0.20999 1.00000 210G 109G 92154M 52.07 4.21
> > 21 0.20000 1.00000 208G 158G 40521M 75.97 6.14
> > 0 3.64000 1.00000 3667G 231G 3249G 6.32 0.51
> > 9 3.57999 1.00000 3667G 262G 3219G 7.15 0.58
> > 2 3.64000 1.00000 3667G 273G 3207G 7.47 0.60
> > 3 3.64000 1.00000 3667G 256G 3224G 6.99 0.57
> > 4 3.64000 1.00000 3667G 239G 3241G 6.54 0.53
> > 24 3.57999 1.00000 3667G 272G 3208G 7.42 0.60
> > TOTAL 24518G 3031G 20241G 12.36
> > MIN/MAX VAR: 0.51/6.14 STDDEV: 39.87
> >
> > # ceph osd tree
> > ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
> > -9 2.46991 root ssd
> > -8 0.40999 host nodew-ssd
> > 18 0.20000 osd.18 up 1.00000 1.00000
> > 5 0.20999 osd.5 up 1.00000 1.00000
> > - -10 0.40997 host nodev-ssd
> > 19 0.21999 osd.19 up 1.00000 1.00000
> > 10 0.18999 osd.10 up 1.00000 1.00000
> > - -11 0.40999 host nodezz-ssd
> > 7 0.20999 osd.7 up 1.00000 1.00000
> > 20 0.20000 osd.20 up 1.00000 1.00000
> > - -12 0.40999 host nodey-ssd
> > 22 0.20000 osd.22 up 1.00000 1.00000
> > 8 0.20999 osd.8 up 1.00000 1.00000
> > - -13 0.41998 host nodex-ssd
> > 23 0.20999 osd.23 up 1.00000 1.00000
> > 1 0.20999 osd.1 up 1.00000 1.00000
> > - -14 0.40999 host nodez-ssd
> > 6 0.20999 osd.6 up 1.00000 1.00000
> > 21 0.20000 osd.21 up 1.00000 1.00000
> > -1 21.71997 root default
> > -2 3.64000 host nodez
> > 0 3.64000 osd.0 up 1.00000 1.00000
> > -3 3.57999 host nodew
> > 9 3.57999 osd.9 up 1.00000 1.00000
> > -4 3.64000 host nodex
> > 2 3.64000 osd.2 up 1.00000 1.00000
> > -5 3.64000 host nodey
> > 3 3.64000 osd.3 up 1.00000 1.00000
> > -6 3.64000 host nodezz
> > 4 3.64000 osd.4 up 1.00000 1.00000
> > -7 3.57999 host nodev
> > 24 3.57999 osd.24 up 1.00000 1.00000
> >
> > # ceph osd crush rule dump
> > [
> > {
> > "rule_id": 0,
> > "rule_name": "replicated_ruleset",
> > "ruleset": 0,
> > "type": 1,
> > "min_size": 1,
> > "max_size": 10,
> > "steps": [
> > {
> > "op": "take",
> > "item": -1,
> > "item_name": "default"
> > },
> > {
> > "op": "chooseleaf_firstn",
> > "num": 0,
> > "type": "host"
> > },
> > {
> > "op": "emit"
> > }
> > ]
> > },
> > {
> > "rule_id": 1,
> > "rule_name": "ssd-tier",
> > "ruleset": 1,
> > "type": 1,
> > "min_size": 1,
> > "max_size": 10,
> > "steps": [
> > {
> > "op": "take",
> > "item": -9,
> > "item_name": "ssd"
> > },
> > {
> > "op": "chooseleaf_firstn",
> > "num": 0,
> > "type": "host"
> > },
> > {
> > "op": "emit"
> > }
> > ]
> > }
> > ]
> > -----BEGIN PGP SIGNATURE-----
> > Version: Mailvelope v1.2.0
> > Comment: https://www.mailvelope.com
> >
> > wsFcBAEBCAAQBQJWIR8JCRDmVDuy+mK58QAAhyAP/3LYWWxCtDUABwzW/rov
> > 5NCHpKgVRkEAUTGIRESFp9egbhr2loaC1pjfkp911Shg6My/C3N6Y9q9MLdq
> > zy7zGSB/GL5XjvS0TurEjBihtDpMF2SbBk5NkrzgVc1fiOuA8UEZl8J2wBtF
> > R81UOluZVULzvmMjbH4uWfD1UovJl30LlAz/MocDJsDDejjfnsM3PXn8NSaE
> > 4AyNkj8tXj8yMZIzxZV25O8NWZXq0JnuOwND+YxT9VxG8k1o3gqg7747j/Uz
> > 0A9/fJ4IkMJdNGyMCVPgoTJy87CjeSfDf0MmK3S5bXtLfKKZTKYv0m/+B8PY
> > KzZcuVTavBhFSLWiT3L2U1OOyPz5AEu2ezE2Y6ElFePc+g38eO/I7kuTSixV
> > +0yZL1tO6vEYZLnwWTWgYFmmrOA5yTBvssGpjpZVPe7swkJG97kvqe/bh2/W
> > OqQ5PEnhn5Gx3vIDHJwvI/PT4MXZk2VU9cpPMPs7PeIQBPZYPi0/WcfT8m+g
> > oclkznsM+BSLMiTT8yBc7/T1kLFQXS42jVXEFAKYnJj8LIk0aMc54Gu25g0w
> > PM6+IFROsMQlGdybbWCPXIXsZ94JjJOBbA3jSP7XkesNvNC9fqlRDJwxBS7h
> > 2F4cUwpZRJZGSAJzIRbbFdDZOftoUjtIiv+GAH1z54o+lq/sR+WNo1ALTB8k
> > uNQ8
> > =z47G
> > -----END PGP SIGNATURE-----
> > ----------------
> > Robert LeBlanc
> > PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> >
> >
> > On Thu, Oct 15, 2015 at 5:49 PM, Christian Balzer wrote:
> >>
> >> Hello,
> >>
> >> Having run into this myself two days ago (setting relative sizing values
> >> doesn't flush things when expected) I'd say that the documentation is
> >> highly misleading when it comes to the relative settings.
> >>
> >> And unclear when it comes to the size/object settings.
> >>
> >> Guess this section needs at least one nice red paragraph and some further
> >> explanations.
> >>
> >> Christian
> >>
> >> On Thu, 15 Oct 2015 17:33:30 -0600 Robert LeBlanc wrote:
> >>
> >>> -----BEGIN PGP SIGNED MESSAGE-----
> >>> Hash: SHA256
> >>>
> >>> One more question. Is max_{bytes,objects} before or after replication
> >>> factor?
> >>> - ----------------
> >>> Robert LeBlanc
> >>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> >>>
> >>>
> >>> On Thu, Oct 15, 2015 at 4:42 PM, LOPEZ Jean-Charles wrote:
> >>> > Hi Robert,
> >>> >
> >>> > yes they do.
> >>> >
> >>> > Pools don?t have a size when you create them hence the couple
> >>> > value/ratio that is to be defined for cache tiering mechanism. Pool
> >>> > only have a number of PGs assigned. So setting the max values and the
> >>> > ratios for dirty and full must be set explicitly to match your
> >>> > configuration.
> >>> >
> >>> > Note that you can at the same time define max_bytes and max_objects.
> >>> > The first of the 2 values that breaches using your ratio settings will
> >>> > trigger eviction and/or flushing. The ratios you choose apply to both
> >>> > values.
> >>> >
> >>> > Cheers
> >>> > JC
> >>> >
> >>> >> On 15 Oct 2015, at 15:02, Robert LeBlanc wrote:
> >>> >>
> >>> >> -----BEGIN PGP SIGNED MESSAGE-----
> >>> >> Hash: SHA256
> >>> >>
> >>> >> hmmm...
> >>> >>
> >>> >> http://docs.ceph.com/docs/master/rados/operations/cache-tiering/#relative-sizing
> >>> >>
> >>> >> makes it sound like it should be based on the size of the pool and
> >>> >> that you don't have to set anything like max bytes/objects. Can you
> >>> >> confirm that cache_target_{dirty,dirty_high,full}_ratio works as a
> >>> >> ratio of target_max_bytes set?
> >>> >> - ----------------
> >>> >> Robert LeBlanc
> >>> >> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> >>> >>
> >>> >>
> >>> >> On Thu, Oct 15, 2015 at 3:32 PM, Nick Fisk wrote:
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>>> -----Original Message-----
> >>> >>>> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On
> >>> >>>> Behalf Of Robert LeBlanc
> >>> >>>> Sent: 15 October 2015 22:06
> >>> >>>> To: ceph-users@xxxxxxxxxxxxxx
> >>> >>>> Subject: Cache Tiering Question
> >>> >>>>
> >>> >>>> -----BEGIN PGP SIGNED MESSAGE-----
> >>> >>>> Hash: SHA256
> >>> >>>>
> >>> >>>> ceph df (ceph version 0.94.3-252-g629b631
> >>> >>>> (629b631488f044150422371ac77dfc005f3de1bc)) is showing some odd
> >>> >>>> results:
> >>> >>>>
> >>> >>>> root@nodez:~# ceph df
> >>> >>>> GLOBAL:
> >>> >>>> SIZE AVAIL RAW USED %RAW USED
> >>> >>>> 24518G 21670G 1602G 6.53
> >>> >>>> POOLS:
> >>> >>>> NAME ID USED %USED MAX AVAIL OBJECTS
> >>> >>>> rbd 0 2723G 11.11 6380G 1115793
> >>> >>>> ssd-pool 2 0 0 732G 1
> >>> >>>>
> >>> >>>> The rbd pool is showing 11.11% used, but if you calculate the
> >>> >>>> numbers
> >>> >>> there
> >>> >>>> it is 2723/6380=42.68%.
> >>> >>>
> >>> >>> I have a feeling that the percentage is based on the amount used of
> >>> >>> the total cluster size. Ie 2723/24518
> >>> >>>
> >>> >>>>
> >>> >>>> Will this cause problems with the relative cache tier settings? Do
> >>> >>>> I need
> >>> >>> to set
> >>> >>>> the percentage based on what Ceph is reporting here?
> >>> >>>
> >>> >>> The flushing/eviction thresholds are based on the target_max_bytes
> >>> >>> number that you set, they have nothing to do with the underlying
> >>> >>> pool size. It's up to you to come up with a sane number for this
> >>> >>> variable.
> >>> >>>
> >>> >>>>
> >>> >>>> Thanks,
> >>> >>>> - ----------------
> >>> >>>> Robert LeBlanc
> >>> >>>> PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
> >>> >>>> ----- BEGIN PGP SIGNATURE-----
> >>> >>>> Version: Mailvelope v1.2.0
> >>> >>>> Comment: https://www.mailvelope.com
> >>> >>>>
> >>> >>>> wsFcBAEBCAAQBQJWIBVGCRDmVDuy+mK58QAAXEYQAKm5IBGn81Hlb9az4
> >>> >>>> 52x
> >>> >>>> hSH6onk7mJE7L2s5FnoJv2sNW4azhDEVKGQBE9vvhIVBhhtKtnqdzu3ytk6E
> >>> >>>> EUFuPBzUWLJyG3wQtp3QC0PdYzlGkS7bowdpZqk9PdaYZYgEdqG/cLEl/eAx
> >>> >>>> LGIUXmr6vIuNhnntGIIYeUAiWXA7b5qzOKbef6OlOp7Mz6Euel9S8ycZlSAR
> >>> >>>> eBQ5hdLSFoFai5ldyV+/hmqLnujOfanRFC8pIYr41aKe7wBOPOargLGQdka3
> >>> >>>> jswmcf+0hV7QqZSOjJijDYvOgRuHBFK6cdyP9SRKxWxG7uH+yDOvya0TqOob
> >>> >>>> 1yDomYC1zD2uzG9+L5Iv6at8fuBF5xFKPqax9N4WQj3Oj9fBwioQVBocNxHc
> >>> >>>> MIlQnvnLeq6OLtdfPoPignTAHIH2RrvAmdwYkSCuopjUSTkmBsyBLIiiz/KI
> >>> >>>> P4mSXAxZb0UF4pbCDgdYG6qUEywR/enGsT1lnmNLx4vY8W/yz9xQ3o3JnIpD
> >>> >>>> pWyo9zJ8Ugnwvihbo7xKe+EZOeJL0YF4BiyAprH5pKFdQcAWcV98zWHnLBxd
> >>> >>>> EFHyN9fHsVdw0UsxIUBZFfM1u4S7fchgVeFfiTSdGqd/dWHQCHKJPNBSJnae
> >>> >>>> aPKTyvg77N6zTn04VGspfenR+svGbkAtUfO2HJ1Kkd4/wZ9GIzsS1ovPZFsM
> >>> >>>> jJe4
> >>> >>>> =YSyj
> >>> >>>> -----END PGP SIGNATURE-----
> >>> >>>> _______________________________________________
> >>> >>>> ceph-users mailing list
> >>> >>>> ceph-users@xxxxxxxxxxxxxx
> >>> >>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>
> >>> >> -----BEGIN PGP SIGNATURE-----
> >>> >> Version: Mailvelope v1.2.0
> >>> >> Comment: https://www.mailvelope.com
> >>> >>
> >>> >> wsFcBAEBCAAQBQJWICJwCRDmVDuy+mK58QAAyTUQALkwOnB++bXto+cM0iSZ
> >>> >> B3nZgvl9FKZnujb0MUIiS29a+Y2nnBpAGgHbF4Y9ngnDQYNZ0yf1DD2wYad2
> >>> >> rll6pYeWRRYSmaBCBfdPlqbbVw8WpjdXLR9FtLFfUR2V+Ghf4U83F8iKiWn1
> >>> >> +6DqouHMA/auHjEr49w+Ue0kpKSfItH/9LkVjYQBKp6E7tyOSsrzcM1milKR
> >>> >> lwsIOewiKvsg4neDLqkdqaO6+bYuaDJmgN+hEqzl7lxbzt5pJbzfknpiAewm
> >>> >> GTw8C2AUbzcYqIhzqWcY9Jiy6ZZkYAPDODsJpkc/Pubnq73jlkllB4JaQpJy
> >>> >> 2964DynNn8jBAI9JJpLyldtKPEofmkumzZ6tPXgLDuo2VuV+hp/wVadZKy2k
> >>> >> PDhms1dpeLFM8NsgOToSpO6Ej1l1857C5+cy3EeTlKqgs6z1QbTwNvUeeCpk
> >>> >> /ORObJQCa7teNEM1c33oEJ3V1LOx7SfsEn1A6PVaaUegmMEEa6Cb8Va2RYl8
> >>> >> 5fhXqIcsU9KWHDmq8+MZ9x67etAucXKJmPQpIzJD6M9WtsWsDupsuJ1MgCKB
> >>> >> pxhqjwujuaZWfF+W3HEuOOP7OcXbj2U3RO1V3HOr9N0cLFTf+vuefIzOtgs1
> >>> >> qdBPrxIUNznfYXarclFuJzCWPzKpDTdKbLwYUcbh9hKayRpll3DGOW7qUX3u
> >>> >> eNXR
> >>> >> =cI+5
> >>> >> -----END PGP SIGNATURE-----
> >>> >> _______________________________________________
> >>> >> ceph-users mailing list
> >>> >> ceph-users@xxxxxxxxxxxxxx
> >>> >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>> >
> >>>
> >>> -----BEGIN PGP SIGNATURE-----
> >>> Version: Mailvelope v1.2.0
> >>> Comment: https://www.mailvelope.com
> >>>
> >>> wsFcBAEBCAAQBQJWIDfDCRDmVDuy+mK58QAA8qkQAIBtEorNvkAwVojMmOcW
> >>> /zEGPw9Hg0OgvoR7gv4DWSKO4y8raek3oL7BNE5WNrpkRkpKfjGe6OLLtTr+
> >>> 9b7K19Cv3oRQHVUG2S+rnwDzsg/4ORL90TZZSh729ThjE823g9PDpB1ThsdD
> >>> DApHvU4OoLEYVepCkxzZx4a8UztyaBnDl8/LCNK7Rzg30UWsiR9kRW4bru5F
> >>> igcFHslBmUSH0trbG0kxA9mrmnWq2m7i0QNVS1nUDJ7crDwqnJrnf17NG7NV
> >>> SQKKsAcuM2lmmAPkLIMy4J1oiBb8JXiCc27Bj+dtBG9Iqh8HdYvvmVd6O8Jv
> >>> bVgMUN7mmGGpuIs040Q3Fn4wSrhtGc5iUpzM5eJnemnrPi5ymE8WayHX6aak
> >>> qA5vfM8WLNKMmPBORqg2DB/1co6OkvHOLAk+ZAUYUo88I+dVp7BIXadaZMhS
> >>> GKbTPfpZgDdn0bHbn4Dyma1a1JVarpQXCaLq4ayvfY7DQuoFVi2eOImxvc+Q
> >>> gFSmmdegK0uto3aTnySR1fRl1Yk9grd+LSwJgmsew4t2AHjAbAYgG1idnvJt
> >>> t5e6Aj4NnNK3f085gkoundV1rrp37lu3Ot82gMq7xyxNmlT/FsAmOFSEelJP
> >>> U26AQHlgDM7oV95IQMnKOtdziIq7NFdspuVuN+umf7JpnuYLbROSREG3dIrq
> >>> qdxB
> >>> =de2k
> >>> -----END PGP SIGNATURE-----
> >>> _______________________________________________
> >>> ceph-users mailing list
> >>> ceph-users@xxxxxxxxxxxxxx
> >>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >>
> >>
> >> --
> >> Christian Balzer Network/Systems Engineer
> >> chibi@xxxxxxx Global OnLine Japan/Fusion Communications
> >> http://www.gol.com/
>
> -----BEGIN PGP SIGNATURE-----
> Version: Mailvelope v1.2.0
> Comment: https://www.mailvelope.com
>
> wsFcBAEBCAAQBQJWISFQCRDmVDuy+mK58QAAXXMP/1vK1wgXKf+eVH2+Adkp
> UlzZEdQ96XxZ+tLbo2+jjBZwbnIxZ8BpjTyLGMLVRwthR7iBEt9klMQRSAy5
> tT82Uk5gRqAJVoMSLTXQhnNJRPYxbTBEP1BBkw10WJu/5l/AvUpNNkejiQ1i
> E/KXNqwKEFf9FUAlWGAwW2naGtjU3Z7HK95K8C9FwP7BcIJA+b/3FhUIXntE
> MLTAeaj6yIftsBD9Hav/RWQxpgA7Db0IHF2EyV4Ry+ds2V25PBg60MeEJpa0
> rJxPgfKB+yawWzzSVKpywQLulxbqafxAnugDrXZfQBfAjFPScrdBWeAbW6Tp
> qzwG8/5/TWj394DzkJ5ary/YjiUJwsliZ6yiBKXIY+OIJcUWb4+Aa8ktswhs
> 3PcyE4HJk/QPSynjF9CyX62lhpXO4lSBcq5AoL5VP48QRYx/ZPTeAjkuBEAx
> CjjL8WcBDUww56/beelecDci7TW+tetdAJ6t3nWgWNkBoDSqjXJV6CQqtOJ6
> rhhnZ9hn8tLQ84anpPp9o8P+XyueDQOdN82IbLUt5qIzRVcwGvCnyZy2RzfB
> s2NS9b2RqRdM6HMFAR57xAoPnmlK9kC9I4LfN5ApCa3RryntLxbT5rI3nHhU
> nvIrjKI7p28QLB5EtnWC7oJsuXAvF4wVJ8QOvjv8VynldWW0FwSUs4yeaUxz
> ib4U
> =AEVh
> -----END PGP SIGNATURE-----
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
