Re: Emergency, I lost 4 monitors but all osd disk are safe

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hey Mohamed,

just send us the output of

ceph -s

and

ceph mon dump

please.

Best,
Malte

On 02.11.23 13:05, Mohamed LAMDAOUAR wrote:
Hi robert,

when I ran this command, I got this error (because the database of the osd
was on the boot disk)

ceph-objectstore-tool \
--type bluestore \
--data-path /var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9 \
--op update-mon-db \
--mon-store-path /home/enyx-admin/backup-osd-9 \
--no-mon-config --debug
2023-11-02T10:59:33.381+0000 7f6724da71c0  1 bdev(0x560257b42400
/var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9/block) open path
/var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9/block

2023-11-02T10:59:33.381+0000 7f6724da71c0  1 bdev(0x560257b42400
/var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9/block) open size
10000827154432 (0x9187fc00000, 9.1 TiB) block_size 4096 (4 KiB) rotational
device, discard not supported

2023-11-02T10:59:33.381+0000 7f6724da71c0  1 bluestore(/var/lib/ceph/
c80891ba-55f3-11ed-9389-919f4368965c/osd.9) _set_cache_sizes cache_size
1073741824 meta 0.45 kv 0.45 data 0.06

2023-11-02T10:59:33.381+0000 7f6724da71c0  1 bdev(0x560257b42c00
/var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9/block) open path
/var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9/block

2023-11-02T10:59:33.381+0000 7f6724da71c0  1 bdev(0x560257b42c00
/var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9/block) open size
10000827154432 (0x9187fc00000, 9.1 TiB) block_size 4096 (4 KiB) rotational
device, discard not supported

2023-11-02T10:59:33.381+0000 7f6724da71c0  1 bluefs add_block_device bdev 1
path /var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9/block size
9.1 TiB

2023-11-02T10:59:33.381+0000 7f6724da71c0  1 bluefs mount

2023-11-02T10:59:33.381+0000 7f6724da71c0  1 bluefs _init_alloc shared, id
1, capacity 0x9187fc00000, block size 0x10000

2023-11-02T10:59:33.441+0000 7f6724da71c0 -1 bluefs _replay 0x0: stop: uuid
369c96dd-2df1-8d88-2722-3f8334920e83 != super.uuid
ba94c6e8-394b-4a78-84d6-9afe1cbc280b,
block dump:
00000000  42 9c 61 78 69 ec 36 9c  96 dd 2d f1 8d 88 27 22
  |B.axi.6...-...'"|
00000010  3f 83 34 92 0e 83 e6 0f  8f 17 fc 3e ec 86 c5 15
  |?.4........>....|
00000020  39 91 13 2e b0 14 92 86  65 75 5c 8e c1 ee fc 18
  |9.......eu\.....|
00000030  f1 7b b2 37 f7 75 70 e2  5e da 79 cd e6 ad 27 40
  |.{.7.up.^.y...'@|
00000040  d6 b8 3b da 81 1f 9b ba  c6 e8 b7 68 bc a1 77 ac
  |..;........h..w.|
00000050  7b a9 a3 cd 9d da b6 57  aa 40 bd ab d0 89 ec e6  |{......W.@
......|
00000060  71 a2 2b 4d 87 74 2f ff  0a bf 3b da 3d da 93 52
  |q.+M.t/...;.=..R|
00000070  1c ea f2 fb 8d e0 a1 e6  ef b5 42 5e 85 87 27 df
  |..........B^..'.|
00000080  ac f1 ae 08 9d c5 71 6f  0f f7 68 ce 28 3d 3e 6e
  |......qo..h.(=>n|
00000090  94 b2 1a dc 3b f0 9e e9  6e 77 dd 95 b6 9e 94 56
  |....;...nw.....V|
000000a0  f2 dd 9a 35 a0 65 78 05  bb a9 5f a1 99 6a 5c a1
  |...5.ex..._..j\.|
000000b0  5d e9 6d 02 83 be 9d 60  d1 82 fc 6c 66 40 11 17  |].m....`...lf@
..|
000000c0  3a 4d 9d 73 f6 ec fb ed  41 db e2 39 15 e1 5f 28
  |:M.s....A..9.._(|
000000d0  c4 ce cf eb 93 f2 88 d5  af ae 11 14 d6 97 74 ff
  |..............t.|
000000e0  4b 7e 73 fe 97 4c 06 2a  3a bc b3 7f 04 94 6c 1d
  |K~s..L.*:.....l.|
000000f0  60 bf b1 42 fa 76 b0 df  33 ff bf 84 36 b1 b5 b3
  |`..B.v..3...6...|
00000100  17 36 d6 b7 7d 4c d4 37  fa 7f 8e 59 1f 72 53 d5
  |.6..}L.7...Y.rS.|
00000110  c4 d0 de d8 4e 13 ca c6  0a 60 87 3c e4 21 2b 1b
  |....N....`.<.!+.|
00000120  00 f2 67 cf 0a 02 01 20  ec ec 7f c1 8f e3 df f8  |..g....
........|
00000130  3f db 7f 60 28 14 8a fa  48 cb c6 f6 c7 9a 3f 71
  |?..`(...H.....?q|
00000140  bf 61 36 30 08 c0 f1 e7  f8 af b5 7f d2 fc ad a1
  |.a60............|
00000150  72 b2 40 ff 82 ff a3 c7  5f f0 a3 0e 8f b2 fe b6  |r.@
....._.......|
00000160  ee 2f 5d fe 90 8b fa 28  8f 95 03 fa 5b ee e3 9c
  |./]....(....[...|
00000170  36 ea 3f 6a 1e c0 fe bb  c2 80 4a 56 ca 96 26 8f
  |6.?j......JV..&.|
00000180  85 03 e0 f8 67 c9 3d a8  fa 97 af c5 c0 00 ce 7f
  |....g.=.........|
00000190  cd 83 ff 36 ff c0 1c f0  7b c1 03 cf b7 b6 56 06
  |...6....{.....V.|
000001a0  8a 30 7b 4d e0 5b 11 31  a0 12 cc d9 5e fb 7f 2d
  |.0{M.[.1....^..-|
000001b0  fb 47 04 df ea 1b 3d 3e  6c 1f f7 07 96 df 97 cf
  |.G....=>l.......|
000001c0  15 60 76 56 0e b6 06 30  3b c0 6f 11 0a 40 19 98  |.`vV...0;.o..@
..|
000001d0  a1 89 fe e3 b6 f3 28 80  83 e5 c1 1d 9c ac da 40
  |......(........@|
000001e0  71 5b 2b 07 eb 07 2e 8a  0f 71 7f 88 aa f5 23 0b
  |q[+......q....#.|
000001f0  03 17 a0 b0 e2 c3 76 e3  68 62 00 53 10 17 02 4a
  |......v.hb.S...J|
00000200  02 ec 1f 72 82 8f 0f 28  fc a0 e0 83 04 3b c0 e3
  |...r...(.....;..|
00000210  6b 25 15 fe ae ce df de  33 29 6c e5 f0 68 c6 1f
  |k%......3)l..h..|
00000220  e9 01 00 ff f1 be c9 c7  f4 f8 73 fc bf cc 80 fe
  |..........s.....|
00000230  73 1d 08 a8 64 62 6f 0e  e3 11 13 15 13 03 81 58
  |s...dbo........X|
00000240  a1 20 10 9b f0 43 e3 7c  68 2c 0f ed 21 34 10 10  |.
...C.|h,..!4..|
00000250  08 04 05 f3 fd de 0f ed  35 ff b0 4d 4d 5d e3 a1
  |........5..MM]..|
00000260  7f 30 08 f0 b0 85 fe e9  06 f0 3f 95 64 f9 2f 3e
  |.0........?.d./>|
00000270  7b 48 e3 7f 3a 0a f5 7f  3c 05 f8 9b ab 1e a3 2f
  |{H..:...<....../|
00000280  6d 62 68 a7 09 f8 4d f5  08 fe 69 e9 df ed f9 a7
  |mbh...M...i.....|
00000290  0c 61 fd 87 fd dc ca e8  0f 59 7f 44 06 15 28 67
  |.a.......Y.D..(g|
000002a0  0d b3 14 34 78 2c a4 3f  d4 07 fc d6 df d2 c1 dc
  |...4x,.?........|
000002b0  fc 8f 07 e8 81 b7 9c 83  bd b9 89 e5 a3 52 90 7f
  |.............R..|
000002c0  39 ed 6f ff 37 20 fc 50  d7 0f 89 f8 a7 bb 58 1e  |9.o.7
.P......X.|
000002d0  5c c4 fe e0 9f 87 02 01  b1 b3 f3 fd 4e 22 43 07
  |\...........N"C.|
000002e0  03 d8 5f f3 ff 71 25 e4  4f ba 07 77 b2 3c e0 d9
  |.._..q%.O..w.<..|
000002f0  d8 fe 6c 8f 73 d0 bf b9  f9 b1 17 7d e8 41 0f bc
  |..l.s......}.A..|
00000300  7e 8b 7a d0 59 e4 e1 98  a0 17 e1 82 80 c0 ec 8f
  |~.z.Y...........|
00000310  8c c0 ec 0f c2 a0 0c 20  08 1d 08 44 f7 ea 6f 6a  |.......
...D..oj|
00000320  3a db c2 de a3 82 00 0f  07 28 e8 af 0f 80 8d 95
  |:........(......|
00000330  15 ca 0a 78 0f f8 03 07  e1 04 b3 3d 64 c0 ef 19
  |...x.......=d...|
00000340  4b c0 5f 74 60 ce 7f e0  c0 50 30 2b cb 3f 70 10
  |K._t`....P0+.?p.|
00000350  ce 87 c3 f9 df 74 ec 6c  ff a4 63 e7 7c 28 dd 7f
  |.....t.l..c.|(..|
00000360  e1 38 ff 29 03 c2 01 e6  fc a7 2e 10 0e 76 f0 bf
  |.8.).........v..|
00000370  70 9c a0 7f cb 78 38 82  39 fe 85 83 82 fe 4d 07
  |p....x8.9.....M.|
00000380  65 83 fc 0b c7 ca 0e fa  17 8e ed ef b6 d9 db ea
  |e...............|
00000390  3f 9c e7 b6 8f f1 56 34  71 85 3d b8 14 a8 60 65
  |?.....V4q.=...`e|
000003a0  f5 90 79 7f e6 93 a4 e5  7b 2b 00 98 f5 4f 40 04  |..y.....{+...O@
.|
000003b0  a0 09 e0 01 83 a0 1c ec  42 42 ac 60 56 56 41 36
  |........BB.`VVA6|
000003c0  41 11 11 56 61 28 04 c2  21 26 c6 01 15 61 85 40
  |A..Va(..!&...a.@|
000003d0  85 20 7c a8 ff 37 c9 63  8a 8b 58 19 08 1b c3 0c  |.
|..7.c..X.....|
000003e0  cc ec 1c 2c 00 40 31 0e  30 94 45 94 93 0d c4 c9
  |...,.@1.0.E.....|
000003f0  c9 ce 21 04 82 b0 41 39  45 20 42 ac 10 21 31 0e  |..!...A9E
B..!1.|
00000400  08 54 84 83 13 f5 f7 f7  06 7d 5b fb df a1 87 70
  |.T.......}[....p|
00000410  72 40 39 50 69 69 45 e5  c4 50 ff 1f ce 58 4b a0  |r@9PiiE.
.P...XK.|
00000420  b8 3b 78 9c 6d 53 41 8e  db 30 0c bc eb 15 04 7a
  |.;x.mSA..0.....z|
00000430  69 17 8d 1f 90 9c da 5d  b4 dd 5b 80 4d 1f 40 4b
  |i......]..[.M.@K|
00000440  b4 c3 b5 2c 1a 92 6c 63  7f 5f 4a 89 03 24 5b 9f
  |...,..lc._J..$[.|
00000450  6c 93 33 1c cd 50 4d 03  11 d7 fd 1e 26 d7 19 03
  |l.3..PM.....&...|
00000460  00 47 ec e9 67 24 1c 40  02 3d 8b 9f c7 60 cc 97
  |.G..g$.@.=...`..|
00000470  bb c7 bc 86 1c c5 cd 36  b3 84 c7 9a 39 9d 39 81
  |.......6....9.9.|
00000480  13 3b 8f 14 32 38 4a 36  72 4b 09 f2 99 e0 d7 f1
  |.;..28J6rK......|
00000490  f7 0f e8 38 8e 2b 46 02  0c 0e 92 74 b9 7e f0 38
  |...8.+F....t.~.8|
000004a0  79 2a 10 72 80 09 26 8c  19 a4 ab a8 a7 13 db 61
  |y*.r..&........a|
000004b0  97 65 77 8a e8 08 a6 28  ef 64 f3 53 63 8c 79 d9
  |.ew....(.d.Sc.y.|
000004c0  e6 bc 59 99 e8 93 96 b7  89 2c a3 07 cc 59 9b 54
  |..Y......,...Y.T|
000004d0  2d ac ec 3d b4 04 3d 2f  14 20 4b 1d 70 53 f4 7a  |-..=..=/.
K.pS.z|
000004e0  04 2b 51 c5 7e c5 01 e1  85 12 f7 01 fe 06 ce e9
  |.+Q.~...........|
000004f0  1b 38 8e 3a d5 7f 00 87  45 fc a2 32 39 5c d0 98
  |.8.:....E..29\..|
00000500  32 74 44 ae 45 3b 18 2f  32 7d 57 96 10 b4 9b 43
  |2tD.E;./2}W....C|
00000510  5f 5b 46 8c 03 a9 19 98  11 02 e5 55 e2 a0 e8 4c
  |_[F........U...L|
00000520  b1 43 4b 45 85 4e 00 89  8e 22 a8 ce f8 a1 f8 b9
  |.CKE.N..."......|
00000530  94 8b 0b cd a3 a5 19 63  4f da 8f 01 70 76 4c 41
  |.......cO...pvLA|
00000540  29 56 ce 67 68 31 b1 85  39 28 4d ca 6a 6e 99 2e
  |)V.gh1..9(M.jn..|
00000550  57 8d e2 bd ac 55 8e 4c  6c d3 7e 93 a1 bf 0e 90
  |W....U.Ll.~.....|
00000560  b2 06 3e 96 6a 51 b8 eb  b4 f5 60 4a 3a 23 8d a2
  |..>.jQ....`J:#..|
00000570  6a 46 9c a6 7a de 8b 62  6d bc f7 9e ea 26 a4 07
  |jF..z..bm....&..|
00000580  fb ff b3 0d e5 9d 17 76  9b 79 91 08 d2 15 bd 37
  |.......v.y.....7|
00000590  66 57 ce 24 0b c5 85 69  dd 0a 25 ef 02 49 d0 53
  |fW.$...i..%..I.S|
000005a0  a0 a8 69 72 e8 24 8e 58  8b d7 f3 f5 82 3e 5d f6
  |..ir.$.X.....>].|
000005b0  45 67 5c 37 e4 50 08 6f  34 8e 32 6a f8 5b 24 f7
  |Eg\7.P.o4.2j.[$.|
000005c0  ab 78 db be ca 5a 81 7a  fc 8e 03 6a e0 7a 11 6e
  |.x...Z.z...j.z.n|
000005d0  2c b6 c8 db 38 b6 e5 bd  73 e3 0f a7 ac a6 7d 32
  |,...8...s.....}2|
000005e0  c3 98 e7 33 86 be dc 04  b9 c8 bc d9 52 14 78 85
  |...3........R.x.|
000005f0  5d 6c 69 49 fd 57 ca a6  d1 2f eb 67 47 7a 3d 6d
  |]liI.W.../.gGz=m|
00000600  c5 7a e9 1b cd 56 c7 fd  03 f1 ee 40 a1 e5 06 82  |.z...V.....@
....|
00000610  6d 78 9c db c1 de cf 3e  a1 66 63 e7 4f 46 c1 dc
  |mx.....>.fc.OF..|
00000620  c4 cc 3c 85 e2 d4 e4 92  cc fc bc 62 3d 2e 2e c7
  |..<........b=...|
00000630  c9 13 98 9c 6c 52 15 92  f3 8b 8a 52 8b 0b f2 f3
  |....lR.....R....|
00000640  52 32 f3 d2 15 0a 8a f2  b3 80 6a f4 14 1c f3 f2
  |R2........j.....|
00000650  4b 32 52 8b 60 3a 14 32  8b b9 52 52 53 32 93 13
  |K2R.`:.2..RRS2..|
00000660  4b 52 53 14 4a f2 15 80  92 93 ff 32 49 4c 76 65
  |KRS.J......2ILve|
00000670  2e 06 00 81 82 25 40 b8  c5 05 78 9c b5 7a 65 54
  |.....%@...x..zeT|
00000680  5c cb d6 6d e3 d2 48 70  97 20 01 82 5b 20 b8 bb  |\..m..Hp. ..[
..|
00000690  43 70 f7 c6 a1 1b 97 20  01 02 04 08 0e 21 c1 5d  |Cp.....
.....!.]|
000006a0  1a 02 04 0d ee ee ee 6e  c1 dd 5f 72 df 77 c6 77
  |.......n.._r.w.w|
000006b0  72 de 39 77 dc fb e3 55  8f ea b1 ab 47 cd 59 ab
  |r.9w...U....G.Y.|
000006c0  6b ed 3d 6b ed aa a5 22  0f 07 8f 0b 00 20 03 00  |k.=k...".....
..|
000006d0  bd 22 3d 72 e9 70 2c 1d  8c 00 00 e0 57 fd f9 13
  |."=r.p,.....W...|
000006e0  c0 1e 64 6f 01 f1 74 b2  30 71 72 b2 03 99 99 40
  |..do..t.0qr....@|
000006f0  40 8e 0e ac 6e 0e e6 2c  8e 26 60 10 98 c5 d1 c9
  |@...n..,.&`.....|
00000700  c2 c1 dc d1 cc d5 de c2  01 c2 62 e5 62 e2 64 0d
  |..........b.b.d.|
00000710  32 03 ab fc 46 68 ac 3a  d7 61 00 04 00 7e 55 c2
  |2...Fh.:.a...~U.|
00000720  9f 84 1a d6 ae f6 a6 0e  26 20 3b 30 2b e4 8f 4b  |........&
;0+..K|
00000730  16 27 07 ab 30 15 25 69  74 54 62 d4 9f 5d d0 65
  |.'..0.%itTb..].e|
00000740  65 24 d4 00 00 98 9f 97  80 6f c8 70 3f bf 43 17
  |e$.......o.p?.C.|
00000750  46 5f 00 00 b0 f0 2a 0a  1a 92 3f 9b 40 20 10 0f  |F_....*...?.@
..|
00000760  0f 8f 8c 8c 8c 94 9c 92  86 86 96 85 85 85 96 ed
  |................|
00000770  25 b5 00 35 37 37 37 07  f7 6b 21 21 a1 17 dc 92
  |%..5777..k!!....|
00000780  f4 e2 f4 cc 0a cc 9c 6f  38 5f e9 bd e2 35 e4 e5
  |.......o8_...5..|
00000790  37 e3 67 15 d3 e4 90 d2  e3 94 31 e4 96 37 e3 55
  |7.g.......1..7.U|
000007a0  b2 7c ad 6c cd a7 6a 23  a4 e5 ac c5 a5 a5 cb ad
  |.|.l..j#........|
000007b0  6b c4 6b 64 ca 67 aa a4  a4 a4 a1 a1 a1 a7 a7 27
  |k.kd.g.........'|
000007c0  a2 eb 2a 6e e8 2d 6a 27  2a 09 91 94 f3 95 93 34
  |..*n.-j'*......4|
000007d0  f1 73 10 75 80 48 42 7c  e5 7d cd cc cc ec ed ed
  |.s.u.HB|.}......|
000007e0  2d ed 20 0e ae be 76 3e  3e 60 30 d8 db db 5b ca  |-.
...v>>`0...[.|
000007f0  34 40 c6 e2 9d 9c 75 a8  82 4d 84 92 c3 47 15 c7  |4@
....u..M...G..|
00000800  58 35 70 a2 86 5b 8a a6  c7 17 6d ef 4c a5 20 25  |X5p..[....m.L.
%|
00000810  95 60 15 b5 70 35 cd 8f  9a 3a 09 3a fa a9 fa 46
  |.`..p5...:.:...F|
00000820  19 46 26 d9 26 16 45 16  d6 50 6b db 4a 5b 87 5a
  |.F&.&.E..Pk.J[.Z|
00000830  07 e7 46 67 48 3b c4 a3  db c3 b3 df d3 77 d4 d7
  |..FgH;.......w..|
00000840  20 a8 d8 30 e4 ab 71 08  d4 38 b4 dc 2c bc ca 22  |
..0..q..8..,.."|
00000850  aa de ea e3 77 eb 98 46  8f 94 66 e3 c8 16 ab 84
  |....w..F..f.....|
00000860  6e eb f8 4e fb e4 2e 87  4f bd ce 69 43 41 4a 41
  |n..N....O..iCAJA|
00000870  a1 2a a1 1f 35 3f 26 e8  24 7c 36 fc 9c 65 92 55
  |.*..5?&.$|6..e.U|
00000880  6c 59 0c b5 86 56 da 56  d6 39 d6 35 3a 37 b6 41
  |lY...V.V.9.5:7.A|
00000890  da fa 3c fb c6 7c c7 02  02 02 22 22 22 e2 e3 e3
  |..<..|...."""...|
000008a0  53 53 53 df 7d ca c8 ce  ce 2e 2e 2e ae ac ac 6c
  |SSS.}..........l|
000008b0  68 68 08 ce 1f 0a 2f 1e  cd 89 69 81 e6 8f e5 54
  |hh..../...i....T|
000008c0  f6 e6 7f 6f 2b 6c ec cd  6d 9c 6b 2c 9a ad a8 1f
  |...o+l..m.k,....|
000008d0  ae ae 1a ff 56 3f ea 3f  e5 1f 34 1f 14 b8 98 1a
  |....V?.?..4.....|
000008e0  3e 9e 17 b2 1a 12 b6 19  16 f9 23 32 fa 38 3a e6
  |>.........#2.8:.|
000008f0  2c 26 f1 2e 31 f9 31 b9  a4 b5 e7 6b 5b 7f 7d c7
  |,&..1.1....k[.}.|
00000900  58 6d f7 5c 63 ef 6c c3  c0 5a 5b fa d2 a4 ff e4
  |Xm.\c.l..Z[.....|
00000910  c2 bb 85 fe a8 ed d5 90  d5 8d f7 1b fb 91 fb 67
  |...............g|
00000920  31 67 b7 09 b7 8f c9 8f  ed ed ed fd fd fd 2d 3d
  |1g............-=|
00000930  e3 ed c3 cb e3 e3 e3 d3  b3 f3 0b 0b 0b 6d c3 2b
  |.............m.+|
00000940  9d a3 ab 7d 53 5b f3 eb  87 1b 1b 1b 5b 3f ce 8e
  |...}S[......[?..|
00000950  ce ae ce 2e 6f 1e 1f 1f  7f 7a ff e9 e9 49 da bd
  |....o....z...I..|
00000960  0d 1b 00 40 c1 95 95 10  d5 f0 98 3b f8 f2 a0 35
  |...@.......;...5|
00000970  3f 45 d4 dd 08 91 4d fa  a6 23 5d 1d d2 be 0b 12
  |?E....M..#].....|
00000980  2f c0 8a 54 e5 ac e3 d2  41 a6 7f a5 f9 2e d8 2e
  |/..T....A.......|
00000990  51 1c c3 40 9f ff f5 bb  10 22 53 d9 b8 dc 1f cc  |Q..@
....."S.....|
000009a0  b6 af bc bf d9 65 26 8f  74 f2 8b 30 f1 4b 81 b6
  |.....e&.t..0.K..|
000009b0  68 ee 93 e2 ee 61 09 8f  5b 1c 74 07 26 cf 26 1b
  |h....a..[.t.&.&.|
000009c0  77 12 28 0e 43 b7 85 ae  14 a3 9d ec 66 f4 31 00
  |w.(.C.......f.1.|
000009d0  7f 2d 2d 24 d3 fb af de  61 fb 77 cb 97 ee 97 95
  |.--$....a.w.....|
000009e0  61 c1 1e 0f 58 e7 a9 8d  f3 77 6a c9 c0 3e 43 50
  |a...X....wj..>CP|
000009f0  e2 91 37 78 49 a4 df 3b  87 15 82 c5 f9 52 fa c5
  |..7xI..;.....R..|
00000a00  87 40 95 39 58 35 89 61  d8 3e 4b a2 41 58 02 e4
  |.@.9X5.a.>K.AX..|
00000a10  51 df a8 8f c3 e8 a8 81  2b e4 5c 68 9f d7 07 24
  |Q.......+.\h...$|
00000a20  6a 5d f5 a9 9c 44 59 b4  d2 7f 75 c1 8b a8 01 d4
  |j]...DY...u.....|
00000a30  46 8a f2 b0 ae d6 8a 12  2e e8 b4 53 f2 12 6d 9b
  |F..........S..m.|
00000a40  63 4c 3b a7 e0 38 11 99  00 cb 3b 80 9f 49 71 6a
  |cL;..8....;..Iqj|
00000a50  93 29 53 01 12 42 21 a9  00 8f 2f 2b a6 0e 31 fc
  |.)S..B!.../+..1.|
00000a60  f9 0d af 95 c4 16 90 08  2f fa 46 e8 c7 d9 7f 7c
  |......../.F....||
00000a70  57 95 f8 86 47 7f 84 90  ad 31 e7 8a 2e 08 f4 30
  |W...G....1.....0|
00000a80  7f 79 57 00 b5 13 3c d0  bc 54 c5 97 05 5e 27 b9
  |.yW...<..T...^'.|
00000a90  19 fb 06 65 21 94 d4 a0  ae 7c 5b 84 c9 b0 85 81
  |...e!....|[.....|
00000aa0  64 d5 5f d1 d9 12 8c c5  a7 14 54 59 be e0 d7 f6
  |d._.......TY....|
00000ab0  97 33 44 57 a0 f1 89 4f  b2 f4 69 02 6c c9 da f6
  |.3DW...O..i.l...|
00000ac0  cd bf dc 84 dc 4e a8 33  e8 69 03 b8 b7 34 64 94
  |.....N.3.i...4d.|
00000ad0  f7 74 09 14 11 71 c7 e4  11 a4 e0 66 08 5f 87 c7
  |.t...q.....f._..|
00000ae0  34 3d c6 33 33 f9 4f e0  a6 27 99 ef b0 b0 1e 64
  |4=.33.O..'.....d|
00000af0  83 58 13 46 70 1d 09 82  22 39 0c cc f4 00 7d c8
  |.X.Fp..."9....}.|
00000b00  7a 22 5d 25 e7 45 45 b9  63 e1 9f 34 ab 3e d8 bc
  |z"]%.EE.c..4.>..|
00000b10  0e 80 d0 16 8e a0 4d c1  c4 ab 22 18 3f f6 ea 30
  |......M...".?..0|
00000b20  67 d0 e2 16 c3 64 70 47  be 29 39 7d 41 f4 c2 3e
  |g....dpG.)9}A..>|
00000b30  ce 92 4d 1c 71 79 fc b3  65 a3 f0 e0 87 ca e4 65
  |..M.qy..e......e|
00000b40  65 04 10 0e 5c 82 4a 2f  0a 5f eb 35 83 9b 80 7f
  |e...\.J/._.5....|
00000b50  2d 4e c7 f7 26 67 56 59  24 e9 38 05 e1 3c d3 69
  |-N..&gVY$.8..<.i|
00000b60  19 d6 42 6b dc e0 14 09  81 ad ba e9 a4 b9 12 58
  |..Bk...........X|
00000b70  d6 18 f8 95 7d 2d 25 5a  41 08 80 5a 7d c8 2f 38
  |....}-%ZA..Z}./8|
00000b80  c7 e7 31 a3 48 b5 17 25  5c 03 27 79 8c 42 e1 e3
  |..1.H..%\.'y.B..|
00000b90  7b 3d 91 17 19 ec 36 62  bd f0 b1 4c fc 58 32 79
  |{=....6b...L.X2y|
00000ba0  1c 4c c5 a0 c2 0f f8 2a  84 c1 16 ec b9 7b 68 4d
  |.L.....*.....{hM|
00000bb0  0c 65 f3 ce 4c 2d 23 bd  07 4a c7 b9 07 80 16 eb
  |.e..L-#..J......|
00000bc0  f2 af 14 fe 0b ef 7a 82  4a 53 32 7e 6c bd 1b 09
  |......z.JS2~l...|
00000bd0  5b 9e 50 aa 4c 75 57 10  94 f8 8a fa a3 74 fd 6b
  |[.P.LuW......t.k|
00000be0  1e a3 b8 8c 52 78 e2 01  46 06 46 66 1a 09 0c d7
  |....Rx..F.Ff....|
00000bf0  8a 68 11 aa 80 91 ae ae  ae a1 6e 05 7e d8 7e 5d
  |.h........n.~.~]|
00000c00  5e ba 61 c1 02 bb ba 73  f7 de b1 78 3c 53 46 5d
  |^.a....s...x<SF]|
00000c10  14 90 e4 4b 56 3b cc f1  2b 47 30 c0 ad c6 7f 46
  |...KV;..+G0....F|
00000c20  59 49 e6 e0 e9 b2 ff 68  6e 22 22 0a a6 85 9f 74
  |YI.....hn""....t|
00000c30  f3 f2 ed 5d f0 68 5e 22  1f a2 c8 ac b3 4b fb 14
  |...].h^".....K..|
00000c40  e0 95 52 57 13 81 ff 69  94 5f 27 cc d9 fa 10 4c
  |..RW...i._'....L|
00000c50  0b 51 75 b9 bf e3 d3 84  82 11 57 23 f8 e9 a2 15
  |.Qu.......W#....|
00000c60  e6 7e 53 18 69 5a 18 13  ae 65 99 ba 94 f1 7c 6f
  |.~S.iZ...e....|o|
00000c70  05 d0 34 44 a1 8f ef df  2d 0d d3 32 a8 85 74 05
  |..4D....-..2..t.|
00000c80  53 0a bb 03 00 d6 b2 e3  dd 66 f9 91 a6 41 f5 67
  |S........f...A.g|
00000c90  4f 24 6a 99 81 da bd 65  50 a6 81 e1 ed 33 5b 20
  |O$j....eP....3[ |
00000ca0  e3 d4 cd fd f8 a4 8f df  85 ac 15 cc 75 85 a3 b9
  |............u...|
00000cb0  75 ed f6 9a b9 b9 cb ca  98 88 c7 4b 01 6e 17 cb
  |u..........K.n..|
00000cc0  db 5b c8 95 79 81 2b 70  06 a1 e5 9b 8b b7 6b eb
  |.[..y.+p......k.|
00000cd0  26 07 e6 94 65 b6 ed 28  9d 3b 81 83 70 8a 5e 2d
  |&...e..(.;..p.^-|
00000ce0  f7 b4 ea 9b 95 7e 77 6f  2a c1 c5 17 fe cb 17 06
  |.....~wo*.......|
00000cf0  e4 e4 7e 57 7d 56 14 ee  46 44 e4 7e 6f 8b b6 d1
  |..~W}V..FD.~o...|
00000d00  01 bc 4f ad 7b 7d 64 d3  a5 3e 7e 87 51 75 75 96
  |..O.{}d..>~.Quu.|
00000d10  43 db 96 8e a5 83 c3 03  25 10 7f 84 85 b2 27 64
  |C.......%.....'d|
00000d20  d7 2b 45 37 3d c0 80 f0  63 0a 27 c5 93 97 4b 0b
  |.+E7=...c.'...K.|
00000d30  df 6e d4 79 95 21 82 db  0d 60 ba 88 91 c7 4c b7
  |.n.y.!...`....L.|
00000d40  65 ce c7 6f 0b 93 62 70  c9 4d 07 80 6f 79 c4 05
  |e..o..bp.M..oy..|
00000d50  4b 4f 5e 81 a0 ff e1 fc  02 1d 4d 59 f2 05 ab 21
  |KO^.......MY...!|
00000d60  e9 6e dd a8 e0 c3 7c 66  6f e6 30 12 15 dc 3c 03
  |.n....|fo.0...<.|
00000d70  d9 04 ec 5a 63 9b 1b 60  f4 7e 6f 41 8c cb 46 f3
  |...Zc..`.~oA..F.|
00000d80  d1 1a 72 70 d8 e7 03 b1  3f 5c d2 3f 5c 6a d3 cd
  |..rp....?\.?\j..|
00000d90  7f dd fa c0 7d 31 d4 74  7b c3 4f a7 9c e6 c4 bb
  |....}1.t{.O.....|
00000da0  44 7c 0f 06 96 b6 c4 a2  18 df d0 03 6a 57 a3 0e
  |D|..........jW..|
00000db0  f2 95 9d 49 85 f7 9d ce  c3 bd b9 82 ea 86 1c 10
  |...I............|
00000dc0  c5 7d 17 db c5 db 63 f6  d3 0d c8 91 23 df e5 41
  |.}....c.....#..A|
00000dd0  43 a5 e6 f6 91 08 43 06  12 dc 2e e1 f0 85 3e 2c
  |C.....C.......>,|
00000de0  f5 4b 6a 1d 6d a6 0d 6d  9f 9e d9 d6 d1 91 cd 60
  |.Kj.m..m.......`|
00000df0  71 c0 e8 93 13 87 9d bf  a3 8e 49 53 19 be 9b f6
  |q.........IS....|
00000e00  ce bb 9d 83 57 59 ca a6  db e9 7b 5b b5 57 9f 70
  |....WY....{[.W.p|
00000e10  67 6e b9 19 94 f2 79 df  c7 41 0b 6f 07 19 58 ee
  |gn....y..A.o..X.|
00000e20  e6 3b 1e 03 1a bf 73 97  f6 e3 76 69 54 04 78 bb
  |.;....s...viT.x.|
00000e30  06 3b a6 22 eb 43 f3 2d  f9 8e c2 59 6a dd b4 02
  |.;.".C.-...Yj...|
00000e40  d6 4c e0 6a 29 ea b9 54  56 8f 6f 2c e4 e1 62 fa
  |.L.j)..TV.o,..b.|
00000e50  93 a0 12 4b 32 ca e5 59  fc 37 b3 f7 3f 16 eb ce
  |...K2..Y.7..?...|
00000e60  ce c0 14 4b 0d 77 93 76  f9 d7 fe a5 f8 d0 ba c3
  |...K.w.v........|
00000e70  1a 76 7c 6b 0e c2 4f 5f  ee 5e f2 f1 54 ba 33 8e
  |.v|k..O_.^..T.3.|
00000e80  ac 06 65 12 07 07 e9 b6  8c 92 a2 1d 38 a2 f1 c4
  |..e.........8...|
00000e90  10 63 84 b6 ad a6 4e 32  ab cd b6 6d dd 0c a6 ae
  |.c....N2...m....|
00000ea0  aa 58 7b bc d4 50 e7 c5  85 0a 6a f8 ae 5d 31 03
  |.X{..P....j..]1.|
00000eb0  ae bb 9b 8b 6f eb 2b 74  40 98 93 0e b8 98 ed 28  |....o.+t@
......(|
00000ec0  8f 22 fb b0 ac 6e 2c d5  89 8c f3 87 b4 33 5f 22
  |."...n,......3_"|
00000ed0  a2 e5 7b 00 33 e0 4e cf  e6 35 12 8b 2a c0 44 7f
  |..{.3.N..5..*.D.|
00000ee0  f5 50 98 66 8f 4e 21 b1  3b 3e c1 7a 82 c1 c5 63
  |.P.f.N!.;>.z...c|
00000ef0  3d a4 67 7c 4d 91 3a 27  fa 6c 99 da a3 de 59 b3
  |=.g|M.:'.l....Y.|
00000f00  f6 19 e7 02 76 b7 2c 6f  b7 7c 48 07 49 60 0b d0
  |....v.,o.|H.I`..|
00000f10  85 75 b8 f3 5a 8c 49 89  4b e0 36 9b bf 75 ab 75
  |.u..Z.I.K.6..u.u|
00000f20  dc d1 c0 9a 58 e0 a1 48  92 c2 18 76 65 4d cc b1
  |....X..H...veM..|
00000f30  95 f9 94 bd dd 9c 12 67  a1 fa e3 00 a2 d9 cd 28
  |.......g.......(|
00000f40  4c d5 45 f8 40 f1 94 8f  48 9d c8 ce 3b fa cf 93  |L.E.@
...H...;...|
00000f50  8b b5 2b af 40 3b 9f 37  ba 2c 04 5e 3c fb ec 31
  |..+.@;.7.,.^<..1|
00000f60  04 ab 22 24 01 bc 16 85  b1 13 fb 38 10 ce 42 80
  |.."$.......8..B.|
00000f70  ea 22 3a cd ac a5 c2 61  ee 6b 10 dd 09 dc 5a fa
  |.":....a.k....Z.|
00000f80  48 72 6f fc d5 0c f0 be  db eb c2 e0 68 50 03 2c
  |Hro.........hP.,|
00000f90  0f f8 24 cf e2 da d6 37  52 59 9a c5 9d 88 ed ce
  |..$....7RY......|
00000fa0  db 71 c9 bc d2 3c 9c 98  ce ac 07 92 ef dc 94 c2
  |.q...<..........|
00000fb0  b3 d5 4a c3 4b 27 48 67  9d c6 48 e2 cf 47 ed 20
  |..J.K'Hg..H..G. |
00000fc0  eb 96 bb bc f5 6f 1b c2  94 21 e5 9f 2e 61 35 c3
  |.....o...!...a5.|
00000fd0  04 48 95 d5 ee 76 ba 9b  73 05 e5 bb ef 46 bf 9d
  |.H...v..s....F..|
00000fe0  3f 73 c1 97 04 30 7f 67  b2 6f 64 71 d8 2f 60 07
  |?s...0.g.odq./`.|
00000ff0  f8 f3 fa 17 47 4c 36 d6  f8 b5 a2 7e 56 d2 c5 9a
  |....GL6....~V...|
00001000


2023-11-02T10:59:33.441+0000 7f6724da71c0  1 bluefs mount shared_bdev_used
= 0

2023-11-02T10:59:33.441+0000 7f6724da71c0  1 bluestore(/var/lib/ceph/
c80891ba-55f3-11ed-9389-919f4368965c/osd.9) _prepare_db_environment set
db_paths to db,9500785796710 db.slow,9500785796710

2023-11-02T10:59:33.441+0000 7f6724da71c0 -1 rocksdb: verify_sharding
unable to list column families: NotFound:

2023-11-02T10:59:33.441+0000 7f6724da71c0 -1 bluestore(/var/lib/ceph/
c80891ba-55f3-11ed-9389-919f4368965c/osd.9) _open_db erroring opening db:

2023-11-02T10:59:33.441+0000 7f6724da71c0  1 bluefs umount

2023-11-02T10:59:33.441+0000 7f6724da71c0  1 bdev(0x560257b42c00
/var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9/block) close

2023-11-02T10:59:33.673+0000 7f6724da71c0  1 bdev(0x560257b42400
/var/lib/ceph/c80891ba-55f3-11ed-9389-919f4368965c/osd.9/block) close

Mount failed with '(5) Input/output error'


8 rue greneta, 75003, Paris, FRANCE
enyx.com
<https://antiphishing.vadesecure.com/v4?f=bWE5cld0MzUzR3BLMGVRdLG7jnjJbjinmRApBW0OzqUbjyQUBC-eQoP-QP_on_dj&i=MDNBanNCVWFVVFptR001Wn8FmozqooQiiVpDAryG6ms&k=yXQk&r=THJuMWVjUHZ1WlRrRll4QwzxCNuHhzLGB-WK0uHgjstxD2ZRhTZ1h7LsTGh6AJfd&s=f60431c0fff3b3cbb45928f29fb98ec224de5d2933de1018be4923bd1edfe281&u=https%3A%2F%2Fspam.exegy.com%2Ffmlurlsvc%2F%3FfewReq%3D%3AB%3AJVMxOj89MS19NjklOy1ibzY7OjE7Oi14Ymxlan9%2BeW42OTxpbz88OTxvMzJobm88am48PT07Pj8%2BaTk5PT09PWhtbT8%2BOWo7PS1%2FNjo9Mzg8Oz08MzwtemJvNjg%2FSjNBTW9lOzg7OzI9Jjg%2FSjNBTW97Ozg7OzI9LXloe382ZW5%2FeG5oS25zbmxyJWhkZi1oNj45LWNvZzY7%26url%3Dhttps%253a%252f%252fwww.enyx.com%252f>
<https://antiphishing.vadesecure.com/v4?f=bWE5cld0MzUzR3BLMGVRdLG7jnjJbjinmRApBW0OzqUbjyQUBC-eQoP-QP_on_dj&i=MDNBanNCVWFVVFptR001Wn8FmozqooQiiVpDAryG6ms&k=yXQk&r=THJuMWVjUHZ1WlRrRll4QwzxCNuHhzLGB-WK0uHgjstxD2ZRhTZ1h7LsTGh6AJfd&s=1a1f6be447b254ad7371210a41df6110af466644c9f408c4eeb5b4d98d7f3267&u=https%3A%2F%2Fspam.exegy.com%2Ffmlurlsvc%2F%3FfewReq%3D%3AB%3AJVMxOj89MS19NjklOy1ibzY7OjE7Oi14Ymxlan9%2BeW42aW8yMzM7b2g8OG5vamg4amhtbm5pbTg8bz9ubTo%2FaTM8bmhqbmk6Pi1%2FNjo9Mzg8Oz08MzwtemJvNjg%2FSjNBTW9lOzg7OzI9Jjg%2FSjNBTW97Ozg7OzI9LXloe382ZW5%2FeG5oS25zbmxyJWhkZi1oNj45LWNvZzY7%26url%3Dhttp%253a%252f%252fenyx.com%252f>
*exegy.com
<https://antiphishing.vadesecure.com/v4?f=bWE5cld0MzUzR3BLMGVRdLG7jnjJbjinmRApBW0OzqUbjyQUBC-eQoP-QP_on_dj&i=MDNBanNCVWFVVFptR001Wn8FmozqooQiiVpDAryG6ms&k=yXQk&r=THJuMWVjUHZ1WlRrRll4QwzxCNuHhzLGB-WK0uHgjstxD2ZRhTZ1h7LsTGh6AJfd&s=9a0739ed5650af623f79b66f75ad57547bede45ec4ea62618c7fb8f73ce9592a&u=https%3A%2F%2Fwww.exegy.com%2F>*

*Mohamed*

*Lamdaouar*

*Infrastructure Engineer*

mohamed.lamdaouar@xxxxxxx



Le jeu. 2 nov. 2023 à 12:51, Mohamed LAMDAOUAR <mohamed.lamdaouar@xxxxxxx>
a écrit :

Thanks Robert,

I tried this but I'm stuck. If you have some time, do help me with that I
will be very happy because I'm lost :(



8 rue greneta, 75003, Paris, FRANCE
enyx.com
<https://antiphishing.vadesecure.com/v4?f=bWE5cld0MzUzR3BLMGVRdLG7jnjJbjinmRApBW0OzqUbjyQUBC-eQoP-QP_on_dj&i=MDNBanNCVWFVVFptR001Wn8FmozqooQiiVpDAryG6ms&k=yXQk&r=THJuMWVjUHZ1WlRrRll4QwzxCNuHhzLGB-WK0uHgjstxD2ZRhTZ1h7LsTGh6AJfd&s=f60431c0fff3b3cbb45928f29fb98ec224de5d2933de1018be4923bd1edfe281&u=https%3A%2F%2Fspam.exegy.com%2Ffmlurlsvc%2F%3FfewReq%3D%3AB%3AJVMxOj89MS19NjklOy1ibzY7OjE7Oi14Ymxlan9%2BeW42OTxpbz88OTxvMzJobm88am48PT07Pj8%2BaTk5PT09PWhtbT8%2BOWo7PS1%2FNjo9Mzg8Oz08MzwtemJvNjg%2FSjNBTW9lOzg7OzI9Jjg%2FSjNBTW97Ozg7OzI9LXloe382ZW5%2FeG5oS25zbmxyJWhkZi1oNj45LWNvZzY7%26url%3Dhttps%253a%252f%252fwww.enyx.com%252f>

<https://antiphishing.vadesecure.com/v4?f=bWE5cld0MzUzR3BLMGVRdLG7jnjJbjinmRApBW0OzqUbjyQUBC-eQoP-QP_on_dj&i=MDNBanNCVWFVVFptR001Wn8FmozqooQiiVpDAryG6ms&k=yXQk&r=THJuMWVjUHZ1WlRrRll4QwzxCNuHhzLGB-WK0uHgjstxD2ZRhTZ1h7LsTGh6AJfd&s=1a1f6be447b254ad7371210a41df6110af466644c9f408c4eeb5b4d98d7f3267&u=https%3A%2F%2Fspam.exegy.com%2Ffmlurlsvc%2F%3FfewReq%3D%3AB%3AJVMxOj89MS19NjklOy1ibzY7OjE7Oi14Ymxlan9%2BeW42aW8yMzM7b2g8OG5vamg4amhtbm5pbTg8bz9ubTo%2FaTM8bmhqbmk6Pi1%2FNjo9Mzg8Oz08MzwtemJvNjg%2FSjNBTW9lOzg7OzI9Jjg%2FSjNBTW97Ozg7OzI9LXloe382ZW5%2FeG5oS25zbmxyJWhkZi1oNj45LWNvZzY7%26url%3Dhttp%253a%252f%252fenyx.com%252f>
*exegy.com
<https://antiphishing.vadesecure.com/v4?f=bWE5cld0MzUzR3BLMGVRdLG7jnjJbjinmRApBW0OzqUbjyQUBC-eQoP-QP_on_dj&i=MDNBanNCVWFVVFptR001Wn8FmozqooQiiVpDAryG6ms&k=yXQk&r=THJuMWVjUHZ1WlRrRll4QwzxCNuHhzLGB-WK0uHgjstxD2ZRhTZ1h7LsTGh6AJfd&s=9a0739ed5650af623f79b66f75ad57547bede45ec4ea62618c7fb8f73ce9592a&u=https%3A%2F%2Fwww.exegy.com%2F>*

*Mohamed*

*Lamdaouar*

*Infrastructure Engineer*

mohamed.lamdaouar@xxxxxxx



Le jeu. 2 nov. 2023 à 12:08, Robert Sander <r.sander@xxxxxxxxxxxxxxxxxxx>
a écrit :

Hi,

On 11/2/23 11:28, Mohamed LAMDAOUAR wrote:

    I have 7 machines on CEPH cluster, the service ceph runs on a docker
container.
   Each machine has 4 hdd of data (available) and 2 nvme sssd (bricked)
    During a reboot, the ssd bricked on 4 machines, the data are
available on
the HDD disk but the nvme is bricked and the system is not available.
is it
possible to recover the data of the cluster (the data disk are all
available)

You can try to recover the MON db from the OSDs, as they keep a copy of
it:


https://antiphishing.vadesecure.com/v4?f=V3p0eFlQOUZ4czh2enpJS7j2EMloAUIO32fbOvv14lAE8TBfIaEnIXo1udoGNqFP&i=SHV0Y1JZQjNyckJFa3dUQgXEoh5gS3KVt16QIfDo2EM&k=ZVd0&r=T0hnMlUyVEgwNmlmdHc1NSadQBRlJgURlsomDarMY4-jUWYEpAndmhRvBGKG0NP9&s=ce3a282420f5ed2a0e4f461ee9ce796e4a8a9a8a48926d26f05d22ec8b4821cf&u=https%3A%2F%2Fdocs.ceph.com%2Fen%2Freef%2Frados%2Ftroubleshooting%2Ftroubleshooting-mon%2F%23monitor-store-failures

Regards
--
Robert Sander
Heinlein Consulting GmbH
Schwedter Str. 8/9b, 10119 Berlin


https://antiphishing.vadesecure.com/v4?f=V3p0eFlQOUZ4czh2enpJS7j2EMloAUIO32fbOvv14lAE8TBfIaEnIXo1udoGNqFP&i=SHV0Y1JZQjNyckJFa3dUQgXEoh5gS3KVt16QIfDo2EM&k=ZVd0&r=T0hnMlUyVEgwNmlmdHc1NSadQBRlJgURlsomDarMY4-jUWYEpAndmhRvBGKG0NP9&s=9be6bbd716ad7f48663775ee06c8a10f00ee18518820dff2f66261e44d97e44b&u=https%3A%2F%2Fwww.heinlein-support.de

Tel: 030 / 405051-43
Fax: 030 / 405051-19

Amtsgericht Berlin-Charlottenburg - HRB 220009 B
Geschäftsführer: Peer Heinlein - Sitz: Berlin
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx


_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux