To me this sounds more like either your MONs didn't have a quorum
anymore or your clients didn't have all MONs in their ceph.conf, maybe
just the failed one? Then the issue is resolved now?
Zitat von Murilo Morais <murilo@xxxxxxxxxxxxxx>:
> Unfortunately I can't verify if ceph reports any inactive PG. As soon as
> the second host disconnects practically everything is locked, nothing
> appears even using "ceph -w". It only appears that the OSDs are offline
> when dcs2 returns.
>
> Note: Apparently there was a new update recently. When I was in the test
> environment, this behavior was not happening, dcs1 was UP with all
services
> without crashing even with dcs2 DOWN, performing reading and writing,
even
> without adding dcs3.
>
> ### COMMANDS ###
> [ceph: root@dcs1 /]# ceph osd tree
> ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
> -1 65.49570 root default
> -3 32.74785 host dcs1
> 0 hdd 2.72899 osd.0 up 1.00000 1.00000
> 1 hdd 2.72899 osd.1 up 1.00000 1.00000
> 2 hdd 2.72899 osd.2 up 1.00000 1.00000
> 3 hdd 2.72899 osd.3 up 1.00000 1.00000
> 4 hdd 2.72899 osd.4 up 1.00000 1.00000
> 5 hdd 2.72899 osd.5 up 1.00000 1.00000
> 6 hdd 2.72899 osd.6 up 1.00000 1.00000
> 7 hdd 2.72899 osd.7 up 1.00000 1.00000
> 8 hdd 2.72899 osd.8 up 1.00000 1.00000
> 9 hdd 2.72899 osd.9 up 1.00000 1.00000
> 10 hdd 2.72899 osd.10 up 1.00000 1.00000
> 11 hdd 2.72899 osd.11 up 1.00000 1.00000
> -5 32.74785 host dcs2
> 12 hdd 2.72899 osd.12 up 1.00000 1.00000
> 13 hdd 2.72899 osd.13 up 1.00000 1.00000
> 14 hdd 2.72899 osd.14 up 1.00000 1.00000
> 15 hdd 2.72899 osd.15 up 1.00000 1.00000
> 16 hdd 2.72899 osd.16 up 1.00000 1.00000
> 17 hdd 2.72899 osd.17 up 1.00000 1.00000
> 18 hdd 2.72899 osd.18 up 1.00000 1.00000
> 19 hdd 2.72899 osd.19 up 1.00000 1.00000
> 20 hdd 2.72899 osd.20 up 1.00000 1.00000
> 21 hdd 2.72899 osd.21 up 1.00000 1.00000
> 22 hdd 2.72899 osd.22 up 1.00000 1.00000
> 23 hdd 2.72899 osd.23 up 1.00000 1.00000
>
>
> [ceph: root@dcs1 /]# ceph osd pool ls detail
> pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash
> rjenkins pg_num 1 pgp_num 1 autoscale_mode on last_change 26 flags
> hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr
> pool 2 'cephfs.ovirt_hosted_engine.meta' replicated size 2 min_size 1
> crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on
> last_change 77 lfor 0/0/47 flags hashpspool stripe_width 0
> pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
> pool 3 'cephfs.ovirt_hosted_engine.data' replicated size 2 min_size 1
> crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on
> last_change 179 lfor 0/0/47 flags hashpspool max_bytes 107374182400
> stripe_width 0 application cephfs
> pool 6 '.nfs' replicated size 2 min_size 1 crush_rule 0 object_hash
> rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 254 lfor
> 0/0/252 flags hashpspool stripe_width 0 application nfs
> pool 7 'cephfs.ovirt_storage_sas.meta' replicated size 2 min_size 1
> crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on
> last_change 322 lfor 0/0/287 flags hashpspool stripe_width 0
> pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
> pool 8 'cephfs.ovirt_storage_sas.data' replicated size 2 min_size 1
> crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on
> last_change 291 lfor 0/0/289 flags hashpspool stripe_width 0 application
> cephfs
> pool 9 'cephfs.ovirt_storage_iso.meta' replicated size 2 min_size 1
> crush_rule 0 object_hash rjenkins pg_num 16 pgp_num 16 autoscale_mode on
> last_change 356 lfor 0/0/325 flags hashpspool stripe_width 0
> pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs
> pool 10 'cephfs.ovirt_storage_iso.data' replicated size 2 min_size 1
> crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on
> last_change 329 lfor 0/0/327 flags hashpspool stripe_width 0 application
> cephfs
>
>
> [ceph: root@dcs1 /]# ceph osd crush rule dump replicated_rule
> {
> "rule_id": 0,
> "rule_name": "replicated_rule",
> "type": 1,
> "steps": [
> {
> "op": "take",
> "item": -1,
> "item_name": "default"
> },
> {
> "op": "chooseleaf_firstn",
> "num": 0,
> "type": "host"
> },
> {
> "op": "emit"
> }
> ]
> }
>
>
> [ceph: root@dcs1 /]# ceph pg ls-by-pool cephfs.ovirt_hosted_engine.data
> PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES*
> OMAP_KEYS* LOG STATE SINCE VERSION REPORTED UP
> ACTING SCRUB_STAMP DEEP_SCRUB_STAMP
> LAST_SCRUB_DURATION SCRUB_SCHEDULING
> 3.0 69 0 0 0 285213095 0
> 0 10057 active+clean 41m 530'20632 530:39461 [1,23]p1
> [1,23]p1 2022-10-13T03:19:33.649837+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T07:24:46.314217+0000
> 3.1 58 0 0 0 242319360 0
> 0 10026 active+clean 41m 530'11926 530:21424 [6,19]p6
> [6,19]p6 2022-10-13T02:15:23.395162+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T11:42:17.682881+0000
> 3.2 71 0 0 0 294629376 0
> 0 10012 active+clean 41m 530'12312 530:25506 [10,16]p10
> [10,16]p10 2022-10-13T06:12:48.839013+0000
> 2022-10-11T21:09:49.405860+0000 1 periodic scrub
> scheduled @ 2022-10-14T12:35:23.917129+0000
> 3.3 63 0 0 0 262520832 0
> 0 10073 active+clean 41m 530'20204 530:42834 [13,11]p13
> [13,11]p13 2022-10-13T01:16:17.672947+0000
> 2022-10-11T16:43:27.935298+0000 1 periodic scrub
> scheduled @ 2022-10-14T11:48:42.643271+0000
> 3.4 59 0 0 0 240611328 0
> 0 10017 active+clean 41m 530'17883 530:32537 [10,22]p10
> [10,22]p10 2022-10-12T22:09:09.376552+0000
> 2022-10-10T15:00:52.196397+0000 1 periodic scrub
> scheduled @ 2022-10-14T01:16:35.682204+0000
> 3.5 67 0 0 0 281018368 0
> 0 10017 active+clean 41m 530'18825 530:31531 [18,3]p18
> [18,3]p18 2022-10-12T18:13:50.835870+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T02:17:12.292237+0000
> 3.6 60 0 0 0 239497216 0
> 0 10079 active+clean 41m 530'22537 530:34790 [0,21]p0
> [0,21]p0 2022-10-12T20:38:44.998414+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T08:12:12.106892+0000
> 3.7 54 0 0 0 221261824 0
> 0 10082 active+clean 41m 530'30718 530:37349 [4,12]p4
> [4,12]p4 2022-10-12T20:26:54.091307+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-13T20:51:54.792643+0000
> 3.8 70 0 0 0 293588992 0
> 0 4527 active+clean 41m 530'4527 530:16905 [11,21]p11
> [11,21]p11 2022-10-13T07:16:50.226814+0000
> 2022-10-10T14:57:31.136809+0000 1 periodic scrub
> scheduled @ 2022-10-14T13:02:27.444761+0000
> 3.9 47 0 0 0 192938407 0
> 0 10065 active+clean 41m 530'11065 530:21345 [19,11]p19
> [19,11]p19 2022-10-13T05:05:36.274216+0000
> 2022-10-10T14:57:31.136809+0000 1 periodic scrub
> scheduled @ 2022-10-14T08:17:25.165367+0000
> 3.a 60 0 0 0 251658240 0
> 0 10044 active+clean 41m 530'14744 530:23145 [18,1]p18
> [18,1]p18 2022-10-13T04:29:38.891055+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T11:10:38.556482+0000
> 3.b 52 0 0 0 209567744 0
> 0 4949 active+clean 41m 530'4949 530:26757 [7,23]p7
> [7,23]p7 2022-10-12T22:08:45.621201+0000
2022-10-10T15:00:36.799456+0000
> 1 periodic scrub scheduled @
> 2022-10-14T02:28:08.061560+0000
> 3.c 68 0 0 0 276607307 0
> 0 10003 active+clean 41m 530'18828 530:39884 [18,8]p18
> [18,8]p18 2022-10-12T18:25:36.991393+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T00:43:12.804024+0000
> 3.d 67 0 0 0 272621888 0
> 0 6708 active+clean 41m 530'8359 530:33988 [13,7]p13
> [13,7]p13 2022-10-12T21:42:29.600145+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-13T23:30:29.341646+0000
> 3.e 68 0 0 0 276746240 0
> 0 5178 active+clean 41m 530'5278 530:16051 [13,1]p13
> [13,1]p13 2022-10-13T05:47:06.004714+0000
2022-10-11T21:04:57.978685+0000
> 1 periodic scrub scheduled @
> 2022-10-14T11:45:33.438178+0000
> 3.f 65 0 0 0 269307904 0
> 0 10056 active+clean 41m 530'34965 530:49963 [23,4]p23
> [23,4]p23 2022-10-13T08:58:09.493284+0000
2022-10-10T15:00:36.390467+0000
> 1 periodic scrub scheduled @
> 2022-10-14T12:18:58.610252+0000
> 3.10 66 0 0 0 271626240 0
> 0 4272 active+clean 41m 530'4431 530:19010 [12,9]p12
> [12,9]p12 2022-10-13T03:52:14.952046+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T07:48:12.441144+0000
> 3.11 58 0 0 0 239075657 0
> 0 6466 active+clean 41m 530'8563 530:24677 [18,0]p18
> [18,0]p18 2022-10-12T22:25:17.255090+0000
2022-10-10T15:00:43.412084+0000
> 1 periodic scrub scheduled @
> 2022-10-14T03:25:34.048845+0000
> 3.12 45 0 0 0 186254336 0
> 0 10084 active+clean 41m 530'16084 530:31273 [6,14]p6
> [6,14]p6 2022-10-13T03:05:14.109923+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T03:35:11.159743+0000
> 3.13 68 0 0 0 275124224 0
> 0 10013 active+clean 41m 530'28676 530:52278 [16,8]p16
> [16,8]p16 2022-10-12T21:46:50.747741+0000
2022-10-11T16:48:56.632027+0000
> 1 periodic scrub scheduled @
> 2022-10-14T07:03:49.125496+0000
> 3.14 58 0 0 0 240123904 0
> 0 7531 active+clean 41m 530'8212 530:26075 [23,4]p23
> [23,4]p23 2022-10-13T04:25:39.131070+0000
2022-10-13T04:25:39.131070+0000
> 4 periodic scrub scheduled @
> 2022-10-14T05:36:16.428326+0000
> 3.15 59 0 0 0 247382016 0
> 0 8890 active+clean 41m 530'8890 530:18892 [23,3]p23
> [23,3]p23 2022-10-13T04:45:48.156899+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T14:55:14.651919+0000
> 3.16 57 0 0 0 237285376 0
> 0 6900 active+clean 41m 530'8766 530:20717 [19,9]p19
> [19,9]p19 2022-10-13T00:13:35.716060+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T07:08:16.779024+0000
> 3.17 56 0 0 0 234303488 0
> 0 10012 active+clean 41m 530'21461 530:31490 [0,13]p0
> [0,13]p0 2022-10-13T07:42:57.775955+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T14:52:30.758744+0000
> 3.18 47 0 0 0 197132288 0
> 0 10001 active+clean 41m 530'14783 530:20829 [10,14]p10
> [10,14]p10 2022-10-13T00:41:44.050740+0000
> 2022-10-10T14:57:31.136809+0000 1 periodic scrub
> scheduled @ 2022-10-14T09:30:02.438044+0000
> 3.19 50 0 0 0 209715200 0
> 0 10058 active+clean 41m 499'19880 530:27891 [8,23]p8
> [8,23]p8 2022-10-13T10:58:13.948274+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T19:55:12.268345+0000
> 3.1a 58 0 0 0 240123904 0
> 0 10037 active+clean 41m 530'36799 530:50997 [16,9]p16
> [16,9]p16 2022-10-13T02:03:18.026427+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T04:55:58.684437+0000
> 3.1b 53 0 0 0 219996160 0
> 0 10051 active+clean 41m 530'18388 530:29223 [0,22]p0
> [0,22]p0 2022-10-12T19:19:25.675030+0000
2022-10-12T19:19:25.675030+0000
> 4 periodic scrub scheduled @
> 2022-10-14T00:21:49.935082+0000
> 3.1c 66 0 0 0 276762624 0
> 0 10027 active+clean 41m 530'16327 530:38127 [20,5]p20
> [20,5]p20 2022-10-13T00:04:49.227288+0000
2022-10-10T15:00:38.834351+0000
> 1 periodic scrub scheduled @
> 2022-10-14T01:15:26.524544+0000
> 3.1d 49 0 0 0 201327104 0
> 0 10020 active+clean 41m 530'26433 530:51593 [17,9]p17
> [17,9]p17 2022-10-13T03:49:02.466987+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T09:04:39.909179+0000
> 3.1e 61 0 0 0 249098595 0
> 0 8790 active+clean 41m 530'8790 530:17807 [3,21]p3
> [3,21]p3 2022-10-12T22:28:19.417597+0000
2022-10-10T15:00:39.474873+0000
> 1 periodic scrub scheduled @
> 2022-10-13T23:49:55.974786+0000
> 3.1f 53 0 0 0 222056448 0
> 0 10053 active+clean 41m 530'35776 530:50234 [0,15]p0
> [0,15]p0 2022-10-13T07:16:46.787818+0000
2022-10-10T14:57:31.136809+0000
> 1 periodic scrub scheduled @
> 2022-10-14T16:24:45.860894+0000
>
> * NOTE: Omap statistics are gathered during deep scrub and may be
> inaccurate soon afterwards depending on utilization. See
> http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for
> further details.
>
> Em qui., 13 de out. de 2022 às 13:54, Eugen Block <eblock@xxxxxx>
escreveu:
>
>> Could you share more details? Does ceph report inactive PGs when one
>> node is down? Please share:
>> ceph osd tree
>> ceph osd pool ls detail
>> ceph osd crush rule dump <rule of affected pool>
>> ceph pg ls-by-pool <affected pool>
>> ceph -s
>>
>> Zitat von Murilo Morais <murilo@xxxxxxxxxxxxxx>:
>>
>> > Thanks for answering.
>> > Marc, but there is no mechanism to prevent IO pause? At the moment I
>> don't
>> > worry about data loss.
>> > I understand that putting it as replica x1 can work, but I need it to
be
>> x2.
>> >
>> > Em qui., 13 de out. de 2022 às 12:26, Marc <Marc@xxxxxxxxxxxxxxxxx>
>> > escreveu:
>> >
>> >>
>> >> >
>> >> > I'm having strange behavior on a new cluster.
>> >>
>> >> Not strange, by design
>> >>
>> >> > I have 3 machines, two of them have the disks. We can name them
like
>> >> > this:
>> >> > dcs1 to dcs3. The dcs1 and dcs2 machines contain the disks.
>> >> >
>> >> > I started bootstrapping through dcs1, added the other hosts and
left
>> mgr
>> >> > on
>> >> > dcs3 only.
>> >> >
>> >> > What is happening is that if I take down dcs2 everything hangs and
>> >> > becomes
>> >> > irresponsible, including the mount points that were pointed to
dcs1.
>> >>
>> >> You have to have disks in 3 machines. (Or set the replication to 1x)
>> >>
>> > _______________________________________________
>> > ceph-users mailing list -- ceph-users@xxxxxxx
>> > To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
>>
>>
>> _______________________________________________
>> ceph-users mailing list -- ceph-users@xxxxxxx
>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>>
