On Thu, Mar 29, 2018 at 3:17 PM Damian Dabrowski <scooty96@xxxxxxxxx> wrote:
Greg, thanks for Your reply!
I think Your idea makes sense, I've did tests and its quite hard to
understand for me. I'll try to explain my situation in few steps
below.
I think that ceph showing progress in recovery but it can only solve
objects which doesn't really changed. It won't try to repair objects
which are really degraded because of norecovery flag. Am i right?
After a while I see blocked requests(as You can see below).
Yeah, so the implementation of this is a bit funky. Basically, when the OSD gets a map specifying norecovery, it will prevent any new recovery ops from starting once it processes that map. But it doesn't change the state of the PGs out of recovery; they just won't queue up more work.
So probably the existing recovery IO was from OSDs that weren't up-to-date yet. Or maybe there's a bug in the norecover implementation; it definitely looks a bit fragile.
But really I just wouldn't use that command. It's an expert flag that you shouldn't use except in some extreme wonky cluster situations (and even those may no longer exist in modern Ceph). For the use case you shared in your first email, I'd just stick with noout.
-Greg
----- FEW SEC AFTER START OSD -----
# ceph status
cluster 848b340a-be27-45cb-ab66-3151d877a5a0
health HEALTH_WARN
140 pgs degraded
1 pgs recovering
92 pgs recovery_wait
140 pgs stuck unclean
recovery 942/5772119 objects degraded (0.016%)
noout,nobackfill,norecover flag(s) set
monmap e10: 3 mons at
{node-19=172.31.0.2:6789/0,node-20=172.31.0.8:6789/0,node-21=172.31.0.6:6789/0}
election epoch 724, quorum 0,1,2 node-19,node-21,node-20
osdmap e18727: 36 osds: 36 up, 30 in
flags noout,nobackfill,norecover
pgmap v20851644: 1472 pgs, 7 pools, 8510 GB data, 1880 kobjects
25204 GB used, 17124 GB / 42329 GB avail
942/5772119 objects degraded (0.016%)
1332 active+clean
92 active+recovery_wait+degraded
47 active+degraded
1 active+recovering+degraded
recovery io 31608 kB/s, 4 objects/s
client io 73399 kB/s rd, 80233 kB/s wr, 1218 op/s
----- 1 MIN AFTER OSD START, RECOVERY STUCK, BLOCKED REQUESTS -----
# ceph status
cluster 848b340a-be27-45cb-ab66-3151d877a5a0
health HEALTH_WARN
140 pgs degraded
1 pgs recovering
109 pgs recovery_wait
140 pgs stuck unclean
80 requests are blocked > 32 sec
recovery 847/5775929 objects degraded (0.015%)
noout,nobackfill,norecover flag(s) set
monmap e10: 3 mons at
{node-19=172.31.0.2:6789/0,node-20=172.31.0.8:6789/0,node-21=172.31.0.6:6789/0}
election epoch 724, quorum 0,1,2 node-19,node-21,node-20
osdmap e18727: 36 osds: 36 up, 30 in
flags noout,nobackfill,norecover
pgmap v20851812: 1472 pgs, 7 pools, 8520 GB data, 1881 kobjects
25234 GB used, 17094 GB / 42329 GB avail
847/5775929 objects degraded (0.015%)
1332 active+clean
109 active+recovery_wait+degraded
30 active+degraded <---- degraded objects count got stuck
1 active+recovering+degraded
recovery io 3743 kB/s, 0 objects/s <---- depend on command execution
this line showing 0 objects/s or doesn't exists
client io 26521 kB/s rd, 64211 kB/s wr, 1212 op/s
----- FEW SECONDS AFTER UNSETTING FLAGS NOOUT, NORECOVERY, NOBACKFILL -----
# ceph status
cluster 848b340a-be27-45cb-ab66-3151d877a5a0
health HEALTH_WARN
134 pgs degraded
134 pgs recovery_wait
134 pgs stuck degraded
134 pgs stuck unclean
recovery 591/5778179 objects degraded (0.010%)
monmap e10: 3 mons at
{node-19=172.31.0.2:6789/0,node-20=172.31.0.8:6789/0,node-21=172.31.0.6:6789/0}
election epoch 724, quorum 0,1,2 node-19,node-21,node-20
osdmap e18730: 36 osds: 36 up, 30 in
pgmap v20851909: 1472 pgs, 7 pools, 8526 GB data, 1881 kobjects
25252 GB used, 17076 GB / 42329 GB avail
591/5778179 objects degraded (0.010%)
1338 active+clean
134 active+recovery_wait+degraded
recovery io 191 MB/s, 26 objects/s
client io 100654 kB/s rd, 184 MB/s wr, 6303 op/s
2018-03-29 18:22 GMT+02:00 Gregory Farnum <gfarnum@xxxxxxxxxx>:
>
> On Thu, Mar 29, 2018 at 7:27 AM Damian Dabrowski <scooty96@xxxxxxxxx> wrote:
>>
>> Hello,
>>
>> Few days ago I had very strange situation.
>>
>> I had to turn off few OSDs for a while. So I've set flags:noout,
>> nobackfill, norecover and then turned off selected OSDs.
>> All was ok, but when I started these OSDs again all VMs went down due
>> to recovery process(even when recovery priority was very low).
>
>
> So you forbade the OSDs from doing any recovery work, but then you turned on
> old ones that required recovery work to function properly?
>
> And your cluster stopped functioning?
>
>
>>
>>
>> There's more important config values:
>> "osd_recovery_threads": "1",
>> "osd_recovery_thread_timeout": "30",
>> "osd_recovery_thread_suicide_timeout": "300",
>> "osd_recovery_delay_start": "0",
>> "osd_recovery_max_active": "1",
>> "osd_recovery_max_single_start": "5",
>> "osd_recovery_max_chunk": "8388608",
>> "osd_client_op_priority": "63",
>> "osd_recovery_op_priority": "1",
>> "osd_recovery_op_warn_multiple": "16",
>> "osd_backfill_full_ratio": "0.85",
>> "osd_backfill_retry_interval": "10",
>> "osd_backfill_scan_min": "64",
>> "osd_backfill_scan_max": "512",
>> "osd_kill_backfill_at": "0",
>> "osd_max_backfills": "1",
>>
>>
>>
>> I don't know why ceph started recovery process if there was
>> norecovery&nobackfill flags enabled but the fact is that it killed all
>> VMs.
>
>
> Did it actually start recovering? Or you just saw client IO pause?
> I confess I don’t know what the behavior will be like with that combined set
> of flags, but I rather suspect it did what you told it to, and some PGs went
> down as a result.
> -Greg
>
>
>>
>>
>> Next, I've turned off noout, nobackfill, norecover flags and it
>> started to look better. VM's went back online and recovery process was
>> still going. I didn't saw performance impact on SSD disks but there
>> was huge impact on spinners.
>> Normally %util is about 25%, but during recovery it was nearly 100%.
>> CPU Load increased on HDD based VMs by ~400%.
>>
>> iostat fragment(during recovery):
>> Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s
>> avgrq-sz avgqu-sz await r_await w_await svctm %util
>> sdh 0.30 1.00 150.90 36.00 13665.60 954.60
>> 156.45 10.63 56.88 25.60 188.02 5.34 99.80
>>
>>
>> Now, I'm little lost, I don't know answers for few questions.
>> 1. Why ceph started recovery even if nobackfill&norecovery option was
>> enabled?
>> 2. Why recovery caused much more performance impact when
>> norecovery&nobackfill options was enabled?
>> 3. Why when norecovery&nobackfill was turned off, cluster started to
>> look better but %util on HDD disks was so big(while
>> recovery_op_priority=1 and client_op_priority=63)? 25% is normal,
>> increased to 100% during recovery?
>>
>>
>> Cluster information:
>> ceph version 0.94.9 (fe6d859066244b97b24f09d46552afc2071e6f90)
>> 3x nodes(CPU E5-2630, 32GB RAM, 6xHDD 2TB with SSD journal, 3x SSD 1TB
>> with NVMe journal), triple replication
>>
>>
>> I would be very grateful If somebody can help me.
>> Sorry if I've done something in wrong way - this is my first time
>> writing on mailing list.
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
