Re: PG problem after reweight (1 PG active+remapped)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Wido,

Yeap its quite old, since 2016. Its from a decommissioned cluster that we just keep in healthy state without much update efforts.
I had in plan to do a clean up of unwanted disks snapshots etc, do a few re-weights, update it to latest stable (just like correctly you mentioned) and then just maintain it. I am not proud of this, but that's the sad story for this cluster.

Thanks for tunables proposal. I will consider this. But I fear that more reweighs will bring more problems.

Can I somehow check the adjustments that are about to be edited with this command prior of enabling them?

I am hoping for a less resource hungry operation that will be focused in this particular PG or problematic OSDS.
Maybe just remove these OSDs completely one by one and reinsert them?
Maybe I can create a new pool and just migrate one by one my disks? Removing old pool should take down with it the faulty PG, correct?

I am starting to get anxious for data integrity or ceph readonly state also in case OSD 6 or 26 have availability issues...

Thanks for quick reply!

Regards,
Nasos Panterlis

From: Wido den Hollander <wido@xxxxxxxx>
Sent: Monday, December 3, 2018 3:53 PM
To: Athanasios Panterlis; ceph-users@xxxxxxxxxxxxxx
Subject: Re: PG problem after reweight (1 PG active+remapped)
 
Hi,

How old is this cluster? As this might be a CRUSH tunables issue where
this pops up.

You can try (might move a lot of data!)

$ ceph osd getcrushmap -o crushmap.backup
$ ceph osd crush tunables optimal

If things go wrong you always have the old CRUSHmap:

$ ceph osd setcrushmap -i crushmap.backup

0.94.10 is EOL as well, I would consider upgrading after this PG becomes
active+clean

Wido

On 12/3/18 2:51 PM, Athanasios Panterlis wrote:
> Hi all,
>
> I am managing a typical small ceph cluster that consists of 4 nodes with
> each one having 7 OSDs (some in hdd pool, some in ssd pool)
>
> Having a healthy cluster and following some space issues due to bad pg
> management from ceph, I tried some reweighs in specific OSDs.
> Unfortunately the reballancing after reweigh a specific pg went to
> active+remmaped state and I have now a couple of misplaced objects
> (3359/6118678 objects misplaced (0.055%))
>
> Cluster had (for iops reasons) the scrub and deep-scrub options
> disabled. I enabled a simple scrub to all PG and waited to run. It
> finished completely without problems I guess. I would like to avoid
> deep-scrub, but if this is going to help I will run it once.
>
> I am thinking of declaring OSD 26 as lost. This way, new pg copy from 6
> will be created, correct?
> Any other less harmful thoughts on how to fix it? 
>
> I attached all the information I could provide. I am also pasting them
> raw below:
>
> *Query for faulty PG:*
>
> {
>     "state": "active+remapped",
>     "snap_trimq": "[]",
>     "epoch": 11755,
>     "up": [
>         6
>     ],
>     "acting": [
>         6,
>         26
>     ],
>     "actingbackfill": [
>         "6",
>         "26"
>     ],
>     "info": {
>         "pgid": "1.11d",
>         "last_update": "11755'60561210",
>         "last_complete": "11755'60561210",
>         "log_tail": "11755'60558123",
>         "last_user_version": 60561210,
>         "last_backfill": "MAX",
>         "purged_snaps": "[1~33,36~22]",
>         "history": {
>             "epoch_created": 31,
>             "last_epoch_started": 11681,
>             "last_epoch_clean": 11681,
>             "last_epoch_split": 0,
>             "same_up_since": 11679,
>             "same_interval_since": 11680,
>             "same_primary_since": 11510,
>             "last_scrub": "449'16483",
>             "last_scrub_stamp": "2016-09-14 15:25:14.228231",
>             "last_deep_scrub": "448'16277",
>             "last_deep_scrub_stamp": "2016-09-13 06:11:45.633007",
>             "last_clean_scrub_stamp": "2016-09-14 15:25:14.228231"
>         },
>         "stats": {
>             "version": "11755'60561210",
>             "reported_seq": "50924585",
>             "reported_epoch": "11755",
>             "state": "active+remapped",
>             "last_fresh": "2018-12-03 12:58:03.289251",
>             "last_change": "2018-11-09 10:54:06.861873",
>             "last_active": "2018-12-03 12:58:03.289251",
>             "last_peered": "2018-12-03 12:58:03.289251",
>             "last_clean": "2018-11-09 10:54:02.622866",
>             "last_became_active": "0.000000",
>             "last_became_peered": "0.000000",
>             "last_unstale": "2018-12-03 12:58:03.289251",
>             "last_undegraded": "2018-12-03 12:58:03.289251",
>             "last_fullsized": "2018-12-03 12:58:03.289251",
>             "mapping_epoch": 11679,
>             "log_start": "11755'60558123",
>             "ondisk_log_start": "11755'60558123",
>             "created": 31,
>             "last_epoch_clean": 11681,
>             "parent": "0.0",
>             "parent_split_bits": 0,
>             "last_scrub": "449'16483",
>             "last_scrub_stamp": "2016-09-14 15:25:14.228231",
>             "last_deep_scrub": "448'16277",
>             "last_deep_scrub_stamp": "2016-09-13 06:11:45.633007",
>             "last_clean_scrub_stamp": "2016-09-14 15:25:14.228231",
>             "log_size": 3087,
>             "ondisk_log_size": 3087,
>             "stats_invalid": "0",
>             "stat_sum": {
>                 "num_bytes": 14031434258,
>                 "num_objects": 3359,
>                 "num_object_clones": 0,
>                 "num_object_copies": 6718,
>                 "num_objects_missing_on_primary": 0,
>                 "num_objects_degraded": 0,
>                 "num_objects_misplaced": 3359,
>                 "num_objects_unfound": 0,
>                 "num_objects_dirty": 3359,
>                 "num_whiteouts": 0,
>                 "num_read": 27359423,
>                 "num_read_kb": 1815932413,
>                 "num_write": 121113356,
>                 "num_write_kb": 2124776643,
>                 "num_scrub_errors": 0,
>                 "num_shallow_scrub_errors": 0,
>                 "num_deep_scrub_errors": 0,
>                 "num_objects_recovered": 65218,
>                 "num_bytes_recovered": 271765903872,
>                 "num_keys_recovered": 0,
>                 "num_objects_omap": 0,
>                 "num_objects_hit_set_archive": 0,
>                 "num_bytes_hit_set_archive": 0
>             },
>             "up": [
>                 6
>             ],
>             "acting": [
>                 6,
>                 26
>             ],
>             "blocked_by": [],
>             "up_primary": 6,
>             "acting_primary": 6
>         },
>         "empty": 0,
>         "dne": 0,
>         "incomplete": 0,
>         "last_epoch_started": 11681,
>         "hit_set_history": {
>             "current_last_update": "0'0",
>             "current_last_stamp": "0.000000",
>             "current_info": {
>                 "begin": "0.000000",
>                 "end": "0.000000",
>                 "version": "0'0",
>                 "using_gmt": "1"
>             },
>             "history": []
>         }
>     },
>     "peer_info": [
>         {
>             "peer": "26",
>             "pgid": "1.11d",
>             "last_update": "11755'60561210",
>             "last_complete": "11755'60561210",
>             "log_tail": "11649'58446601",
>             "last_user_version": 58449647,
>             "last_backfill": "MAX",
>             "purged_snaps": "[1~33,36~22]",
>             "history": {
>                 "epoch_created": 31,
>                 "last_epoch_started": 11681,
>                 "last_epoch_clean": 11681,
>                 "last_epoch_split": 0,
>                 "same_up_since": 11679,
>                 "same_interval_since": 11680,
>                 "same_primary_since": 11510,
>                 "last_scrub": "449'16483",
>                 "last_scrub_stamp": "2016-09-14 15:25:14.228231",
>                 "last_deep_scrub": "448'16277",
>                 "last_deep_scrub_stamp": "2016-09-13 06:11:45.633007",
>                 "last_clean_scrub_stamp": "2016-09-14 15:25:14.228231"
>             },
>             "stats": {
>                 "version": "11678'58449646",
>                 "reported_seq": "48950066",
>                 "reported_epoch": "11678",
>                 "state": "active+clean",
>                 "last_fresh": "2018-11-09 10:54:02.263168",
>                 "last_change": "2018-11-09 08:01:12.116827",
>                 "last_active": "2018-11-09 10:54:02.263168",
>                 "last_peered": "2018-11-09 10:54:02.263168",
>                 "last_clean": "2018-11-09 10:54:02.263168",
>                 "last_became_active": "0.000000",
>                 "last_became_peered": "0.000000",
>                 "last_unstale": "2018-11-09 10:54:02.263168",
>                 "last_undegraded": "2018-11-09 10:54:02.263168",
>                 "last_fullsized": "2018-11-09 10:54:02.263168",
>                 "mapping_epoch": 11679,
>                 "log_start": "11649'58446601",
>                 "ondisk_log_start": "11649'58446601",
>                 "created": 31,
>                 "last_epoch_clean": 11610,
>                 "parent": "0.0",
>                 "parent_split_bits": 0,
>                 "last_scrub": "449'16483",
>                 "last_scrub_stamp": "2016-09-14 15:25:14.228231",
>                 "last_deep_scrub": "448'16277",
>                 "last_deep_scrub_stamp": "2016-09-13 06:11:45.633007",
>                 "last_clean_scrub_stamp": "2016-09-14 15:25:14.228231",
>                 "log_size": 3045,
>                 "ondisk_log_size": 3045,
>                 "stats_invalid": "0",
>                 "stat_sum": {
>                     "num_bytes": 18153595392,
>                     "num_objects": 4344,
>                     "num_object_clones": 0,
>                     "num_object_copies": 8688,
>                     "num_objects_missing_on_primary": 0,
>                     "num_objects_degraded": 0,
>                     "num_objects_misplaced": 0,
>                     "num_objects_unfound": 0,
>                     "num_objects_dirty": 4344,
>                     "num_whiteouts": 0,
>                     "num_read": 26674601,
>                     "num_read_kb": 1767105243,
>                     "num_write": 116892449,
>                     "num_write_kb": 2073693377,
>                     "num_scrub_errors": 0,
>                     "num_shallow_scrub_errors": 0,
>                     "num_deep_scrub_errors": 0,
>                     "num_objects_recovered": 65218,
>                     "num_bytes_recovered": 271765903872,
>                     "num_keys_recovered": 0,
>                     "num_objects_omap": 0,
>                     "num_objects_hit_set_archive": 0,
>                     "num_bytes_hit_set_archive": 0
>                 },
>                 "up": [
>                     6
>                 ],
>                 "acting": [
>                     6,
>                     26
>                 ],
>                 "blocked_by": [],
>                 "up_primary": 6,
>                 "acting_primary": 6
>             },
>             "empty": 0,
>             "dne": 0,
>             "incomplete": 0,
>             "last_epoch_started": 11681,
>             "hit_set_history": {
>                 "current_last_update": "0'0",
>                 "current_last_stamp": "0.000000",
>                 "current_info": {
>                     "begin": "0.000000",
>                     "end": "0.000000",
>                     "version": "0'0",
>                     "using_gmt": "1"
>                 },
>                 "history": []
>             }
>         }
>     ],
>     "recovery_state": [
>         {
>             "name": "Started\/Primary\/Active",
>             "enter_time": "2018-11-09 10:54:06.825830",
>             "might_have_unfound": [],
>             "recovery_progress": {
>                 "backfill_targets": [],
>                 "waiting_on_backfill": [],
>                 "last_backfill_started": "-1\/0\/\/0",
>                 "backfill_info": {
>                     "begin": "-1\/0\/\/0",
>                     "end": "-1\/0\/\/0",
>                     "objects": []
>                 },
>                 "peer_backfill_info": [],
>                 "backfills_in_flight": [],
>                 "recovering": [],
>                 "pg_backend": {
>                     "pull_from_peer": [],
>                     "pushing": []
>                 }
>             },
>             "scrub": {
>                 "scrubber.epoch_start": "0",
>                 "scrubber.active": 0,
>                 "scrubber.waiting_on": 0,
>                 "scrubber.waiting_on_whom": []
>             }
>         },
>         {
>             "name": "Started",
>             "enter_time": "2018-11-09 10:54:05.789621"
>         }
>     ],
>     "agent_state": {}
> }
>
> *Ceph status*
>
>     health HEALTH_WARN
>             1 pgs stuck unclean
>             recovery 3359/6118678 objects misplaced (0.055%)
>             noout,nodeep-scrub flag(s) set
>      monmap e3: 3 mons at
> {0=192.168.1.1:6789/0,1=192.168.1.2:6789/0,2=192.168.1.3:6789/0}
>             election epoch 4882, quorum 0,1,2 0,1,2
>      osdmap e11755: 27 osds: 27 up, 27 in; 1 remapped pgs
>             flags noout,nodeep-scrub
>       pgmap v62734988: 1024 pgs, 2 pools, 10183 GB data, 2557 kobjects
>             23768 GB used, 48720 GB / 72488 GB avail
>             3359/6118678 objects misplaced (0.055%)
>                 1023 active+clean
>                    1 active+remapped
>   client io 141 kB/s rd, 14068 kB/s wr, 925 op/s
>
> 2018-12-03 12:58:52.109913 mon.0 [INF] pgmap v62734987: 1024 pgs: 1
> active+remapped, 1023 active+clean; 10183 GB data, 23768 GB used, 48720
> GB / 72488 GB avail; 8325 kB/s rd, 16182 kB/s wr, 1704 op/s;
> 3359/6118678 objects misplaced (0.055%)
>
> *OSD tree*
>
> ID  WEIGHT   REWEIGHT SIZE   USE    AVAIL  %USE  VAR  TYPE NAME        
>      
> -11 37.19995        - 37204G 10257G 26946G 27.57 0.84 root hdd          
>    
> -12  9.29999        -  9301G  2531G  6769G 27.22 0.83     host hdd-node1
>  18  4.64999  1.00000  4650G  1226G  3424G 26.37 0.80         osd.18    
>    
>  19  4.64999  1.00000  4650G  1305G  3345G 28.06 0.86         osd.19    
>    
> -13  9.29999        -  9301G  2665G  6635G 28.66 0.87     host hdd-node2
>  20  4.64999  1.00000  4650G  1361G  3289G 29.27 0.89         osd.20    
>    
>  21  4.64999  1.00000  4650G  1304G  3346G 28.05 0.86         osd.21    
>    
> -14  9.29999        -  9301G  2628G  6672G 28.26 0.86     host hdd-node3
>  22  4.64999  1.00000  4650G  1396G  3254G 30.02 0.92         osd.22    
>    
>  23  4.64999  1.00000  4650G  1232G  3418G 26.50 0.81         osd.23    
>    
> -15  9.29999        -  9301G  2431G  6869G 26.15 0.80     host hdd-node4
>  24  4.64999  1.00000  4650G  1218G  3432G 26.20 0.80         osd.24    
>    
>  25  4.64999  1.00000  4650G  1213G  3436G 26.09 0.80         osd.25    
>    
>  -1 35.14995        - 35284G 13512G 21771G 38.30 1.17 root default      
>    
>  -2  9.25000        -  9285G  3431G  5853G 36.96 1.13     host node1    
>   0  1.84999  1.00000  1857G   765G  1091G 41.24 1.26         osd.0    
>      
>   1  1.84999  1.00000  1857G   633G  1224G 34.09 1.04         osd.1    
>      
>   6  1.84999  1.00000  1857G   777G  1079G 41.88 1.28         osd.6    
>      
>   7  1.84999  0.89999  1857G   752G  1104G 40.54 1.24         osd.7    
>      
>   8  1.84999  1.00000  1857G   502G  1354G 27.06 0.83         osd.8    
>      
>  -3  9.24995        -  9285G  3562G  5722G 38.37 1.17     host node2    
>   2  1.84999  1.00000  1857G   766G  1090G 41.27 1.26         osd.2    
>      
>   3  1.84999  0.70000  1857G   674G  1182G 36.33 1.11         osd.3    
>      
>   9  1.84999  1.00000  1857G   580G  1276G 31.28 0.95         osd.9    
>      
>  10  1.84999  1.00000  1857G   814G  1042G 43.88 1.34         osd.10    
>    
>  11  1.84999  1.00000  1857G   725G  1131G 39.07 1.19         osd.11    
>    
>  -4  9.25000        -  9285G  3561G  5724G 38.35 1.17     host node3    
>   4  1.84999  1.00000  1857G   684G  1172G 36.88 1.12         osd.4    
>      
>   5  1.84999  1.00000  1857G   633G  1223G 34.11 1.04         osd.5    
>      
>  12  1.84999  1.00000  1857G   696G  1160G 37.49 1.14         osd.12    
>    
>  13  1.84999  0.70000  1857G   741G  1116G 39.90 1.22         osd.13    
>    
>  14  1.84999  0.89999  1857G   805G  1051G 43.37 1.32         osd.14    
>    
>  -5  7.39999        -  7428G  2957G  4470G 39.81 1.21     host node4    
>  15  1.84999  0.79999  1857G   742G  1115G 39.96 1.22         osd.15    
>    
>  16  1.84999  1.00000  1857G   634G  1222G 34.15 1.04         osd.16    
>    
>  17  1.84999  0.89999  1857G   803G  1053G 43.26 1.32         osd.17    
>    
>  26  1.84999  0.81000  1857G   777G  1079G 41.89 1.28         osd.26    
>    
>                 TOTAL 72488G 23770G 48718G 32.79                        
>    
> MIN/MAX VAR: 0.80/1.34  STDDEV: 6.56
>
> *PG dump*
>
> version 62735224
> stamp 2018-12-03 13:02:52.799643
> last_osdmap_epoch 11755
> last_pg_scan 9537
> full_ratio 0.95
> nearfull_ratio 0.85
> pg_stat objects mip degr misp unf bytes log disklog state state_stamp v
> reported up up_primary acting acting_primary last_scrub scrub_stamp
> last_deep_scrub deep_scrub_stamp
>
> ///active+clean ones removed///
>
> 1.11d 3359 0 0 3359 0 14031434258 3034 3034 active+remapped 2018-11-09
> 10:54:06.861873 11755'60561357 11755:50924695 [6] 6 [6,26] 6 449'16483
> 2016-09-14 15:25:14.228231 448'16277 2016-09-13 06:11:45.633007
>
> ///active+clean ones removed///
>
> pool 1 1738101 0 0 3359 0 7253679601802 1562466 1562466
> pool 2 881071 0 0 0 0 3682217717410 1561924 1561924
>  sum 2619172 0 0 3359 0 10935897319212 3124390 3124390
> osdstat kbused kbavail kb hb in hb out
> 0 803034868 1144251736 1947286604 [1,2,3,4,5,10,12,13,14,15,16,26] []
> 1 663754736 1283531868 1947286604 [0,2,3,5,10,12,13,15,16,17,26] []
> 2 803619260 1143667344 1947286604 [0,1,3,4,5,6,7,8,13,14,15,16,17,26] []
> 3 707438640 1239847964 1947286604 [0,1,2,4,5,6,7,8,12,13,14,15,17] []
> 4 718194072 1229092532 1947286604 [2,3,5,6,8,10,11,15,16,17,26] []
> 5 664279112 1283007492 1947286604 [0,1,2,3,4,6,8,10,11,15,17,26] []
> 6 815455088 1131831516 1947286604 [2,4,5,7,9,10,11,12,13,14,15,16,17,26] []
> 7 789396940 1157889664 1947286604 [2,4,6,8,9,10,11,12,13,14,15,16,17,26] []
> 8 526871252 1420415352 1947286604 [2,3,4,5,7,9,10,11,12,13,15,16,17] []
> 9 609147992 1338138612 1947286604 [0,1,4,7,8,10,13,14,15,16,17,26] []
> 10 854451916 1092834688 1947286604
> [0,1,4,5,7,8,9,11,12,13,14,15,16,17,26] []
> 11 760893328 1186393276 1947286604 [1,4,5,6,7,8,10,12,13,14,15,16,17,26] []
> 12 730109256 1217177348 1947286604 [0,6,7,8,9,10,11,13,15,16,17,26] []
> 13 777029008 1170257596 1947286604 [0,1,2,3,6,7,8,9,10,11,12,14,15,16] []
> 14 844469760 1102816844 1947286604 [0,1,2,3,6,7,9,10,11,13,15,16,17,26] []
> 15 778122444 1169164160 1947286604 [0,2,3,4,6,9,10,11,14,16] []
> 16 664960388 1282326216 1947286604 [1,2,3,4,5,6,7,8,10,11,12,13,15,17] []
> 17 842428012 1104858592 1947286604 [0,1,2,3,4,5,8,9,10,11,12,13,14,16,18] []
> 18 1285869748 3590537232 4876406980 [0,1,17,20,21,22,23,24,25,26] []
> 19 1368764192 3507642788 4876406980 [0,1,18,20,21,22,23,24,25,26] []
> 20 1427417120 3448989860 4876406980 [4,17,18,19,21,22,23,24,25,26] []
> 21 1367928664 3508478316 4876406980 [4,5,18,19,20,22,23,24,25,26] []
> 22 1464361956 3412045024 4876406980 [4,5,18,19,20,21,23,24,25,26] []
> 23 1292415092 3583991888 4876406980 [4,5,18,19,20,21,22,24,25,26] []
> 24 1277731204 3598675776 4876406980 [4,5,18,19,20,21,22,23,25,26] []
> 25 1272703828 3603703152 4876406980 [4,5,18,19,20,21,22,23,24,26] []
> 26 815682292 1131604312 1947286604 [0,1,2,3,4,5,6,7,8,10,11,12,13,25] []
>
> *Ceph health detail*
>
> HEALTH_WARN 1 pgs stuck unclean; recovery 3359/6120420 objects misplaced
> (0.055%); noout,nodeep-scrub flag(s) set
> pg 1.11d is stuck unclean for 2081576.511195, current state
> active+remapped, last acting [6,26]
> recovery 3359/6120420 objects misplaced (0.055%)
> noout,nodeep-scrub flag(s) set
>
> *Ceph version*
>
> ceph version 0.94.10 (b1e0532418e4631af01acbc0cedd426f1905f4af)
>
> Regards,
> Nasos Pan
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux