Re: OSD is near full and slow in accessing storage from client

[Cassiano Pilipavicius <cassiano@xxxxxxxxxxx>]

I think that more pgs help to distribute the data more evenly, but I dont know if its recommended with a low OSDs number. I remember read somewhere in the docs a guideline for the mas pgs number/OSD, but was from an really old ceph version, maybe things has changed.

Em 11/12/2017 12:39 PM, gjprabu escreveu:

Hi Cassiano,

       Thanks for your valuable feedback and will wait for some time till new osd sync get complete. Also for by increasing pg count it is the issue will solve? our setup pool size for data and metadata pg number is 250. Is this correct for 7 OSD with 2 replica. Also currently stored data size is 17TB.

ceph osd df

ID WEIGHT  REWEIGHT SIZE   USE    AVAIL %USE  VAR  PGS

0 3.29749  1.00000  3376G  2814G  562G 83.35 1.23 165

1 3.26869  1.00000  3347G  1923G 1423G 57.48 0.85 152

2 3.27339  1.00000  3351G  1980G 1371G 59.10 0.88 161

3 3.24089  1.00000  3318G  2131G 1187G 64.23 0.95 168

4 3.24089  1.00000  3318G  2998G  319G 90.36 1.34 176

5 3.32669  1.00000  3406G  2476G  930G 72.68 1.08 165

6 3.27800  1.00000  3356G  1518G 1838G 45.24 0.67 166

              TOTAL 23476G 15843G 7632G 67.49         

MIN/MAX VAR: 0.67/1.34  STDDEV: 14.53

ceph osd tree

ID WEIGHT   TYPE NAME            UP/DOWN REWEIGHT PRIMARY-AFFINITY

-1 22.92604 root default                                          

-2  3.29749     host intcfs-osd1                                  

0  3.29749         osd.0             up  1.00000          1.00000

-3  3.26869     host intcfs-osd2                                  

1  3.26869         osd.1             up  1.00000          1.00000

-4  3.27339     host intcfs-osd3                                  

2  3.27339         osd.2             up  1.00000          1.00000

-5  3.24089     host intcfs-osd4                                  

3  3.24089         osd.3             up  1.00000          1.00000

-6  3.24089     host intcfs-osd5                                  

4  3.24089         osd.4             up  1.00000          1.00000

-7  3.32669     host intcfs-osd6                                  

5  3.32669         osd.5             up  1.00000          1.00000

-8  3.27800     host intcfs-osd7                                  

6  3.27800         osd.6             up  1.00000          1.00000

ceph osd pool ls detail

pool 0 'rbd' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 64 pgp_num 64 last_change 1 flags hashpspool stripe_width 0

pool 3 'downloads_data' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 250 pgp_num 250 last_change 39 flags hashpspool crash_replay_interval 45 stripe_width 0

pool 4 'downloads_metadata' replicated size 2 min_size 1 crush_ruleset 0 object_hash rjenkins pg_num 250 pgp_num 250 last_change 36 flags hashpspool stripe_width 0

Regards

Prabu GJ

---- On Sun, 12 Nov 2017 19:20:34 +0530 Cassiano Pilipavicius <cassiano@xxxxxxxxxxx> wrote ----

I am also not an expert, but it looks like you have big data volumes on few PGs, from what I've seen, the pg data is only deleted from the old OSD when is completed copied to the new osd.

So, if 1 pg have 100G por example, only when it is fully copied to the new OSD, the space will be released on the old OSD.

If you have a busy cluster/network, it may take a good while. Maybe just wait a litle and check from time to time and the space will eventually be released.

Em 11/12/2017 11:44 AM, Sébastien VIGNERON escreveu:

_______________________________________________

ceph-users mailing list

ceph-users@xxxxxxxxxxxxxx

http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

I’m not an expert either so if someone in the list have some ideas on this problem, don’t be shy, share them with us.

For now, I only have hypothese that the OSD space will be recovered as soon as the recovery process is complete. 

Hope everything will get back in order soon (before reaching 95% or above).

I saw some messages on the list about the fstrim tool which can help reclaim unused free space, but i don’t know if it’s apply to your case.

Cordialement / Best regards,

Sébastien VIGNERON 

CRIANN, 

Ingénieur / Engineer

Technopôle du Madrillet 

745, avenue de l'Université 

76800 Saint-Etienne du Rouvray - France 

tél. +33 2 32 91 42 91 

fax. +33 2 32 91 42 92 

http://www.criann.fr 

mailto:sebastien.vigneron@xxxxxxxxx

support: support@xxxxxxxxx

Le 12 nov. 2017 à 13:29, gjprabu <gjprabu@xxxxxxxxxxxx> a écrit :

Hi Sebastien,

    Below is the query details. I am not that much expert and still learning . pg's are not stuck stat before adding osd and pg are slowly clearing stat to active-clean. Today morning there was around 53 active+undersized+degraded+remapped+wait_backfill and now it is 21 only, hope its going on and i am seeing the space keep increasing in newly added OSD (osd.6) 

ID WEIGHT  REWEIGHT SIZE   USE    AVAIL %USE  VAR  PGS

0 3.29749  1.00000  3376G  2814G  562G 83.35 1.23 165  ( Available Spaces not reduced after adding new OSD)

1 3.26869  1.00000  3347G  1923G 1423G 57.48 0.85 152

2 3.27339  1.00000  3351G  1980G 1371G 59.10 0.88 161

3 3.24089  1.00000  3318G  2131G 1187G 64.23 0.95 168

4 3.24089  1.00000  3318G  2998G  319G 90.36 1.34 176  ( Available Spaces not reduced after adding new OSD)

5 3.32669  1.00000  3406G  2476G  930G 72.68 1.08 165  ( Available Spaces not reduced after adding new OSD)

6 3.27800  1.00000  3356G  1518G 1838G 45.24 0.67 166

              TOTAL 23476G 15843G 7632G 67.49         

MIN/MAX VAR: 0.67/1.34  STDDEV: 14.53

...

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com