Re: Problem with CephFS - No space left on device

[Rodrigo Embeita <rodrigo@xxxxxxxxxxxxxxx>]

Hi Yoann, thanks a lot for your help.

  
root@pf-us1-dfs3:/home/rodrigo# ceph osd crush tree
ID CLASS WEIGHT   TYPE NAME             
-1       72.77390 root default          
-3       29.10956     host pf-us1-dfs1  
0   hdd  7.27739         osd.0         
5   hdd  7.27739         osd.5         
6   hdd  7.27739         osd.6         
8   hdd  7.27739         osd.8         
-5       29.10956     host pf-us1-dfs2  
1   hdd  7.27739         osd.1         
3   hdd  7.27739         osd.3         
7   hdd  7.27739         osd.7         
9   hdd  7.27739         osd.9         
-7       14.55478     host pf-us1-dfs3  
2   hdd  7.27739         osd.2         
4   hdd  7.27739         osd.4        

root@pf-us1-dfs3:/home/rodrigo# ceph osd crush rule ls
replicated_rule

root@pf-us1-dfs3:/home/rodrigo# ceph osd crush rule dump
[
   {
       "rule_id": 0,
       "rule_name": "replicated_rule",
       "ruleset": 0,
       "type": 1,
       "min_size": 1,
       "max_size": 10,
       "steps": [
           {
               "op": "take",
               "item": -1,
               "item_name": "default"
           },
           {
               "op": "chooseleaf_firstn",
               "num": 0,
               "type": "host"
           },
           {
               "op": "emit"
           }
       ]
   }
]

On Tue, Jan 8, 2019 at 11:35 AM Yoann Moulin <yoann.moulin@xxxxxxx> wrote:

Hello,

> Hi Yoann, thanks for your response.
> Here are the results of the commands.
>
> root@pf-us1-dfs2:/var/log/ceph# ceph osd df
> ID CLASS WEIGHT  REWEIGHT SIZE    USE     AVAIL   %USE  VAR  PGS  
> 0   hdd 7.27739  1.00000 7.3 TiB 6.7 TiB 571 GiB 92.33 1.74 310  
> 5   hdd 7.27739  1.00000 7.3 TiB 5.6 TiB 1.7 TiB 77.18 1.45 271  
> 6   hdd 7.27739  1.00000 7.3 TiB 609 GiB 6.7 TiB  8.17 0.15  49  
> 8   hdd 7.27739  1.00000 7.3 TiB 2.5 GiB 7.3 TiB  0.03    0  42  
> 1   hdd 7.27739  1.00000 7.3 TiB 5.6 TiB 1.7 TiB 77.28 1.45 285  
> 3   hdd 7.27739  1.00000 7.3 TiB 6.9 TiB 371 GiB 95.02 1.79 296  
> 7   hdd 7.27739  1.00000 7.3 TiB 360 GiB 6.9 TiB  4.84 0.09  53  
> 9   hdd 7.27739  1.00000 7.3 TiB 4.1 GiB 7.3 TiB  0.06 0.00  38  
> 2   hdd 7.27739  1.00000 7.3 TiB 6.7 TiB 576 GiB 92.27 1.74 321  
> 4   hdd 7.27739  1.00000 7.3 TiB 6.1 TiB 1.2 TiB 84.10 1.58 351  
>                    TOTAL  73 TiB  39 TiB  34 TiB 53.13           
> MIN/MAX VAR: 0/1.79  STDDEV: 41.15

It looks like you don't have a good balance between your OSD, what is your failure domain ?

could you provide your crush map http://docs.ceph.com/docs/luminous/rados/operations/crush-map/

ceph osd crush tree
ceph osd crush rule ls
ceph osd crush rule dump


> root@pf-us1-dfs2:/var/log/ceph# ceph osd pool ls detail
> pool 1 'poolcephfs' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 128 pgp_num 128 last_change 471 fla
> gs hashpspool,full stripe_width 0
> pool 2 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 471 lf
> or 0/439 flags hashpspool,full stripe_width 0 application cephfs
> pool 3 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 47
> 1 lfor 0/448 flags hashpspool,full stripe_width 0 application cephfs
> pool 4 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 471 flags ha
> shpspool,full stripe_width 0 application rgw
> pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 47
> 1 flags hashpspool,full stripe_width 0 application rgw
> pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 471 f
> lags hashpspool,full stripe_width 0 application rgw
> pool 7 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 last_change 471 fl
> ags hashpspool,full stripe_width 0 application rgw

You may need to increase the pg num for cephfs_data pool. But before, you must understand what is the impact https://ceph.com/pgcalc/
you can't decrease pg_num, if it set too high you may have trouble in your cluster.

> root@pf-us1-dfs2:/var/log/ceph# ceph osd tree
> ID CLASS WEIGHT   TYPE NAME            STATUS REWEIGHT PRI-AFF  
> -1       72.77390 root default                                  
> -3       29.10956     host pf-us1-dfs1                          
> 0   hdd  7.27739         osd.0            up  1.00000 1.00000  
> 5   hdd  7.27739         osd.5            up  1.00000 1.00000  
> 6   hdd  7.27739         osd.6            up  1.00000 1.00000  
> 8   hdd  7.27739         osd.8            up  1.00000 1.00000  
> -5       29.10956     host pf-us1-dfs2                          
> 1   hdd  7.27739         osd.1            up  1.00000 1.00000  
> 3   hdd  7.27739         osd.3            up  1.00000 1.00000  
> 7   hdd  7.27739         osd.7            up  1.00000 1.00000  
> 9   hdd  7.27739         osd.9            up  1.00000 1.00000  
> -7       14.55478     host pf-us1-dfs3                          
> 2   hdd  7.27739         osd.2            up  1.00000 1.00000  
> 4   hdd  7.27739         osd.4            up  1.00000 1.00000

You really should add 2 disks to pf-us1-dfs3, currently, the cluster tries to balance data between the 3 hosts, (replica 3, failure domain set to
'host' I guess). Each host will store 1/3 of data (1 replica) pf-us1-dfs3  only have half of the 2 others, you won't be able to put more than 3x
(osd.2+osd.4) even though there are free spaces on others OSDs.

Best regards,

Yoann

> On Tue, Jan 8, 2019 at 10:36 AM Yoann Moulin <yoann.moulin@xxxxxxx <mailto:yoann.moulin@xxxxxxx>> wrote:
>
>     Hello,
>
>     > Hi guys, I need your help.
>     > I'm new with Cephfs and we started using it as file storage.
>     > Today we are getting no space left on device but I'm seeing that we have plenty space on the filesystem.
>     > Filesystem              Size  Used Avail Use% Mounted on
>     > 192.168.51.8,192.168.51.6,192.168.51.118:6789:/pagefreezer/smhosts   73T   39T   35T  54% /mnt/cephfs
>     >
>     > We have 35TB of disk space. I've added 2 additional OSD disks with 7TB each but I'm getting the error "No space left on device" every time
>     that
>     > I want to add a new file.
>     > After adding the 2 additional OSD disks I'm seeing that the load is beign distributed among the cluster.
>     > Please I need your help.
>
>     Could you give us the output of
>
>     ceph osd df
>     ceph osd pool ls detail
>     ceph osd tree
>
>     Best regards,
>
>     --
>     Yoann Moulin
>     EPFL IC-IT
>     _______________________________________________
>     ceph-users mailing list
>     ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
>     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>


--
Yoann Moulin
EPFL IC-IT
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com