Re: CephFS: No space left on device

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Mon, Oct 10, 2016 at 9:06 AM, Davie De Smet
<davie.desmet@xxxxxxxxxxxx> wrote:
> Hi,
>
>
>
> I don’t want to hijack this topic but the behavior described below is the
> same as what I am seeing:
>
>
>
>
>
> [root@osd5-freu ~]# ceph daemonperf /var/run/ceph/ceph-mds.osd5-freu.asok
> |
>
> -----mds------ --mds_server-- ---objecter--- -----mds_cache-----
> ---mds_log----
>
> rlat inos caps|hsr  hcs  hcr |writ read actv|recd recy stry purg|segs evts
> subm|
>
>   0  8.0M 3.4M|  0    0    3 |  0    0    0 |  0    0  2.8M   0 | 33   24k
> 5
>
>   5  8.0M 3.4M|  0    0  230 | 24    1    0 |  0    0  2.8M   0 | 33   24k
> 381
>
>   1  8.0M 3.4M|  0    0  130 | 16    0    0 |  0    0  2.8M   0 | 34   24k
> 366
>
>   4  8.0M 3.4M|  0    0   63 | 96    1    2 |  0    0  2.8M   0 | 34   25k
> 129
>
> 30  7.9M 3.4M|  0    1  190 |479    0    3 |  0    0  2.8M  77 | 34   23k
> 414
>
>   0  7.9M 3.4M|  0    0  194 | 21    0    0 |  0    0  2.8M   1 | 32   23k
> 132
>
>   0  8.0M 3.4M|  0    0  181 | 27    0    0 |  0    0  2.8M   0 | 33   24k
> 400
>
>   0  8.0M 3.4M|  0    0  129 | 16    0    1 |  0    0  2.8M   0 | 33   24k
> 109
>
>   3  7.9M 3.4M|  0    0  110 |194    0    3 |  0    0  2.8M  56 | 32   22k
> 261
>
>   0  8.0M 3.4M|  0    0  128 | 20    0    1 |  0    0  2.8M   1 | 32   22k
> 102
>
>   0  8.0M 3.4M|  0    0  164 | 25    0    0 |  0    0  2.8M   1 | 32   23k
> 137
>
>   1  8.0M 3.4M|  0    0  157 | 13    0    0 |  0    0  2.8M   0 | 32   23k
> 515
>
>   0  8.0M 3.4M|  0    0  137 | 24    2    0 |  0    0  2.8M   0 | 32   23k
> 107
>
>
>
>
>
> [root@osd5-freu ~]# ceph daemon mds.osd5-freu perf dump | grep -i
> 'stray\|purge'
>
>         "num_strays": 2829348,
> |
>
>         "num_strays_purging": 1,
> |
>
>         "num_strays_delayed": 1,
> |
>
>         "num_purge_ops": 11,
> |
>
>         "strays_created": 3302679,
> |
>
>         "strays_purged": 463340,
> |
>
>         "strays_reintegrated": 9994,
> |
>
>         "strays_migrated": 0,
>
>
>
>
>
> [root@osd5-freu ~]# ceph daemon mds.osd5-freu config show|grep purge
>
>     "filer_max_purge_ops": "10",
> |
>
>     "mds_max_purge_files": "1000",
> |
>
>     "mds_max_purge_ops": "8192",
> |
>
>     "mds_max_purge_ops_per_pg": "2",
>
>
>
>
>
> When upgrading from infernalis to jewel, I was getting no space left on
> device when deleting files (creating was not a problem). When changing the
> default of mds_bal_fragment_size_max to 3000000, the problem was resolved (I
> was able to delete data again) but I remain with 2.8M strays that don’t get
> purged. I’m using 2 CephFS kernel (4.7.5) clients at the same time. When
> trying to resolve this issue, I bounced into this issue
> http://tracker.ceph.com/issues/13777 . I have tried the following:
>
>
>
> ·         Re-mounting the clients
>
> ·         flushed the journal
>
> ·         restarted the MDS servers (active / passive)
>
> ·         changed the caps for the MDS cephx user from “ ceph auth caps
> mds.osd5-freu mds 'allow' mon 'allow profile mds' osd 'allow rwx' ” to “
> ceph auth caps mds.osd5-freu mds 'allow' mon 'allow rwx' osd 'allow rwx' ”
>
> ·         Tweaked the default settings up for mds_max_purge_files and
> mds_max_purge_ops_per_pg.
>
>
>
> Unfortunatly alll of this did not help. The MDS strays are still at 2.8M and
> growing. We do use hardlinks on our system. Any tips for actions that I can
> take to resolve this?

Hmm. How many hard links do you have? If you hard linked 2.8 million
files and then deleted the original links, you can resolve this by
doing something to open (or maybe even just touch?) the remaining
links (presumably by crawling the whole FS). That's definitely "not
optimal" and I'll bring this up in our dev standup tomorrow.

If you don't have nearly that many hard links, things appear to have
gotten stuck in some way.
-Greg

>
>
>
> Kind regards,
>
>
>
> Davie De Smet
>
>
>
>
>
> From: ceph-users [mailto:ceph-users-bounces@xxxxxxxxxxxxxx] On Behalf Of
> Mykola Dvornik
> Sent: Tuesday, October 4, 2016 2:07 PM
> To: John Spray <jspray@xxxxxxxxxx>
> Cc: ceph-users <ceph-users@xxxxxxxx>
> Subject: Re:  CephFS: No space left on device
>
>
>
> To my best knowledge nobody used hardlinks within fs.
>
> So I have unmounted everything to see what would happen:
>
> [root@005-s-ragnarok ragnarok]# ceph daemon mds.fast-test session ls
> []
>
> -----mds------ --mds_server-- ---objecter--- -----mds_cache-----
> ---mds_log----
> rlat inos caps|hsr  hcs  hcr |writ read actv|recd recy stry purg|segs evts
> subm|
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>   0   99k   0 |  0    0    0 |  0    0    0 |  0    0   27k   0 | 31   27k
> 0
>
> The amount of objects is stry stays the same over the time.
>
> Then I mount one of the clients and start deleting again
>
> ceph tell mds.fast-test injectargs --mds-max-purge-files 64 (default):
> 2016-10-04 13:58:13.754666 7f39e0010700  0 client.1522041 ms_handle_reset on
> XXX.XXX.XXX.XXX:6800/5261
> 2016-10-04 13:58:13.773739 7f39e0010700  0 client.1522042 ms_handle_reset on
> XXX.XXX.XXX.XXX:6800/5261
> mds_max_purge_files = '64' (unchangeable)
>
> -----mds------ --mds_server-- ---objecter--- -----mds_cache-----
> ---mds_log----
> rlat inos caps|hsr  hcs  hcr |writ read actv|recd recy stry purg|segs evts
> subm|
>   0  100k  40k|  0    0  1.1k| 50    0   68 |  0    0   40k  46 | 35   21k
> 635
>   0  100k  40k|  0    0  1.1k| 32    0   68 |  0    0   40k  31 | 35   22k
> 625
>   0  101k  39k|  0    0  935 | 46    0   69 |  0    0   41k  43 | 31   22k
> 516
>   0  101k  39k|  0    0  833 | 80    0   64 |  0    0   41k  75 | 32   23k
> 495
>   0  101k  39k|  0    0  1.1k| 73    0   64 |  0    0   42k  73 | 33   24k
> 649
>   0  100k  39k|  0    0  1.1k| 84    0   68 |  0    0   42k  79 | 31   22k
> 651
>   0  100k  39k|  0    0  1.1k|100    0   67 |  0    0   42k 100 | 31   22k
> 695
>   0  101k  33k|  0    0  1.1k| 38    0   69 |  0    0   43k  36 | 33   23k
> 607
>   0  101k  33k|  0    0  1.1k| 72    0   68 |  0    0   44k  72 | 33   24k
> 668
>   0  102k  33k|  0    0  1.2k| 64    0   68 |  0    0   44k  64 | 34   24k
> 666
>   0  100k  33k|  0    0  1.0k|418    0  360 |  0    0   45k  33 | 35   25k
> 573
>   0  100k  33k|  0    0  1.2k| 19    0  310 |  0    0   45k  19 | 36   25k
> 624
>   0  101k  33k|  0    0  1.2k| 33    0  236 |  0    0   46k  31 | 37   26k
> 633
>   0  102k  33k|  0    0  1.1k| 54    0  176 |  0    0   46k  54 | 37   27k
> 618
>   0  102k  33k|  0    0  1.1k| 65    0  133 |  0    0   47k  63 | 39   27k
> 639
>   0  100k  33k|  0    0  804 | 87    0   93 |  0    0   47k  79 | 39   28k
> 485
>   0  100k  33k|  0    0  1.2k| 62    0   85 |  0    0   48k  62 | 40   28k
> 670
>   0  101k  28k|  0    1  1.0k|109    0   65 |  0    0   48k 103 | 41   29k
> 617
>   0  101k  28k|  0    0  1.1k| 92    0   65 |  0    0   49k  92 | 42   30k
> 690
>   0  102k  28k|  0    0  1.1k| 80    0   65 |  0    0   49k  78 | 43   30k
> 672
>   0  100k  28k|  0    0  1.0k|234    0  261 |  0    0   50k  35 | 34   24k
> 582
>   0  100k  28k|  0    0  1.1k| 71    0  258 |  0    0   50k  71 | 35   25k
> 667
>   0  101k  26k|  0    0  1.2k| 97    0  259 |  0    0   51k  95 | 36   26k
> 706
>   0  102k  26k|  0    0  1.0k| 53    0  258 |  0    0   51k  53 | 37   26k
> 569
>
> ceph tell mds.fast-test injectargs --mds-max-purge-files 1000:
> 2016-10-04 14:03:20.449961 7fd9e1012700  0 client.1522044 ms_handle_reset on
> XXX.XXX.XXX.XXX:6800/5261
> 2016-10-04 14:03:20.469952 7fd9e1012700  0 client.1522045 ms_handle_reset on
> XXX.XXX.XXX.XXX:6800/5261
> mds_max_purge_files = '1000' (unchangeable)
>
>
> -----mds------ --mds_server-- ---objecter--- -----mds_cache-----
> ---mds_log----
> rlat inos caps|hsr  hcs  hcr |writ read actv|recd recy stry purg|segs evts
> subm|
>   0   99k 1.0k|  0    0    0 |111    0  260 |  0    0   68k 110 | 39   29k
> 111
>   0   99k 1.0k|  0    0    0 |198    0  260 |  0    0   68k 198 | 39   29k
> 198
>   0   52k 1.0k|  0    0    0 |109    0  264 |  0    0   68k 102 | 39   23k
> 106
>   0   52k 1.0k|  0    0    0 |130    0  265 |  0    0   68k 125 | 39   23k
> 125
>   0   52k 1.0k|  0    1    0 |127    0  265 |  0    0   67k 127 | 39   23k
> 127
>   0   52k 1.0k|  0    0    0 | 84    0  264 |  0    0   67k  84 | 39   24k
> 84
>   0   52k 1.0k|  0    0    0 | 80    0  263 |  0    0   67k  80 | 39   24k
> 80
>   0   52k 1.0k|  0    0    0 | 89    0  260 |  0    0   67k  87 | 32   24k
> 89
>   0   52k 1.0k|  0    0    0 |134    0  259 |  0    0   67k 134 | 32   24k
> 134
>   0   52k 1.0k|  0    0    0 |155    0  259 |  0    0   67k 152 | 33   24k
> 154
>   0   52k 1.0k|  0    0    0 | 99    0  257 |  0    0   67k  99 | 33   24k
> 99
>   0   52k 1.0k|  0    0    0 | 84    0  257 |  0    0   67k  84 | 33   24k
> 84
>   0   52k 1.0k|  0    0    0 |117    0  257 |  0    0   67k 115 | 33   24k
> 115
>   0   52k 1.0k|  0    0    0 |122    0  257 |  0    0   66k 122 | 33   24k
> 122
>   0   52k 1.0k|  0    0    0 | 73    0  257 |  0    0   66k  73 | 33   24k
> 73
>   0   52k 1.0k|  0    0    0 |123    0  257 |  0    0   66k 123 | 33   25k
> 123
>   0   52k 1.0k|  0    0    0 | 87    0  257 |  0    0   66k  87 | 33   25k
> 87
>   0   52k 1.0k|  0    0    0 | 85    0  257 |  0    0   66k  83 | 33   25k
> 83
>   0   52k 1.0k|  0    0    0 | 55    0  257 |  0    0   66k  55 | 33   25k
> 55
>   0   52k 1.0k|  0    0    0 | 34    0  257 |  0    0   66k  34 | 33   25k
> 34
>   0   52k 1.0k|  0    0    0 | 58    0  257 |  0    0   66k  58 | 33   25k
> 58
>   0   52k 1.0k|  0    0    0 | 35    0  257 |  0    0   66k  35 | 33   25k
> 35
>   0   52k 1.0k|  0    0    0 | 65    0  259 |  0    0   66k  63 | 31   22k
> 64
>   0   52k 1.0k|  0    0    0 | 52    0  258 |  0    0   66k  52 | 31   23k
> 52
>
> Seems like purge rate is virtually not sensitive to mds_max_purge_files.
> BTW, the rm completed well before the stry approached the ground state.
>
> -Mykola
>
>
>
>
>
>
>
> On 4 October 2016 at 09:16, John Spray <jspray@xxxxxxxxxx> wrote:
>
> (Re-adding list)
>
> The 7.5k stray dentries while idle is probably indicating that clients
> are holding onto references to them (unless you unmount the clients
> and they don't purge, in which case you may well have found a bug).
> The other way you can end up with lots of dentries sitting in stray
> dirs is if you had lots of hard links and unlinked the original
> location but left the hard link in place.
>
> The rate at which your files are purging seems to roughly correspond
> to mds_max_purge_files, so I'd definitely try changing that to get
> things purging faster.
>
> John
>
>
> On Mon, Oct 3, 2016 at 3:21 PM, Mykola Dvornik <mykola.dvornik@xxxxxxxxx>
> wrote:
>> Hi John,
>>
>> This is how the daemonperf looks like :
>>
>> background
>>
>> -----mds------ --mds_server-- ---objecter--- -----mds_cache-----
>> ---mds_log----
>> rlat inos caps|hsr  hcs  hcr |writ read actv|recd recy stry purg|segs evts
>> subm|
>>   0   99k 177k|  0    0    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    0    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    0    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    5    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 1
>>   0   99k 177k|  0    0    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    0    0 |  2    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    2    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    2    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    1    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    2    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    0    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    1    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>   0   99k 177k|  0    6    0 |  0    0    0 |  0    0  7.5k   0 | 31   22k
>> 0
>>
>> with 4 rm instances
>>
>> -----mds------ --mds_server-- ---objecter--- -----mds_cache-----
>> ---mds_log----
>> rlat inos caps|hsr  hcs  hcr |writ read actv|recd recy stry purg|segs evts
>> subm|
>>   0  172k 174k|  0    5  3.1k| 85    0   34 |  0    0   79k  83 | 45   31k
>> 1.6k
>>   0  174k 174k|  0    0  3.0k| 76    0   35 |  0    0   80k  72 | 48   32k
>> 1.6k
>>   0  175k 174k|  0    0  2.7k| 81    0   37 |  0    0   81k  69 | 42   28k
>> 1.4k
>>   3  175k 174k|  0    2  468 | 41    0   17 |  0    0   82k  35 | 42   28k
>> 276
>>   0  177k 174k|  0    2  2.2k|134    0   41 |  0    0   83k 118 | 44   29k
>> 1.2k
>>   0  178k 174k|  0    1  2.7k|123    0   33 |  0    0   84k 121 | 46   31k
>> 1.5k
>>   0  179k 162k|  0    2  2.6k|133    0   32 |  0    0   85k 131 | 48   32k
>> 1.4k
>>   0  181k 162k|  0    0  2.3k|113    0   36 |  0    0   86k 102 | 40   27k
>> 1.2k
>>   0  182k 162k|  0    1  2.7k| 83    0   36 |  0    0   87k  81 | 42   28k
>> 1.4k
>>   0  183k 162k|  0    6  2.6k| 22    0   35 |  0    0   89k  22 | 43   30k
>> 1.3k
>>   0  184k 162k|  0    1  2.5k|  9    0   35 |  0    0   90k   7 | 45   31k
>> 1.2k
>>   0  186k 155k|  0    3  2.5k|  2    0   36 |  0    0   91k   0 | 47   32k
>> 1.2k
>>   0  187k 155k|  0    3  1.9k| 18    0   49 |  0    0   92k   0 | 48   32k
>> 970
>>   0  188k 155k|  0    2  2.5k| 46    0   30 |  0    0   93k  32 | 48   33k
>> 1.3k
>>   0  189k 155k|  0    0  2.4k| 55    0   36 |  0    0   95k  50 | 50   34k
>> 1.2k
>>   0  190k 155k|  0    0  2.7k|  2    0   36 |  0    0   96k   0 | 52   36k
>> 1.3k
>>   0  192k 150k|  0    1  3.0k| 30    0   37 |  0    0   97k  28 | 54   37k
>> 1.5k
>>   0  183k 150k|  0    0  2.7k| 58    0   40 |  0    0   99k  50 | 56   39k
>> 1.4k
>>   0  184k 150k|  0    0  3.2k| 12    0   41 |  0    0  100k  10 | 59   40k
>> 1.6k
>>   0  185k 150k|  0    0  2.1k|  3    0   41 |  0    0  102k   0 | 60   41k
>> 1.0k
>>   0  186k 150k|  0    5  1.6k| 12    0   41 |  0    0  102k  10 | 62   42k
>> 837
>>   0  186k 148k|  0    0  1.0k| 62    0   32 |  0    0  103k  57 | 62   43k
>> 575
>>   0  170k 148k|  0    0  858 | 31    0   25 |  0    0  103k  27 | 40   27k
>> 458
>>   5  165k 148k|  0    2  865 | 77    2   28 |  0    0  104k  45 | 41   28k
>> 495
>>
>> with all the rm instances killed
>>
>> -----mds------ --mds_server-- ---objecter--- -----mds_cache-----
>> ---mds_log----
>> rlat inos caps|hsr  hcs  hcr |writ read actv|recd recy stry purg|segs evts
>> subm|
>>   0  194k 147k|  0    0    0 | 64    0   32 |  0    0  117k  63 | 31   22k
>> 63
>>   0  194k 147k|  0    0    0 | 58    0   32 |  0    0  117k  58 | 31   22k
>> 58
>>   0  194k 147k|  0    0    0 | 49    0   32 |  0    0  117k  49 | 31   22k
>> 50
>>   0  194k 147k|  0    5    0 | 65    0   32 |  0    0  117k  65 | 31   22k
>> 65
>>   0  194k 147k|  0    0    0 | 42    0   32 |  0    0  117k  40 | 31   22k
>> 40
>>   0  194k 147k|  0    0    0 |  7    0   32 |  0    0  117k   7 | 31   22k
>> 7
>>   0  194k 147k|  0    2    0 | 23    0   32 |  0    0  117k  23 | 31   22k
>> 23
>>   0  194k 147k|  0    3    0 | 61    0   32 |  0    0  116k  61 | 31   23k
>> 62
>>   0  194k 147k|  0    0    0 | 59    0   32 |  0    0  116k  59 | 31   23k
>> 59
>>   0  194k 147k|  0    2    0 |107    0   32 |  0    0  116k 103 | 31   22k
>> 103
>>   0  194k 147k|  0    1    0 |126    0   32 |  0    0  116k 125 | 31   22k
>> 125
>>   0  194k 147k|  0    6    0 | 74    0   32 |  0    0  116k  74 | 31   22k
>> 74
>>   0  194k 147k|  0    1    0 | 37    0   32 |  0    0  116k  37 | 31   23k
>> 37
>>   0  194k 147k|  0    2    0 | 96    0   32 |  0    0  116k  96 | 31   23k
>> 96
>>   0  194k 147k|  0    2    0 |111    0   33 |  0    0  116k 110 | 31   23k
>> 110
>>   0  194k 147k|  0    3    0 |105    0   33 |  0    0  116k 105 | 31   23k
>> 105
>>   0  194k 147k|  0    1    0 | 79    0   33 |  0    0  116k  79 | 31   23k
>> 79
>>   0  194k 147k|  0    0    0 | 67    0   33 |  0    0  116k  67 | 31   23k
>> 68
>>   0  194k 147k|  0    0    0 | 75    0   33 |  0    0  116k  75 | 31   23k
>> 75
>>   0  194k 147k|  0    1    0 | 54    0   35 |  0    0  116k  51 | 31   23k
>> 51
>>   0  194k 147k|  0    0    0 | 40    0   35 |  0    0  115k  40 | 31   23k
>> 40
>>   0  194k 147k|  0    0    0 | 32    0   35 |  0    0  115k  32 | 31   23k
>> 32
>>   0  194k 147k|  0    5    0 | 43    0   35 |  0    0  115k  43 | 31   23k
>> 43
>>   0  194k 147k|  0    0    0 |  7    0   35 |  0    0  115k   7 | 31   23k
>> 7
>>
>> So I guess the purge ops are extremely slow.
>>
>> The first question is it OK to have 7.5K objects in stry when cluster is
>> idle for a while?
>>
>> The second question is who to blame for the slow purges, i.e. MDS or OSDs?
>>
>> Regards,
>>
>> -Mykola
>>
>>
>> On 2 October 2016 at 23:48, Mykola Dvornik <mykola.dvornik@xxxxxxxxx>
>> wrote:
>>>
>>> Hi Johan,
>>>
>>> Many thanks for your reply. I will try to play with the mds tunables and
>>> report back to your ASAP.
>>>
>>> So far I see that mds log contains a lot of errors of the following kind:
>>>
>>> 2016-10-02 11:58:03.002769 7f8372d54700  0 mds.0.cache.dir(100056ddecd)
>>> _fetched  badness: got (but i already had) [inode 10005729a77 [2,head]
>>> ~mds0/stray1/10005729a77 auth v67464942 s=196728 nl=0 n(v0 b196728 1=1+0)
>>> (iversion lock) 0x7f84acae82a0] mode 33204 mtime 2016-08-07
>>> 23:06:29.776298
>>>
>>> 2016-10-02 11:58:03.002789 7f8372d54700 -1 log_channel(cluster) log [ERR]
>>> : loaded dup inode 10005729a77 [2,head] v68621 at
>>>
>>> /users/mykola/mms/NCSHNO/final/120nm-uniform-h8200/j002654.out/m_xrange192-320_yrange192-320_016232.dump,
>>> but inode 10005729a77.head v67464942 already exists at
>>> ~mds0/stray1/10005729a77
>>>
>>> Those folders within mds.0.cache.dir that got badness report a size of
>>> 16EB on the clients. rm on them fails with 'Directory not empty'.
>>>
>>> As for the "Client failing to respond to cache pressure", I have 2 kernel
>>> clients on 4.4.21, 1 on 4.7.5 and 16 fuse clients always running the most
>>> recent release version of ceph-fuse. The funny thing is that every single
>>> client misbehaves from time to time. I am aware of quite discussion about
>>> this issue on the ML, but cannot really follow how to debug it.
>>>
>>> Regards,
>>>
>>> -Mykola
>>>
>>> On 2 October 2016 at 22:27, John Spray <jspray@xxxxxxxxxx> wrote:
>>>>
>>>> On Sun, Oct 2, 2016 at 11:09 AM, Mykola Dvornik
>>>> <mykola.dvornik@xxxxxxxxx> wrote:
>>>> > After upgrading to 10.2.3 we frequently see messages like
>>>>
>>>> From which version did you upgrade?
>>>>
>>>> > 'rm: cannot remove '...': No space left on device
>>>> >
>>>> > The folders we are trying to delete contain approx. 50K files 193 KB
>>>> > each.
>>>>
>>>> My guess would be that you are hitting the new
>>>> mds_bal_fragment_size_max check.  This limits the number of entries
>>>> that the MDS will create in a single directory fragment, to avoid
>>>> overwhelming the OSD with oversized objects.  It is 100000 by default.
>>>> This limit also applies to "stray" directories where unlinked files
>>>> are put while they wait to be purged, so you could get into this state
>>>> while doing lots of deletions.  There are ten stray directories that
>>>> get a roughly even share of files, so if you have more than about one
>>>> million files waiting to be purged, you could see this condition.
>>>>
>>>> The "Client failing to respond to cache pressure" messages may play a
>>>> part here -- if you have misbehaving clients then they may cause the
>>>> MDS to delay purging stray files, leading to a backlog.  If your
>>>> clients are by any chance older kernel clients, you should upgrade
>>>> them.  You can also unmount/remount them to clear this state, although
>>>> it will reoccur until the clients are updated (or until the bug is
>>>> fixed, if you're running latest clients already).
>>>>
>>>> The high level counters for strays are part of the default output of
>>>> "ceph daemonperf mds.<id>" when run on the MDS server (the "stry" and
>>>> "purg" columns).  You can look at these to watch how fast the MDS is
>>>> clearing out strays.  If your backlog is just because it's not doing
>>>> it fast enough, then you can look at tuning mds_max_purge_files and
>>>> mds_max_purge_ops to adjust the throttles on purging.  Those settings
>>>> can be adjusted without restarting the MDS using the "injectargs"
>>>> command
>>>>
>>>> (http://docs.ceph.com/docs/master/rados/operations/control/#mds-subsystem)
>>>>
>>>> Let us know how you get on.
>>>>
>>>> John
>>>>
>>>>
>>>> > The cluster state and storage available are both OK:
>>>> >
>>>> >     cluster 98d72518-6619-4b5c-b148-9a781ef13bcb
>>>> >      health HEALTH_WARN
>>>> >             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
>>>> > pressure
>>>> >             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
>>>> > pressure
>>>> >             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
>>>> > pressure
>>>> >             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
>>>> > pressure
>>>> >             mds0: Client XXX.XXX.XXX.XXX failing to respond to cache
>>>> > pressure
>>>> >      monmap e1: 1 mons at {000-s-ragnarok=XXX.XXX.XXX.XXX:6789/0}
>>>> >             election epoch 11, quorum 0 000-s-ragnarok
>>>> >       fsmap e62643: 1/1/1 up {0=000-s-ragnarok=up:active}
>>>> >      osdmap e20203: 16 osds: 16 up, 16 in
>>>> >             flags sortbitwise
>>>> >       pgmap v15284654: 1088 pgs, 2 pools, 11263 GB data, 40801
>>>> > kobjects
>>>> >             23048 GB used, 6745 GB / 29793 GB avail
>>>> >                 1085 active+clean
>>>> >                    2 active+clean+scrubbing
>>>> >                    1 active+clean+scrubbing+deep
>>>> >
>>>> >
>>>> > Has anybody experienced this issue so far?
>>>> >
>>>> > Regards,
>>>> > --
>>>> >  Mykola
>>>> >
>>>> > _______________________________________________
>>>> > ceph-users mailing list
>>>> > ceph-users@xxxxxxxxxxxxxx
>>>> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>>> >
>>>
>>>
>>>
>>>
>>> --
>>>  Mykola
>>
>>
>>
>>
>> --
>>  Mykola
>
>
>
>
> --
>
>  Mykola
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux