Thanks, Luis.
I've just set max_bytes and max_objects:
target_max_objects: 1000000 (1M)
target_max_bytes: 1099511627776 (1TB)
but nothing appears to be happening. Is there a way to force flushing?
Thanks,
Shaw
On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito <periquito@xxxxxxxxx> wrote:
Not looking at anything else, you didn't set the max_bytes or
max_objects for it to start flushing...
On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong <shaw@xxxxxxxx> wrote:
> Dear all,
>
> Thanks a lot for the very insightful comments/suggestions!
>
> There are 3 OSD servers in our pilot Ceph cluster, each with 2x 1TB SSDs
> (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the bluestore
> backend, with the first NVMe as the WAL and DB devices for OSDs on the HDDs.
> And we try to create a cache tier out of the second NVMes.
>
> Here are the outputs of the commands suggested by David:
>
> 1) # ceph df
> GLOBAL:
> SIZE AVAIL RAW USED %RAW USED
> 265T 262T 2847G 1.05
> POOLS:
> NAME ID USED %USED MAX AVAIL OBJECTS
> cephfs_data 1 0 0 248T 0
> cephfs_metadata 2 8515k 0 248T 24
> cephfs_cache 3 1381G 100.00 0 355385
>
> 2) # ceph osd df
> 0 hdd 7.27829 1.00000 7452G 2076M 7450G 0.03 0.03 174
> 1 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 169
> 2 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 173
> 3 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 159
> 4 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 173
> 5 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 162
> 6 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 149
> 7 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 179
> 8 hdd 7.27829 1.00000 7452G 2076M 7450G 0.03 0.03 163
> 9 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 194
> 10 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 185
> 11 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 168
> 36 nvme 1.09149 1.00000 1117G 855G 262G 76.53 73.01 79
> 12 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 180
> 13 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 168
> 14 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 178
> 15 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 170
> 16 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 149
> 17 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 203
> 18 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 173
> 19 hdd 7.27829 1.00000 7452G 2076M 7450G 0.03 0.03 158
> 20 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 154
> 21 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 160
> 22 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 167
> 23 hdd 7.27829 1.00000 7452G 2076M 7450G 0.03 0.03 188
> 37 nvme 1.09149 1.00000 1117G 1061G 57214M 95.00 90.63 98
> 24 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 187
> 25 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 200
> 26 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 147
> 27 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 171
> 28 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 162
> 29 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 152
> 30 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 174
> 31 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 176
> 32 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 182
> 33 hdd 7.27829 1.00000 7452G 2072M 7450G 0.03 0.03 155
> 34 hdd 7.27829 1.00000 7452G 2076M 7450G 0.03 0.03 166
> 35 hdd 7.27829 1.00000 7452G 2076M 7450G 0.03 0.03 176
> 38 nvme 1.09149 1.00000 1117G 857G 260G 76.71 73.18 79
> TOTAL 265T 2847G 262T 1.05
> MIN/MAX VAR: 0.03/90.63 STDDEV: 22.81
>
> 3) # ceph osd tree
> -1 265.29291 root default
> -3 88.43097 host pulpo-osd01
> 0 hdd 7.27829 osd.0 up 1.00000 1.00000
> 1 hdd 7.27829 osd.1 up 1.00000 1.00000
> 2 hdd 7.27829 osd.2 up 1.00000 1.00000
> 3 hdd 7.27829 osd.3 up 1.00000 1.00000
> 4 hdd 7.27829 osd.4 up 1.00000 1.00000
> 5 hdd 7.27829 osd.5 up 1.00000 1.00000
> 6 hdd 7.27829 osd.6 up 1.00000 1.00000
> 7 hdd 7.27829 osd.7 up 1.00000 1.00000
> 8 hdd 7.27829 osd.8 up 1.00000 1.00000
> 9 hdd 7.27829 osd.9 up 1.00000 1.00000
> 10 hdd 7.27829 osd.10 up 1.00000 1.00000
> 11 hdd 7.27829 osd.11 up 1.00000 1.00000
> 36 nvme 1.09149 osd.36 up 1.00000 1.00000
> -5 88.43097 host pulpo-osd02
> 12 hdd 7.27829 osd.12 up 1.00000 1.00000
> 13 hdd 7.27829 osd.13 up 1.00000 1.00000
> 14 hdd 7.27829 osd.14 up 1.00000 1.00000
> 15 hdd 7.27829 osd.15 up 1.00000 1.00000
> 16 hdd 7.27829 osd.16 up 1.00000 1.00000
> 17 hdd 7.27829 osd.17 up 1.00000 1.00000
> 18 hdd 7.27829 osd.18 up 1.00000 1.00000
> 19 hdd 7.27829 osd.19 up 1.00000 1.00000
> 20 hdd 7.27829 osd.20 up 1.00000 1.00000
> 21 hdd 7.27829 osd.21 up 1.00000 1.00000
> 22 hdd 7.27829 osd.22 up 1.00000 1.00000
> 23 hdd 7.27829 osd.23 up 1.00000 1.00000
> 37 nvme 1.09149 osd.37 up 1.00000 1.00000
> 36 nvme 1.09149 osd.36 up 1.00000 1.00000
> -5 88.43097 host pulpo-osd02
> 12 hdd 7.27829 osd.12 up 1.00000 1.00000
> 13 hdd 7.27829 osd.13 up 1.00000 1.00000
> 14 hdd 7.27829 osd.14 up 1.00000 1.00000
> 15 hdd 7.27829 osd.15 up 1.00000 1.00000
> 16 hdd 7.27829 osd.16 up 1.00000 1.00000
> 17 hdd 7.27829 osd.17 up 1.00000 1.00000
> 18 hdd 7.27829 osd.18 up 1.00000 1.00000
> 19 hdd 7.27829 osd.19 up 1.00000 1.00000
> 20 hdd 7.27829 osd.20 up 1.00000 1.00000
> 21 hdd 7.27829 osd.21 up 1.00000 1.00000
> 22 hdd 7.27829 osd.22 up 1.00000 1.00000
> 23 hdd 7.27829 osd.23 up 1.00000 1.00000
> 37 nvme 1.09149 osd.37 up 1.00000 1.00000
> -7 88.43097 host pulpo-osd03
> 24 hdd 7.27829 osd.24 up 1.00000 1.00000
> 25 hdd 7.27829 osd.25 up 1.00000 1.00000
> 26 hdd 7.27829 osd.26 up 1.00000 1.00000
> 27 hdd 7.27829 osd.27 up 1.00000 1.00000
> 28 hdd 7.27829 osd.28 up 1.00000 1.00000
> 29 hdd 7.27829 osd.29 up 1.00000 1.00000
> 30 hdd 7.27829 osd.30 up 1.00000 1.00000
> 31 hdd 7.27829 osd.31 up 1.00000 1.00000
> 32 hdd 7.27829 osd.32 up 1.00000 1.00000
> 33 hdd 7.27829 osd.33 up 1.00000 1.00000
> 34 hdd 7.27829 osd.34 up 1.00000 1.00000
> 35 hdd 7.27829 osd.35 up 1.00000 1.00000
> 38 nvme 1.09149 osd.38 up 1.00000 1.00000
>
> 4) # ceph osd pool get cephfs_cache all
> min_size: 2
> crash_replay_interval: 0
> pg_num: 128
> pgp_num: 128
> crush_rule: pulpo_nvme
> hashpspool: true
> nodelete: false
> nopgchange: false
> nosizechange: false
> write_fadvise_dontneed: false
> noscrub: false
> nodeep-scrub: false
> hit_set_type: bloom
> hit_set_period: 14400
> hit_set_count: 12
> hit_set_fpp: 0.05
> use_gmt_hitset: 1
> auid: 0
> target_max_objects: 0
> target_max_bytes: 0
> cache_target_dirty_ratio: 0.4
> cache_target_dirty_high_ratio: 0.6
> cache_target_full_ratio: 0.8
> cache_min_flush_age: 0
> cache_min_evict_age: 0
> min_read_recency_for_promote: 0
> min_write_recency_for_promote: 0
> fast_read: 0
> hit_set_grade_decay_rate: 0
> crash_replay_interval: 0
>
> Do you see anything wrong? We had written some small files to the CephFS
> before we tried to write the big 1TB file. What is puzzling to me is that no
> data has been written back to the data pool.
>
> Best,
> Shaw
>
> On Fri, Oct 6, 2017 at 6:46 AM, David Turner <drakonstein@xxxxxxxxx> wrote:
>>
>>
>>
>> On Fri, Oct 6, 2017, 1:05 AM Christian Balzer <chibi@xxxxxxx> wrote:
>>>
>>>
>>> Hello,
>>>
>>> On Fri, 06 Oct 2017 03:30:41 +0000 David Turner wrote:
>>>
>>> > You're missing most all of the important bits. What the osds in your
>>> > cluster look like, your tree, and your cache pool settings.
>>> >
>>> > ceph df
>>> > ceph osd df
>>> > ceph osd tree
>>> > ceph osd pool get cephfs_cache all
>>> >
>>> Especially the last one.
>>>
>>> My money is on not having set target_max_objects and target_max_bytes to
>>> sensible values along with the ratios.
>>> In short, not having read the (albeit spotty) documentation.
>>>
>>> > You have your writeback cache on 3 nvme drives. It looks like you have
>>> > 1.6TB available between them for the cache. I don't know the behavior
>>> > of a
>>> > writeback cache tier on cephfs for large files, but I would guess that
>>> > it
>>> > can only hold full files and not flush partial files.
>>>
>>> I VERY much doubt that, if so it would be a massive flaw.
>>> One assumes that cache operations work on the RADOS object level, no
>>> matter what.
>>
>> I hope that it is on the rados level, but not a single object had been
>> flushed to the backing pool. So I hazarded a guess. Seeing his settings will
>> shed more light.
>>>
>>>
>>> > That would mean your
>>> > cache needs to have enough space for any file being written to the
>>> > cluster.
>>> > In this case a 1.3TB file with 3x replication would require 3.9TB (more
>>> > than double what you have available) of available space in your
>>> > writeback
>>> > cache.
>>> >
>>> > There are very few use cases that benefit from a cache tier. The docs
>>> > for
>>> > Luminous warn as much.
>>> You keep repeating that like a broken record.
>>>
>>> And while certainly not false I for one wouldn't be able to use (justify
>>> using) Ceph w/o cache tiers in our main use case.
>>>
>>>
>>> In this case I assume they were following and old cheat sheet or such,
>>> suggesting the previously required cache tier with EC pools.
>>
>>
>> http://docs.ceph.com/docs/luminous/rados/operations/ cache-tiering/
>>
>> I know I keep repeating it, especially recently as there have been a lot
>> of people asking about it. The Luminous docs added a large section about how
>> it is probably not what you want. Like me, it is not saying that there are
>> no use cases for it. There was no information provided about the use case
>> and I made some suggestions/guesses. I'm also guessing that they are
>> following a guide where a writeback cache was necessary for CephFS to use EC
>> prior to Luminous. I also usually add that people should test it out and
>> find what works best for them. I will always defer to your practical use of
>> cache tiers as well, especially when using rbds.
>>
>> I manage a cluster that I intend to continue running a writeback cache in
>> front of CephFS on the same drives as the EC pool. The use case receives a
>> good enough benefit from the cache tier that it isn't even required to use
>> flash media to see it. It is used for video editing and the files are
>> usually modified and read within the first 24 hours and then left in cold
>> storage until deleted. I have the cache timed to keep everything in it for
>> 24 hours and then evict it by using a minimum time to flush and evict at 24
>> hours and a target max bytes of 0. All files are in there for that time and
>> then it never has to decide what to keep as it doesn't keep anything longer
>> than that. Luckily read performance from cold storage is not a requirement
>> of this cluster as any read operation has to first read it from EC storage,
>> write it to replica storage, and then read it from replica storage... Yuck.
>>>
>>>
>>> Christian
>>>
>>> >What is your goal by implementing this cache? If the
>>> > answer is to utilize extra space on the nvmes, then just remove it and
>>> > say
>>> > thank you. The better use of nvmes in that case are as a part of the
>>> > bluestore stack and give your osds larger DB partitions. Keeping your
>>> > metadata pool on nvmes is still a good idea.
>>> >
>>> > On Thu, Oct 5, 2017, 7:45 PM Shawfeng Dong <shaw@xxxxxxxx> wrote:
>>> >
>>> > > Dear all,
>>> > >
>>> > > We just set up a Ceph cluster, running the latest stable release Ceph
>>> > > v12.2.0 (Luminous):
>>> > > # ceph --version
>>> > > ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24 d43bab910c)
>>> > > luminous
>>> > > (rc)
>>> > >
>>> > > The goal is to serve Ceph filesystem, for which we created 3 pools:
>>> > > # ceph osd lspools
>>> > > 1 cephfs_data,2 cephfs_metadata,3 cephfs_cache,
>>> > > where
>>> > > * cephfs_data is the data pool (36 OSDs on HDDs), which is
>>> > > erased-coded;
>>> > > * cephfs_metadata is the metadata pool
>>> > > * cephfs_cache is the cache tier (3 OSDs on NVMes) for cephfs_data.
>>> > > The
>>> > > cache-mode is writeback.
>>> > >
>>> > > Everything had worked fine, until today when we tried to copy a 1.3TB
>>> > > file
>>> > > to the CephFS. We got the "No space left on device" error!
>>> > >
>>> > > 'ceph -s' says some OSDs are full:
>>> > > # ceph -s
>>> > > cluster:
>>> > > id: e18516bf-39cb-4670-9f13-88ccb7d19769
>>> > > health: HEALTH_ERR
>>> > > full flag(s) set
>>> > > 1 full osd(s)
>>> > > 1 pools have many more objects per pg than average
>>> > >
>>> > > services:
>>> > > mon: 3 daemons, quorum pulpo-admin,pulpo-mon01,pulpo-mds01
>>> > > mgr: pulpo-mds01(active), standbys: pulpo-admin, pulpo-mon01
>>> > > mds: pulpos-1/1/1 up {0=pulpo-mds01=up:active}
>>> > > osd: 39 osds: 39 up, 39 in
>>> > > flags full
>>> > >
>>> > > data:
>>> > > pools: 3 pools, 2176 pgs
>>> > > objects: 347k objects, 1381 GB
>>> > > usage: 2847 GB used, 262 TB / 265 TB avail
>>> > > pgs: 2176 active+clean
>>> > >
>>> > > io:
>>> > > client: 19301 kB/s rd, 2935 op/s rd, 0 op/s wr
>>> > >
>>> > > And indeed the cache pool is full:
>>> > > # rados df
>>> > > POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY
>>> > > UNFOUND
>>> > > DEGRADED RD_OPS RD
>>> > > WR_OPS WR
>>> > > cephfs_cache 1381G 355385 0 710770 0
>>> > > 0
>>> > > 0 10004954 15
>>> > > 22G 1398063 1611G
>>> > > cephfs_data 0 0 0 0 0
>>> > > 0
>>> > > 0 0
>>> > > 0 0 0
>>> > > cephfs_metadata 8515k 24 0 72 0
>>> > > 0
>>> > > 0 3 3
>>> > > 072 3953 10541k
>>> > >
>>> > > total_objects 355409
>>> > > total_used 2847G
>>> > > total_avail 262T
>>> > > total_space 265T
>>> > >
>>> > > However, the data pool is completely empty! So it seems that data has
>>> > > only
>>> > > been written to the cache pool, but not written back to the data
>>> > > pool.
>>> > >
>>> > > I am really at a loss whether this is due to a setup error on my
>>> > > part, or
>>> > > a Luminous bug. Could anyone shed some light on this? Please let me
>>> > > know if
>>> > > you need any further info.
>>> > >
>>> > > Best,
>>> > > Shaw
>>> > > _______________________________________________
>>> > > ceph-users mailing list
>>> > > ceph-users@xxxxxxxxxxxxxx
>>> > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>>> > >
>>>
>>>
>>> --
>>> Christian Balzer Network/Systems Engineer
>>> chibi@xxxxxxx Rakuten Communications
>>> _______________________________________________
>>> ceph-users mailing list
>>> ceph-users@xxxxxxxxxxxxxx
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>>
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@xxxxxxxxxxxxxx
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>>
>
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@xxxxxxxxxxxxxx
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph. com
>
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
