Is scrubbing and deep scrubbing necessary for Snaptrim operation to happen ?
Karun Josy
On Fri, Jan 26, 2018 at 9:29 PM, Karun Josy <karunjosy1@xxxxxxxxx> wrote:
Thank you for your quick response!I used the command to fetch the snap_trimq from many pgs, however it seems they don't have any in queue ?For eg :====================$ echo $(( $(ceph pg 55.4a query | grep snap_trimq | cut -d[ -f2 | cut -d] -f1 | tr ',' '\n' | wc -l) - 1 ))0$ echo $(( $(ceph pg 55.5a query | grep snap_trimq | cut -d[ -f2 | cut -d] -f1 | tr ',' '\n' | wc -l) - 1 ))0$ echo $(( $(ceph pg 55.88 query | grep snap_trimq | cut -d[ -f2 | cut -d] -f1 | tr ',' '\n' | wc -l) - 1 ))0$ echo $(( $(ceph pg 55.55 query | grep snap_trimq | cut -d[ -f2 | cut -d] -f1 | tr ',' '\n' | wc -l) - 1 ))0$ echo $(( $(ceph pg 54.a query | grep snap_trimq | cut -d[ -f2 | cut -d] -f1 | tr ',' '\n' | wc -l) - 1 ))0$ echo $(( $(ceph pg 34.1d query | grep snap_trimq | cut -d[ -f2 | cut -d] -f1 | tr ',' '\n' | wc -l) - 1 ))0$ echo $(( $(ceph pg 1.3f query | grep snap_trimq | cut -d[ -f2 | cut -d] -f1 | tr ',' '\n' | wc -l) - 1 ))0=====================While going through the PG query, I find that these PGs have no value in purged_snaps section too.For eg :
ceph pg 55.80 query--------{"peer": "83(3)","pgid": "55.80s3","last_update": "43360'15121927","last_complete": "43345'15073146","log_tail": "43335'15064480","last_user_version": 15066124,"last_backfill": "MAX","last_backfill_bitwise": 1,"purged_snaps": [],"history": {"epoch_created": 5950,"epoch_pool_created": 5950,"last_epoch_started": 43339,"last_interval_started": 43338,"last_epoch_clean": 43340,"last_interval_clean": 43338,"last_epoch_split": 0,"last_epoch_marked_full": 42032,"same_up_since": 43338,"same_interval_since": 43338,"same_primary_since": 43276,"last_scrub": "35299'13072533","last_scrub_stamp": "2018-01-18 14:01:19.557972","last_deep_scrub": "31372'12176860","last_deep_scrub_stamp": "2018-01-15 12:21:17.025305","last_clean_scrub_stamp": "2018-01-18 14:01:19.557972"},Not sure if it is related.The cluster is not open to any new clients. However we see a steady growth of space usage every day.And worst case scenario, it might grow faster than we can add more space, which will be dangerous.Any help is really appreciated.Karun JosyOn Fri, Jan 26, 2018 at 8:23 PM, David Turner <drakonstein@xxxxxxxxx> wrote:"snap_trimq": "[]",
That is exactly what you're looking for to see how many objects a PG still had that need to be cleaned up. I think something like this should give you the number of objects in the snap_trimq for a PG.
echo $(( $(ceph pg $pg query | grep snap_trimq | cut -d[ -f2 | cut -d] -f1 | tr ',' '\n' | wc -l) - 1 ))
Note, I'm not at a computer and topping this from my phone so it's not pretty and I know of a few ways to do that better, but that should work all the same.
For your needs a visual inspection of several PGs should be sufficient to see if there is anything in the snap_trimq to begin with.
On Fri, Jan 26, 2018, 9:18 AM Karun Josy <karunjosy1@xxxxxxxxx> wrote:Hi David,Thank you for the response. To be honest, I am afraid it is going to be a issue in our cluster.It seems snaptrim has not been going on for sometime now , maybe because we were expanding the cluster adding nodes for the past few weeks.I would be really glad if you can guide me how to overcome this.
Cluster has about 30TB data and 11 million objects. With about 100 disks spread across 16 nodes. Version is 12.2.2Searching through the mailing lists I can see many cases where the performance were affected while snaptrimming.Can you help me figure out these :- How to find snaptrim queue of a PG.- Can snaptrim be started just on 1 PG- How can I make sure cluster IO performance is not affected ?I read about osd_snap_trim_sleep , how can it be changed ?Is this the command : ceph tell osd.* injectargs '--osd_snap_trim_sleep 0.005'If yes what is the recommended value that we can use ?Also, what all parameters should we be concerned about? I would really appreciate any suggestions.Below is a brief extract of a PG queried----------------------------ceph pg 55.77 query{"state": "active+clean","snap_trimq": "[]",-------"pgid": "55.77s7","last_update": "43353'17222404","last_complete": "42773'16814984","log_tail": "42763'16812644","last_user_version": 16814144,"last_backfill": "MAX","last_backfill_bitwise": 1,"purged_snaps": [],"history": {"epoch_created": 5950,---------Karun JosyOn Fri, Jan 26, 2018 at 6:36 PM, David Turner <drakonstein@xxxxxxxxx> wrote:You may find the information in this ML thread useful. https://www.spinics.net/lists/ceph-users/msg41279.h tml It talks about a couple ways to track your snaptrim queue.On Fri, Jan 26, 2018 at 2:09 AM Karun Josy <karunjosy1@xxxxxxxxx> wrote:______________________________Hi,We have set no scrub , no deep scrub flag on a ceph cluster.When we are deleting snapshots we are not seeing any change in usage space.I understand that Ceph OSDs delete data asynchronously, so deleting a snapshot doesn’t free up the disk space immediately. But we are not seeing any change for sometime.What can be possible reason ? Any suggestions would be really helpful as the cluster size seems to be growing each day even though snapshots are deleted.Karun_________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
