Shell Script For Flush and Evicting Objects from Cache Tier

[Romit Misra <romit.misra@xxxxxxxxxxxx>]

The Script Reads a File, (Object Listing from the cache pool via the rados ls -p), and starts flushing, and evicting to the Base Tier.

It respects any blocked requests on the Cluster, and toggles recovery throttles based on the same (This is the case if the Flush + Evict) is going in at the Same Time with Recovery on Base / Cache Tier

Thanks

Romit

#!/bin/bash
#set -x
count=0 
req_blocked=0

#Threshold for Counter, Initially Set at 10, can be changed according to Traffic Thresholds of the cluster and many particiapting Dynamics
thr=10

#Total Count of Objects Flushed and Evicted

total=0
check_var=0
start=`date +%s`
while IFS= read -r line; do
    
    #Read the Object from the File 
    echo "Object read from file: $line"
    #First Flush the Object to the Base Tier, Flush Does not Neccessarlily Evict
    rados -p .in-chennai-1.rgw.buckets.cache cache-flush "$line"
    #Now Evict The Object, Should free space on cache tier, and Zero Out
    rados -p .in-chennai-1.rgw.buckets.cache cache-evict "$line"
    
    count=$(expr $count + 1)
    total=$(expr $total + 1)
    echo "Object Flushed Till Now --> " + $total

    #Reset Counters for this batch, spit out the statistics and Sleep for some time and Start Again
    #Should We sleep only on Blocked Requests and not on every batch is a open ended question
     
    if  [ $count -eq 10 ]; then
        
        #Update the End Time, and calculate the Runtime
        end=`date +%s`
        runtime=$((end-start))
        
        #Dump the Runtime for the Flush + Evict Activity
        echo "Runtime for Flush ---> " + $runtime
        
        #Sleep, Commented for now
        #sleep 0.3
        
        #Check for Blocked Requests and Throttle (Just Sleep), Lets Not Agitate the GOD of PG and OSD
        #If There are blocked Requests, sleep for 5 seconds, and again check
        #The 5 could be replaced with some random number, this is just the basic setup in place
        check_var=$(sudo ceph -s | grep blocked | wc -l)
        while [ $check_var -eq 1 ]
        do 
           #Stopping Recovery, so that any recovery event causing Blocked Requests, can be throttled
           if [ $req_blocked -eq 0 ]; then
               req_blocked=1
               sudo ceph osd set norebalance
               sudo ceph osd set nobackfill
           fi
           echo "Blocked Requests, Sleeping for 5, before Next Evict + Flush of Object"
           sleep 5
           check_var=$(sudo ceph -s | grep blocked | wc -l)
        done
        
        #If the req_blocked flag is set to 1, then the recovery flags have been set, to stop the recovery.      Reset them 
        if [ $req_blocked -eq 1 ]; then
            req_blocked=0
            sudo ceph osd unset norebalance
            sudo ceph osd unset nobackfill
        fi
           
        #Reset The Start time and count to zero
	start=`date +%s`
        count=0
    fi
done < "$1"

_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com