Re: problem while removing images

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello,

On Mon, 26 Mar 2018 18:20:22 -0300 Thiago Gonzaga wrote:

> Hey Guys,
> 
> Is it normal to get all process stuck while a image is deleting?
> 
If you'd tell us all the details of your cluster (SW and HW) we might be
more helpful and point to specific issues. 

So to generically answer your question, yes.

Deleting images, snapshots and a number of what would be considered normal
standard tasks for Ceph can create insane I/O storms, completely
saturating your storage.

Which you would see if you monitored your storage nodes and can quickly
verify with atop or iostat. 

Some of these things can be mitigated by throttling Ceph down where such
knobs exist, others have gotten better over time with newer versions, but
penultimately your storage system will be the bottleneck.

Christian

> I'm removing 21g layering image and it takes ages, while the image is
> removing I have lots of problems in the log like this
> 
> I ran rbd create rbd/test -s 21G --image-feature layering
> deleted the image while deleted that tried to create another one and format
> it
> root@ceph-mon-3:/home/tgonzaga# rbd create rbd/test2 -s 10G --image-feature
> layering
> root@ceph-mon-3:/home/tgonzaga# rbd map test2
> /dev/rbd0
> 
> it's stuck until the image gets completely deleted
> 
> 2018-03-26 21:15:30.431835 [WRN]  Health check update: 551 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:15:00.373288 [WRN]  Health check update: 555 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:14:54.365829 [WRN]  Health check update: 533 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:14:44.351739 [WRN]  Health check update: 531 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:14:38.342610 [WRN]  Health check update: 541 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:14:10.734288 [WRN]  Health check update: 555 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:14:05.733855 [WRN]  Health check update: 557 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:14:00.288994 [WRN]  Health check update: 538 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:13:50.275008 [WRN]  Health check update: 534 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:13:44.267499 [WRN]  Health check update: 536 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:13:38.258227 [WRN]  Health check update: 171 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:13:32.250539 [WRN]  Health check update: 170 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 2018-03-26 21:13:26.241810 [WRN]  Health check update: 163 slow requests
> are blocked > 32 sec (REQUEST_SLOW)
> 
> And I can't perform anything on other images.
> 
> Any ideas?
> 
> Regards
> 
> 
> *Thiago Gonzaga*
> SaaSOps Software Architect
> o. 1 (512) 2018-287 x2119
> Skype: thiago.gonzaga20
> [image: Aurea]
> <http://www.aurea.com/?utm_source=email-signature&utm_medium=email>


-- 
Christian Balzer        Network/Systems Engineer                
chibi@xxxxxxx   	Rakuten Communications
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



[Index of Archives]     [Information on CEPH]     [Linux Filesystem Development]     [Ceph Development]     [Ceph Large]     [Ceph Dev]     [Linux USB Development]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [xfs]


  Powered by Linux