Hi Konstantin, Thank you for the reply. I tried setting ulimit to 32768 when I saw 25726 number in lsof output and then after 2 disks deletion again it got an error and checked lsof and which is above 35000. I'm not sure how to handle it. I rebooted the monitor node, but the open files kept growing. root@ceph-mon01 ~# lsof | wc -l 49296 root@ceph-mon01 ~# Thanks, Pardh On Thu, Apr 18, 2024 at 11:36 PM Konstantin Shalygin <k0ste@xxxxxxxx> wrote: > Hi, > > Your shell seems reached the default file discriptors limit (1024 mostly) > and your cluster maybe more than 1000 OSD > > Try to set command `ulimit -n 10240` before rbd rm task > > > k > Sent from my iPhone > > > On 18 Apr 2024, at 23:50, Pardhiv Karri <meher4india@xxxxxxxxx> wrote: > > > > Hi, > > > > Trying to delete images in a Ceph pool is causing errors in one of > > the clusters. I rebooted all the monitor nodes sequentially to see if the > > error went away, but it still persists. What is the best way to fix this? > > The Ceph cluster is in an OK state, with no rebalancing or scrubbing > > happening (I did set the noscrub and deep-noscrub flags) and also no load > > on the cluster, very few IO. > > > > root@ceph-mon01 ~# rbd rm > 000dca3d-4f2b-4033-b8f5-95458e0c3444_disk_delete > > -p compute > > Removing image: 31% complete...2024-04-18 20:42:52.525135 7f6de0c79700 -1 > > NetHandler create_socket couldn't create socket (24) Too many open files > > Removing image: 32% complete...2024-04-18 20:42:52.539882 7f6de9c7b700 -1 > > NetHandler create_socket couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.541508 7f6de947a700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.546613 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.558133 7f6de9c7b700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.573819 7f6de947a700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.589733 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > Removing image: 33% complete...2024-04-18 20:42:52.643489 7f6de9c7b700 -1 > > NetHandler create_socket couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.727262 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.737135 7f6de9c7b700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.743292 7f6de947a700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.746167 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.757404 7f6de9c7b700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > Removing image: 34% complete...2024-04-18 20:42:52.773182 7f6de947a700 -1 > > NetHandler create_socket couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.773222 7f6de947a700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.789847 7f6de0c79700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > 2024-04-18 20:42:52.844201 7f6de9c7b700 -1 NetHandler create_socket > > couldn't create socket (24) Too many open files > > > > ^C > > root@ceph-mon01 ~# > > > > > > Thanks, > > Pardh > > _______________________________________________ > > ceph-users mailing list -- ceph-users@xxxxxxx > > To unsubscribe send an email to ceph-users-leave@xxxxxxx > > -- *Pardhiv Karri* "Rise and Rise again until LAMBS become LIONS" _______________________________________________ ceph-users mailing list -- ceph-users@xxxxxxx To unsubscribe send an email to ceph-users-leave@xxxxxxx
 |