Just before updrading to 3.3final, we had an rsync collision on our gluster filesystem which left us with undeletable dirs. The transport is IPoIB over 4 bricks as shown below. $ gluster volume info Volume Name: gli Type: Distribute Volume ID: 76cc5e88-0ac4-42ac-a4a3-31bf2ba611d4 Status: Started Number of Bricks: 4 Transport-type: tcp,rdma Bricks: Brick1: pbs1ib:/bducgl Brick2: pbs2ib:/bducgl Brick3: pbs3ib:/bducgl Brick4: pbs4ib:/bducgl Options Reconfigured: performance.io-cache: on performance.quick-read: on performance.io-thread-count: 64 The last 100 line of the gluster log is here: <http://pastie.org/4036243> The problem started when I was running an rsync with --delete while moving a large /home to the gluster dir to prune the target of older cruft in a /home move.. A user directed a job to start reading from a directory that was simultaneously being pruned and all hell broke loose. Now we're left with a number of dirs and files that can't be deleted. 'du' says there are files in an apparently empty dir: root at bduc-login:/gl/tvanerp/adni 561 $ du -sh * 40K dicom altho 'ls' can't see them from the client; root at bduc-login:/gl/tvanerp/adni 562 $ ls -lat dicom total 40 drwxrwxr-x 3 tvanerp tvanerp 94214 Jun 5 23:40 ./ drwxrwxr-x 3 tvanerp tvanerp 72 Apr 23 13:11 ../ From the bricks you can see that there are still files there: Tue Jun 05 23:18:02 [3.06 2.42 1.65][457.48/606] root at bduc-login:~ 556 $ ssh pbs2 'ls -lat /bducgl/tvanerp/adni/dicom' total 24 drwxrwxr-x 3 7335 7335 28672 2012-06-05 22:59 . drwxr-xr-x 2 7335 7335 8192 2012-06-05 14:50 128_S_0947_20080422_MPRAGE drwxrwxr-x 3 7335 7335 18 2012-04-23 13:11 .. Tue Jun 05 23:18:26 [2.82 2.42 1.67][457.62/606] root at bduc-login:~ 557 $ ssh pbs3 'ls -lat /bducgl/tvanerp/adni/dicom' total 28 drwxrwxr-x 3 7335 7335 28672 2012-06-05 23:00 . drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:52 128_S_0947_20080422_MPRAGE drwxrwxr-x 3 7335 7335 18 2012-04-23 13:11 .. Tue Jun 05 23:18:51 [2.73 2.43 1.69][457.7/606] root at bduc-login:~ 558 $ ssh pbs4 'ls -lat /bducgl/tvanerp/adni/dicom' total 32 drwxrwxr-x 3 7335 7335 36864 2012-06-05 22:59 . drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:50 128_S_0947_20080422_MPRAGE ---------T 2 root root 0 2012-05-20 18:35 adni_pib_subjects.txt drwxrwxr-x 3 7335 7335 18 2012-04-23 13:11 .. but the client is not able to see/delete them, even as root. Suggestions? harry -- Harry Mangalam,Research Computing, OIT, Rm 225 MSTB, UC Irvine [mailcode 2225] Google Voice Multiplexer: (949) 478-4487 415 South Circle View Dr, Irvine, CA, 92697 [shipping] Lat/Long: (33.642025,-117.844414) [paste into Google Maps] -- -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://gluster.org/pipermail/gluster-users/attachments/20120605/8d9fd407/attachment-0001.htm>
