Hi,
On 07/01/2015 06:09 PM, Gregory Farnum wrote:
On Mon, Jun 29, 2015 at 1:44 PM, Burkhard Linke
<Burkhard.Linke@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx> wrote:
Hi,
I've noticed that a number of placement groups in our setup contain objects,
but no actual data
(ceph pg dump | grep remapped during a hard disk replace operation):
7.616 2636 0 0 5272 0 4194304 3003 3003
active+remapped+wait_backfill 2015-06-29 13:43:28.716687 159913'33987
160091:526298 [30,6,36] 30
[30,36,3] 30 153699'33892 2015-06-29 07:30:16.030470
149573'32565 2015-06-23 07:00:21.948563
7.60a 2696 0 0 5392 0 0 3046 3046
active+remapped+wait_backfill 2015-06-29 13:43:09.847541 159919'34627
160091:388532 [2,36,3] 2
[2,36,31] 2 153669'34496 2015-06-28 20:09:51.850005
153669'34496 2015-06-28 20:09:51.850005
7.60d 2694 0 2 5388 0 0 3026 3026
active+remapped+wait_backfill 2015-06-29 13:43:27.202928 159939'33708
160091:392535 [31,6,38] 31
[31,38,3] 31 152584'33610 2015-06-29 07:11:37.484500
152584'33610 2015-06-29 07:11:37.484500
....
Pool 7 was used a data pool in cephfs, but almost all files stored in that
pool have been removed:
~# rados df
pool name KB objects clones degraded unfound
rd rd KB wr wr KB
cephfs_test_data 940066 5537838 0 202 0
2022238 1434381904 21823705 3064326550
Is it possible to remove these "zombie" objects, since they influence
maintenance operations like backfilling or recovery?
That's odd; the actual objects should have been deleted (not just
truncated). Have you used this pool for anything else (CephFS metadata
storage, RGW bucket indexes, etc)? What version of Ceph are you
running and what workload did you do to induce this issue?
Ceph version is 0.94.2 (5fb85614ca8f354284c713a2f9c610860720bbf3)
running Ubuntu 14.04 with kernel 3.13.0-55-generic.
The cephfs_test_data has only been used as cephfs data pool in a backup
scenario using rsync. It contained a mix of files resulted from several
rsync attempts from a failing NAS device. Most files were small (kbyte
range). The total number of files in that pool was about 10-15 million
before almost all files were removed. The total size of the pool was
about 10 TB.
Since I want to remove the pool completely I'm currently trying to
locate the remaining files in the filesystem, but that's a low priority
task at the moment.
Best regards,
Burkhard
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
