The disks are on seperate partition and I'm using the btrfs file system.
They are mounted under /data/osd0 osd1.....
I remove the snapshots and the the system was reporting HEALTH WARNING.
two of the osd went down
ceph ods stat reports:
2011-03-21 19:14:00.122945 7f8c1d83e720 -- :/26712 messenger.start
2011-03-21 19:14:00.123344 7f8c1d83e720 -- :/26712 --> mon0 10.0.6.10:6789/0 -- auth(proto 0 30 bytes) v1 -- ?+0 0x242d4c0
2011-03-21 19:14:00.123701 7f8c1d83d700 -- 10.0.6.10:0/26712 learned my addr 10.0.6.10:0/26712
2011-03-21 19:14:00.124305 7f8c1b1c7700 -- 10.0.6.10:0/26712 <== mon0 10.0.6.10:6789/0 1 ==== auth_reply(proto 1 0 Success) v1 ==== 24+0+0 (709083268 0 0) 0x242d4c0 con 0x242f280
2011-03-21 19:14:00.124349 7f8c1b1c7700 -- 10.0.6.10:0/26712 --> mon0 10.0.6.10:6789/0 -- mon_subscribe({monmap=0+}) v1 -- ?+0 0x242f5d0
2011-03-21 19:14:00.124667 7f8c1b1c7700 -- 10.0.6.10:0/26712 <== mon0 10.0.6.10:6789/0 2 ==== mon_map v1 ==== 187+0+0 (4038329719 0 0) 0x242d4c0 con 0x242f280
2011-03-21 19:14:00.124746 7f8c1b1c7700 -- 10.0.6.10:0/26712 <== mon0 10.0.6.10:6789/0 3 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (3131629013 0 0) 0x242f5d0 con 0x242f280
2011-03-21 19:14:00.124744 mon <- [osd,stat]
2011-03-21 19:14:00.124824 7f8c1d83e720 -- 10.0.6.10:0/26712 --> mon0 10.0.6.10:6789/0 -- mon_command(osd stat v 0) v1 -- ?+0 0x242d4c0
2011-03-21 19:14:00.125131 7f8c1b1c7700 -- 10.0.6.10:0/26712 <== mon0 10.0.6.10:6789/0 4 ==== mon_command_ack([osd,stat]=0 e426: 4 osds: 2 up, 2 in v426) v1 ==== 69+0+0 (3071290324 0 0) 0x242d4c0 con 0x242f280
2011-03-21 19:14:00.125155 mon0 -> 'e426: 4 osds: 2 up, 2 in' (0)
2011-03-21 19:14:00.125559 7f8c1d83e720 -- 10.0.6.10:0/26712 shutdown complete.
I restarted the cluser and it seemd ok again. The data is accessable.
Now ods2 has also cleared some data.
osd0 1.1GB
osd1 1.1GB
osd2 1.2GB
osd3 24GB
But du is reporting 110MB on the mounted filesystem.
Is there a way to recover as it seems as if something is corupt in my system.
It also seems as some of my ods has difficulties to stay up, not sure what I have done wrong.
Maybe the best is to restart with a new file system :-)
----- Ursprungligt meddelande -----
FrÃn: "Ben De Luca" <bdeluca@xxxxxxxxx>
Till: "Gregory Farnum" <gregory.farnum@xxxxxxxxxxxxx>
Kopia: "Martin Wilderoth" <martin.wilderoth@xxxxxxxxxx>, ceph-devel@xxxxxxxxxxxxxxx
Skickat: mÃndag, 21 mar 2011 18:32:46
Ãmne: Re: Disk allocation
Sorry to jump into the converstation, how slow can the deletion of
files actually be?
One of the tests I ran a few weeks ago had me generating files,
deleting them and then writing them again from a number of clients. I
noticed that the space would never freed up again. I have my OSD's and
their journals on dedicated partions.
I had planned on asking more on this once I had a stable system again.
On Mon, Mar 21, 2011 at 3:17 PM, Gregory Farnum
<gregory.farnum@xxxxxxxxxxxxx> wrote:
> On Sat, Mar 19, 2011 at 11:43 PM, Martin Wilderoth
> <martin.wilderoth@xxxxxxxxxx> wrote:
>> I have a small ceph cluster with 4 osd ( 2 disks on 2 hosts).
>>
>> I have been adding and removing files from the file system, mounted as ceph on an other host.
>>
>> Now I have removed most of the data on the file system, so I only have 300 MB left plus two snapshots.
>>
>> The problem is that looking at the disks the are allocating 88G of data
>> on the ceph filesystem.
> There are a few possibilities:
> 1) You've hosted your OSDs on a partition that's shared with the rest
> of the computer. In that case the reported used space will include
> whatever else is on the partition, not just the Ceph files. (This can
> include Ceph debug logs, so even if nothing used to be there but you
> were logging on that partition that can build up pretty quickly.)
> 2) You deleted the files quickly and just haven't given enough time
> for the file deletion to propagate to the OSDs. Because the POSIX
> filesystem is layered over an object store, this can take some time.
> 3) Your snapshots contain a lot of files, so nothing (or very little)
> actually got deleted. Snapshots are pretty cool but they aren't
> miraculous disk space!
> Given the uneven distribution of disk space I suspect option #2, but I
> could be mistaken. :) Let us know!
> -Greg
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@xxxxxxxxxxxxxxx
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
