On Aug 29, 2016, at 14:34, Mehmet <ceph@xxxxxxxxxx> wrote:
Hey JC,
after setting up the ceph-cluster i tried to migrate an image from one
of our production vm into ceph via
# rados -p rbd put ...
but i have got always "file too large". I guess this file
# -rw-r--r-- 1 ceph ceph 100G Jul 31 01:04
vm-101-disk-2__head_383C3223__0
is the result of this :) - did not thought that there will be
something stay in ceph after the mentioned error above.
Seems i was wrong...
This could match the time where the issue happened first time...:
1. i tried to put via "rados -p rbd put..." this did not worked (tried
to put a ~400G file...)
2. after ~ 1 week i see the blocked requests after first running
"deep-scrub" (default where ceph starts deep-scrubbing)
I guess the deleting of this file should solve the issue.
Did you see my mail where i wrote the test results of this?
# osd_scrub_chunk_max = 5
# osd_deep_scrub_stride = 1048576
Only corner note.
This seems more to me like a pure radios object of 100GB that was
uploaded to the cluster. From the name it could be a VM disk image
that was uploaded as an object. If it was an RBD object, it’s size
would be in the boundaries of an RBD objects (order 12=4K order
25=32MB).
Verify that when you do a "rados -p rbd ls | grep vm-101-disk-2”
command, you can see an object named vm-101-disk-2.
root@:~# rados -p rbd ls | grep vm-101-disk-2
rbd_id.vm-101-disk-2
vm-101-disk-2
Verify if you have an RBD named this way “rbd -p rbd ls | grep
vm-101-disk-2"
root@:~# rbd -p rbd ls | grep vm-101-disk-2
vm-101-disk-2
As I’m not familiar with proxmox so I’d suggest the following:
If yes to 1, for security, copy this file somewhere else and then to
a
rados -p rbd rm vm-101-disk-2.
root@:~# rbd -p rbd info vm-101-disk-2
rbd image 'vm-101-disk-2':
size 400 GB in 102400 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.5e7d1238e1f29
format: 2
features: layering, exclusive-lock, object-map, fast-diff,
deep-flatten
flags:
The VM with the id "101" is up and running. This is using
"vm-101-disk-2" as disk - i have moved the disk sucessfully in another
way :) (same name :/) after "rados put" did not worked. And as we can
see here the objects for this image also exists within ceph
root@:~# rados -p rbd ls | grep "rbd_data.5e7d1238e1f29" | wc -l
53011
I assumed here to get 102400 objects but as ceph is doing thin
provisining this should be ok.
If no to 1, for security, copy this file somewhere else and then to a
rm -rf vm-101-disk-2__head_383C3223__0
I should be able to delete the mentioned "100G file".
Make sure all your PG copies show the same content and wait for the
next scrub to see what is happening.
Will make a backup of this file and in addition from the vm within
proxmox tomorrow on all involved osds and then start a deep-scrub and
of course keep you informed.
If anything goes wrong you will be able to upload an object with the
exact same content from the file you copied.
Is proxmox using such huge objects for something to your knowledge
(VM
boot image or something else)? Can you search the proxmox mailing
list
and open tickets to verify.
As i already wrote in this eMail i guess that i am the cause for this
:*( with the wrong usage of "rados put".
Proxmox is using librbd to talk with ceph so it should not be able to
create such a large one file.
And is this the cause of the long deep scrub? I do think so but I’m
not in front of the cluster.
Let it see :) - i hope that my next eMail will close this issue.
Thank you very much for your help!
Best regards,
- Mehmet
