Hello Peter,
Thank you for your answer.
In our setup we have the virtual machines running in KVM, and accessing the ceph storage using librbd.
The rbd cache is set to "writethrough until flush = true". Here it is the result of ceph config show | grep cache :
# ceph --admin-daemon /run/ceph/guests/ceph-client.cinder.30601.140275026217152.asok config show | grep cache
"debug_objectcacher": "0\/5",
"mon_osd_cache_size": "10",
"mon_cache_target_full_warn_ratio": "0.66",
"mon_warn_on_cache_pools_without_hit_sets": "true",
"client_cache_size": "16384",
"client_cache_mid": "0.75",
"mds_cache_size": "100000",
"mds_cache_mid": "0.7",
"mds_dump_cache_on_map": "false",
"mds_dump_cache_after_rejoin": "false",
"osd_pool_default_cache_target_dirty_ratio": "0.4",
"osd_pool_default_cache_target_full_ratio": "0.8",
"osd_pool_default_cache_min_flush_age": "0",
"osd_pool_default_cache_min_evict_age": "0",
"osd_tier_default_cache_mode": "writeback",
"osd_tier_default_cache_hit_set_count": "4",
"osd_tier_default_cache_hit_set_period": "1200",
"osd_tier_default_cache_hit_set_type": "bloom",
"osd_tier_default_cache_min_read_recency_for_promote": "1",
"osd_map_cache_size": "500",
"osd_pg_object_context_cache_count": "64",
"leveldb_cache_size": "134217728",
"rocksdb_cache_size": "0",
"filestore_omap_header_cache_size": "1024",
"filestore_fd_cache_size": "128",
"filestore_fd_cache_shards": "16",
"keyvaluestore_header_cache_size": "4096",
"rbd_cache": "true",
"rbd_cache_writethrough_until_flush": "true",
"rbd_cache_size": "134217728",
"rbd_cache_max_dirty": "100663296",
"rbd_cache_target_dirty": "67108864",
"rbd_cache_max_dirty_age": "1",
"rbd_cache_max_dirty_object": "0",
"rbd_cache_block_writes_upfront": "false",
"rgw_cache_enabled": "true",
"rgw_cache_lru_size": "10000",
"rgw_keystone_token_cache_size": "10000",
"rgw_bucket_quota_cache_size": "10000",
I did some tests and the problem has appeared when I was using ext4 in the VM, but not in the case of xfs.
I did an other test when I was calling a sync at the end of the while loop, and in this case the issue did NOT appeared.
Kind regards,
Laszlo
On 08.04.2017 00:07, Peter Maloney wrote:
You should describe your configuration...
krbd? librbd? cephfs?
is rbd_cache = true?
rbd cache writethrough until flush = true?
is it kvm?
maybe the filesystem in the VM is relevant
(I saw something similar testing cephfs... if I blacklisted a client and
then force unmounted, I would get whole files (appended I think) or ends
of files (new files) with zeros)
On 04/07/17 13:36, Laszlo Budai wrote:
Hello,
we have observed that there are null characters written into the open
files when hard rebooting a VM. Is tis a known issue?
Our VM is using ceph (0.94.10) storage.
we have a script like this:
while sleep 1; do date >> somefile ; done
if we hard reset the VM while the above line is running we end up with
NULL characters in the file :
00000000 54 68 75 20 41 70 72 20 20 36 20 31 34 3a 33 37 |Thu Apr
6 14:37|
00000010 3a 33 33 20 43 45 53 54 20 32 30 31 37 0a 54 68 |:33 CEST
2017.Th|
00000020 75 20 41 70 72 20 20 36 20 31 34 3a 33 37 3a 33 |u Apr 6
14:37:3|
00000030 34 20 43 45 53 54 20 32 30 31 37 0a 54 68 75 20 |4 CEST
2017.Thu |
00000040 41 70 72 20 20 36 20 31 34 3a 33 37 3a 33 35 20 |Apr 6
14:37:35 |
00000050 43 45 53 54 20 32 30 31 37 0a 54 68 75 20 41 70 |CEST
2017.Thu Ap|
00000060 72 20 20 36 20 31 34 3a 33 37 3a 33 36 20 43 45 |r 6
14:37:36 CE|
00000070 53 54 20 32 30 31 37 0a 54 68 75 20 41 70 72 20 |ST
2017.Thu Apr |
00000080 20 36 20 31 34 3a 33 37 3a 33 39 20 43 45 53 54 | 6
14:37:39 CEST|
00000090 20 32 30 31 37 0a 54 68 75 20 41 70 72 20 20 36 | 2017.Thu
Apr 6|
000000a0 20 31 34 3a 33 37 3a 34 30 20 43 45 53 54 20 32 | 14:37:40
CEST 2|
000000b0 30 31 37 0a 54 68 75 20 41 70 72 20 20 36 20 31 |017.Thu
Apr 6 1|
000000c0 34 3a 33 37 3a 34 31 20 43 45 53 54 20 32 30 31 |4:37:41
CEST 201|
000000d0 37 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|7...............|
000000e0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
|................|
We've observed the same in the syslog file also.
Any thoughts about it?
kind regards,
Laszlo
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
