Apparently I missed including the mailing list in the response. ---------- Forwarded message ---------- From: Yehuda Sadeh <yehuda@xxxxxxxxxxx> Date: Thu, Feb 20, 2014 at 9:04 AM Subject: Re: [rgw] increase the first chunk size To: Ray Lv <raylv@xxxxxxxxxxxxx> On Thu, Feb 20, 2014 at 5:08 AM, Ray Lv <raylv@xxxxxxxxxxxxx> wrote: > Hi, > > Currently, the first chunk size of a radosgw object is 512KB. > > Here is a case that most of data workload gets to radosgw is ~ 4MB. With the > current first chunk size, each radosgw object is stripped to two chunks > (512K + 3.5M). And we're using several large disks on each host with 40TB > capacity. So there will be 10 millions of files on each host. If the first > chunk size is increased to 4MB, the number of files on each host will be > reduced by 50%. It will be benifitial to performance of read because of > reduced dcache and inode cache footprint in main memory (in other words, > increased cache hit ratio). > > The questions are: > > What's the rationale behind for the current first chunk size? The head side conforms to the read chunk side. The gateway reads in 512k chunks. This is the basic read unit, and when accessing an object we access the head only once and read the entire data + all its attributes in one compound rados operation. This is done to ensure atomicity. We don't have any knowledge as to whether the operation is deemed to fail later on (e.g., insufficient permissions), so it's going to read it anyway. > Are there any side effects if it is increased to 4MB? Will require reading the head in a single operation, which can be a problem with regard to concurrency, add latency to all operations (will take more time to stream data back to the client). Unauthorized requests will use more resources. Yehuda _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |