Hi Sagi, thanks a lot for the analysis. I didn't know about about the inline_data_size parameter in nvmet. It is at PAGE_SIZE on our systems. Will rerun our benchmarks with echo 2097152 > /sys/kernel/config/nvmet/ports/1/param_inline_data_size echo 2097152 > /sys/kernel/config/nvmet/ports/2/param_inline_data_size before enabling the port. Best Danil. On Wed, Jul 10, 2019 at 9:11 PM Sagi Grimberg <sagi@xxxxxxxxxxx> wrote: > > > >> I still do not understand why this should give any notice-able > >> performance advantage. > > > > Usually omitting invalidations gives a healthy bump. > > > > Also, RDMA WRITE is generally faster than READ at the HW level in > > various ways. > > Yes, but this should be essentially identical to running nvme-rdma > with 512KB of immediate-data (the nvme term is in-capsule data). > > In the upstream nvme target we have inline_data_size port attribute > that is tunable for that (defaults to PAGE_SIZE). -- Danil Kipnis Linux Kernel Developer 1&1 IONOS Cloud GmbH | Greifswalder Str. 207 | 10405 Berlin | Germany E-mail: danil.kipnis@xxxxxxxxxxxxxxx | Web: www.ionos.de Head Office: Berlin, Germany District Court Berlin Charlottenburg, Registration number: HRB 125506 B Executive Management: Christoph Steffens, Matthias Steinberg, Achim Weiss Member of United Internet This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail.
