On 06/27/2012 11:54 AM, Stefan Priebe wrote:
Am 27.06.2012 um 19:23 schrieb "Jim Schutt"<jaschut@xxxxxxxxxx>:
On 06/27/2012 09:19 AM, Stefan Priebe wrote:
Am 27.06.2012 16:55, schrieb Jim Schutt:
This is my current best tuning for my hardware, which uses
24 SAS drives/server, and 1 OSD/drive with a journal partition
on the outer tracks and btrfs for the data store.
Which raid level do you use?
No RAID. Each OSD directly accesses a single
disk, via a partition for the journal and a partition
for the btrfs file store for that OSD.
So you have 24 threads x 24 osds 576 threads running?
Actually, when my 166-client test is running,
"ps -o pid,nlwp,args -C ceph-osd"
tells me that I typically have ~1200 threads/OSD.
Since NPTL uses a 1:1 threading model, I recently had
to increase /proc/sys/kernel/pid_max from the default
32768 to get them all to fit.....
The NICs are Chelsio T4, but I'm not using any of the
TCP stateful offload features for this testing.
I don't know if they have ntuple support, but the
ethtool version I'm using (2.6.33) doesn't mention it.
For kernels I switch back and forth between latest development
kernel from Linus's tree, or latest stable kernel, depending
on where the kernel development cycle is. I usually switch
to the development kernel around -rc4 or so.
Crazy that this works for you. Btrfs is crashing to me in 20s while running full speed on ssd.
Hmmm. The only other obvious difference, based on
what I remember from your other posts, is that you're
testing against RBD, right? I've been testing exclusively
with the Linux kernel client.
???
-- Jim
Stefan
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
