Attached is the diff file between the config files of kernels 21 & 27. I see a lot of DMA changes and schedulers. I've asked you before if you compiled your own kernel or do you use unmodified distro kernels? If you compile your own, then you could import the 21 config to the 27 compile process and then compare the performance of the two kernels. 2009/9/17 Linux Raid Study <linuxraid.study@xxxxxxxxx>: > Hi: > > I tried to change the Raid5 stripe cache size by doing > "echo 262144 > /sys/block/md0/md/stripe_cache_size" > But if I do "cat /sys/block/md0/md/stripe_cache_size", it still shows 1024! > > Can someone please suggest how to increase stripe cache size? > > thanks! > > > 2009/9/10 Keld Jørn Simonsen <keld@xxxxxxxx>: >> Hi >> >> There is some data on raid5 performance (and other raid types) at our >> wiki at: http://linux-raid.osdl.org/index.php/Performance >> >> When you have something you want to share with others, then we can add >> your info to the data there, either by a URL or by direct inclusion in >> the wiki. >> >> best regards >> keld >> >> On Wed, Sep 09, 2009 at 06:45:31PM -0700, Linux Raid Study wrote: >>> I build/compile my own kernel.. >>> >>> Have you done any benchmarks for Read RAID5 perf? >>> >>> Thanks! >>> >>> On 9/9/09, Majed B. <majedb@xxxxxxxxx> wrote: >>> > Well, that's my point. To make sure that your kernels are identical in >>> > options, compare the .config files found in /usr/src/linux-<version> >>> > >>> > You may find changes and differences in areas like bluetooth, >>> > wireless, ...etc. These don't matter. >>> > >>> > Did you compile your own kernel, or are you using one compiled for a >>> > specific distro? >>> > >>> > On Thu, Sep 10, 2009 at 4:17 AM, Linux Raid >>> > >>> > Study<linuxraid.study@xxxxxxxxx> wrote: >>> > > Yes - the setup is the same. The only thing that changes is the kernel. >>> > > >>> > > I have 4 HDDs (2 from WD VelociRaptor 150GB/10k RPM and 2 from Seagate >>> > > 320GB/7.2k RPM). The HDDs are connected using two Silicon Image SATA >>> > > controllers (Sil3132) connected to my host using PCIe. >>> > > >>> > > I use the default kernel scheduling scheme. >>> > > >>> > > Thanks! >>> > > >>> > > On 9/9/09, Majed B. <majedb@xxxxxxxxx> wrote: >>> > >> Are you sure you're using the same Kernel Scheduler on both kernels? >>> > >> >>> > >> Have you checked and verified that the same drivers are being loaded? >>> > >> (IDE vs. AHCI -- proprietary drivers vs. open source) >>> > >> >>> > >> This is interesting! >>> > >> >>> > >> >>> > >> On Thu, Sep 10, 2009 at 1:54 AM, Linux Raid >>> > >> Study<linuxraid.study@xxxxxxxxx> wrote: >>> > >> > Hello: >>> > >> > >>> > >> > I'm benchmarking RAID5 Read performance with 4 disks and notice perf >>> > >> > drop when using kernel 2.6.27. >>> > >> > The perf with 2.6.27 is 150MB/s whereas with kernel 2.6.21 (and same >>> > >> > setup), the perf is 180MB/s/ >>> > >> > >>> > >> > >>> > >> > >>> > >> > Some of the tests I have done: >>> > >> > 1. I ran experiments using iozone also but notice that READ perf with >>> > >> > RAID5 in 2.6.27 is ~30% less as compared to 2.6.21. >>> > >> > Write perf is similar but Read is an issue. >>> > >> > >>> > >> > 2. On 2.6.27, I played around with chunk size and blocksize parameters >>> > >> > and got ~5% improvement but not much. I used following cmds: >>> > >> > >>> > >> > #mdadm --create /dev/md0 --level=5 --assume-clean --raid-devices=4 >>> > >> > /dev/sda /dev/sdb /dev/sdc /dev/sdd --chunk=32 >>> > >> > #mke2fs -b 4096 -E stride=8 /dev/md0 >>> > >> > >>> > >> > #time dd if=/dev/md0 of=/dev/null bs=94208 count=1024 >>> > >> > >>> > >> > >>> > >> > Is this perf drop a known issue? Can someone please recommend some >>> > >> > performance tuning tips? >>> > >> > >>> > >> > Thanks a lot for your time and help! >>> > >> >>> > >> > -- >>> > >> > To unsubscribe from this list: send the line "unsubscribe linux-raid" in >>> > >> > the body of a message to majordomo@xxxxxxxxxxxxxxx >>> > >> > More majordomo info at http://vger.kernel.org/majordomo-info.html >>> > >> > >>> > >> >>> > >> >>> > >> >>> > >> -- >>> > >> >>> > >> Majed B. >>> > >> >>> > > >>> > >>> > >>> > >>> > >>> > -- >>> > >>> > Majed B. >>> > >>> -- >>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in >>> the body of a message to majordomo@xxxxxxxxxxxxxxx >>> More majordomo info at http://vger.kernel.org/majordomo-info.html >> > -- Majed B.
--- 21 2009-09-17 05:50:44.091089881 +0300 +++ u 2009-09-17 06:31:01.532090560 +0300 @@ -1,21 +1,30 @@ CONFIG_64BIT=y +CONFIG_64BIT_PHYS_ADDR=y CONFIG_8139TOO=y CONFIG_8139TOO_PIO=y +CONFIG_ANON_INODES=y CONFIG_ARCH_FLATMEM_ENABLE=y CONFIG_ARCH_MAY_HAVE_PC_FDC=y +CONFIG_ARCH_POPULATES_NODE_MAP=y CONFIG_ARCH_SPARSEMEM_ENABLE=y +CONFIG_ARCH_SUPPORTS_OPROFILE=y CONFIG_ASYNC_CORE=y CONFIG_ASYNC_MEMCPY=y +CONFIG_ASYNC_PQ=y CONFIG_ASYNC_PQXOR=y CONFIG_ASYNC_R6RECOV=y +CONFIG_ASYNC_RAID6_RECOV=y CONFIG_ASYNC_XOR=y CONFIG_ATA=y +CONFIG_ATA_SFF=y CONFIG_BASE_FULL=y CONFIG_BINFMT_ELF32=y CONFIG_BINFMT_ELF=y CONFIG_BINFMT_MISC=y CONFIG_BITREVERSE=y +CONFIG_BLK_DEV=y CONFIG_BLK_DEV_CMD64X=y +CONFIG_BLK_DEV_DM=y CONFIG_BLK_DEV_IDE=y CONFIG_BLK_DEV_IDECD=y CONFIG_BLK_DEV_IDEDISK=y @@ -29,11 +38,18 @@ CONFIG_BLK_DEV_SD=y CONFIG_BLK_DEV_VIA82CXXX=y CONFIG_BLOCK=y +CONFIG_BLOCK_COMPAT=y +CONFIG_BOUNCE=y CONFIG_BRIDGE=y CONFIG_BSD_PROCESS_ACCT=y +CONFIG_BSD_PROCESS_ACCT_V3=y CONFIG_BUG=y CONFIG_BUILD_ELF64=y +CONFIG_BUILD_ID_DISABLE=y +CONFIG_CEVT_R4K=y +CONFIG_CLASSIC_RCU=y CONFIG_COMPAT=y +CONFIG_COMPAT_BRK=y CONFIG_CPU_BIG_ENDIAN=y CONFIG_CPU_HAS_PREFETCH=y CONFIG_CPU_HAS_SYNC=y @@ -45,27 +61,46 @@ CONFIG_CRC32=y CONFIG_CRYPTO=y CONFIG_CRYPTO_ALGAPI=y +CONFIG_CRYPTO_BLKCIPHER=y +CONFIG_CRYPTO_CBC=y +CONFIG_CRYPTO_DES=y CONFIG_CRYPTO_HASH=y CONFIG_CRYPTO_HMAC=y +CONFIG_CRYPTO_HW=y CONFIG_CRYPTO_MANAGER=y +CONFIG_CRYPTO_MD5=y CONFIG_DEBUG_INFO=y CONFIG_DEBUG_KERNEL=y CONFIG_DEFAULT_CFQ=y +CONFIG_DEVKMEM=y +CONFIG_DEVPORT=y +CONFIG_DMADEVICES=y CONFIG_DMA_ENGINE=y +CONFIG_DM_CRYPT=y +CONFIG_DM_MIRROR=y +CONFIG_DM_SNAPSHOT=y +CONFIG_DM_ZERO=y CONFIG_DNOTIFY=y CONFIG_DRM=y CONFIG_DRM_RADEON=y CONFIG_DUMMY_CONSOLE=y CONFIG_E1000=y +CONFIG_E1000E=y CONFIG_E1000_NAPI=y CONFIG_ELF_CORE=y CONFIG_EMBEDDED=y CONFIG_ENABLE_MUST_CHECK=y +CONFIG_ENABLE_WARN_DEPRECATED=y CONFIG_EPOLL=y +CONFIG_EVENTFD=y CONFIG_EXPERIMENTAL=y CONFIG_EXT2_FS=y +CONFIG_EXT2_FS_POSIX_ACL=y +CONFIG_EXT2_FS_XATTR=y CONFIG_EXT3_FS=y +CONFIG_EXT3_FS_POSIX_ACL=y CONFIG_EXT3_FS_XATTR=y +CONFIG_FAIR_GROUP_SCHED=y CONFIG_FAST_ACCESS_TO_THREAD_POINTER=y CONFIG_FAT_FS=y CONFIG_FB=y @@ -81,32 +116,55 @@ CONFIG_FORCED_INLINING=y CONFIG_FRAMEBUFFER_CONSOLE=y CONFIG_FS_MBCACHE=y +CONFIG_FS_POSIX_ACL=y CONFIG_FUSION=y CONFIG_FUSION_SAS=y CONFIG_FUTEX=y CONFIG_GENERIC_CALIBRATE_DELAY=y +CONFIG_GENERIC_CLOCKEVENTS=y +CONFIG_GENERIC_CLOCKEVENTS_BUILD=y +CONFIG_GENERIC_CMOS_UPDATE=y CONFIG_GENERIC_FIND_NEXT_BIT=y CONFIG_GENERIC_HARDIRQS=y CONFIG_GENERIC_HWEIGHT=y CONFIG_GENERIC_IRQ_PROBE=y CONFIG_GENERIC_ISA_DMA=y CONFIG_GENERIC_TIME=y +CONFIG_GROUP_SCHED=y +CONFIG_HARDWARE_WATCHPOINTS=y +CONFIG_HAS_DMA=y CONFIG_HAS_IOMEM=y CONFIG_HAS_IOPORT=y +CONFIG_HAVE_ARCH_KGDB=y +CONFIG_HAVE_FTRACE=y +CONFIG_HAVE_GET_CYCLES_32=y +CONFIG_HAVE_IDE=y CONFIG_HAVE_MEMORY_PRESENT=y +CONFIG_HAVE_OPROFILE=y +CONFIG_HAVE_TRACE_CLOCK=y +CONFIG_HAVE_TRACE_CLOCK_32_TO_64=y CONFIG_HID=y +CONFIG_HID_SUPPORT=y CONFIG_HOTPLUG=y CONFIG_HUGETLBFS=y CONFIG_HUGETLB_PAGE=y +CONFIG_HWMON=y +CONFIG_HWTIMER_HOOKS=y CONFIG_HW_CONSOLE=y CONFIG_HW_HAS_PCI=y +CONFIG_HZ_100=y CONFIG_HZ_250=y CONFIG_I2C=y CONFIG_I2C_ALGOBIT=y +CONFIG_I2C_BOARDINFO=y CONFIG_I2C_CHARDEV=y +CONFIG_I2C_HELPER_AUTO=y CONFIG_IDE=y CONFIG_IDEDISK_MULTI_MODE=y CONFIG_IDEPCI_SHARE_IRQ=y +CONFIG_IDE_PROC_FS=y +CONFIG_IKCONFIG=y +CONFIG_IKCONFIG_PROC=y CONFIG_INET=y CONFIG_INET_DIAG=y CONFIG_INET_TCP_DIAG=y @@ -120,12 +178,20 @@ CONFIG_INPUT_MOUSE=y CONFIG_INPUT_MOUSEDEV=y CONFIG_INPUT_MOUSEDEV_PSAUX=y +CONFIG_IOSCHED_AS=y CONFIG_IOSCHED_CFQ=y +CONFIG_IOSCHED_DEADLINE=y CONFIG_IOSCHED_NOOP=y CONFIG_IP_FIB_HASH=y +CONFIG_IP_MROUTE=y CONFIG_IP_MULTICAST=y +CONFIG_IP_PIMSM_V1=y +CONFIG_IP_PIMSM_V2=y CONFIG_IP_PNP=y +CONFIG_IP_PNP_BOOTP=y CONFIG_IP_PNP_DHCP=y +CONFIG_IP_PNP_RARP=y +CONFIG_IRQ_CPU=y CONFIG_IRQ_PER_CPU=y CONFIG_ISA=y CONFIG_ISO9660_FS=y @@ -137,6 +203,7 @@ CONFIG_JOLIET=y CONFIG_KALLSYMS=y CONFIG_KEYBOARD_ATKBD=y +CONFIG_KMOD=y CONFIG_LEGACY_PTYS=y CONFIG_LLC=y CONFIG_LOCKD=y @@ -147,6 +214,9 @@ CONFIG_LOGO_LINUX_CLUT224=y CONFIG_MAGIC_SYSRQ=y CONFIG_MD=y +CONFIG_MD_FAULTY=y +CONFIG_MD_LINEAR=y +CONFIG_MD_MULTIPATH=y CONFIG_MD_RAID0=y CONFIG_MD_RAID10=y CONFIG_MD_RAID1=y @@ -158,6 +228,7 @@ CONFIG_MIPS32_O32=y CONFIG_MIPS=y CONFIG_MIPS_MT_DISABLED=y +CONFIG_MISC_DEVICES=y CONFIG_MMU=y CONFIG_MODULES=y CONFIG_MODULE_UNLOAD=y @@ -171,6 +242,7 @@ CONFIG_MTD_CFI_AMDSTD=y CONFIG_MTD_CFI_GEOMETRY=y CONFIG_MTD_CFI_I1=y +CONFIG_MTD_CFI_I2=y CONFIG_MTD_CFI_NOSWAP=y CONFIG_MTD_CFI_UTIL=y CONFIG_MTD_CHAR=y @@ -178,16 +250,29 @@ CONFIG_MTD_GEN_PROBE=y CONFIG_MTD_JEDECPROBE=y CONFIG_MTD_MAP_BANK_WIDTH_1=y +CONFIG_MTD_MAP_BANK_WIDTH_2=y +CONFIG_MTD_MAP_BANK_WIDTH_4=y +CONFIG_MTD_NAND=y +CONFIG_MTD_NAND_IDS=y CONFIG_MTD_PARTITIONS=y +CONFIG_MTD_PHYSMAP=y CONFIG_MTD_SLRAM=y CONFIG_NATSEMI=y CONFIG_NET=y +CONFIG_NETCONSOLE=y CONFIG_NETDEVICES=y +CONFIG_NETDEV_10000=y +CONFIG_NETDEV_1000=y +CONFIG_NETPOLL=y +CONFIG_NETPOLL_TRAP=y +CONFIG_NETWORK_FILESYSTEMS=y CONFIG_NET_ETHERNET=y CONFIG_NET_PCI=y +CONFIG_NET_POLL_CONTROLLER=y CONFIG_NFS_COMMON=y CONFIG_NFS_FS=y CONFIG_NFS_V3=y +CONFIG_NFS_V4=y CONFIG_NLS=y CONFIG_NLS_CODEPAGE_437=y CONFIG_NLS_ISO8859_1=y @@ -195,8 +280,11 @@ CONFIG_OPROFILE=y CONFIG_PACKET=y CONFIG_PACKET_MMAP=y +CONFIG_PAGEFLAGS_EXTENDED=y CONFIG_PAGE_SIZE_4KB=y CONFIG_PCI=y +CONFIG_PCI_DOMAINS=y +CONFIG_PCI_LEGACY=y CONFIG_PCI_MSI=y CONFIG_PLIST=y CONFIG_POSIX_MQUEUE=y @@ -205,12 +293,15 @@ CONFIG_PRINTK=y CONFIG_PROC_FS=y CONFIG_PROC_KCORE=y +CONFIG_PROC_PAGE_MONITOR=y CONFIG_PROC_SYSCTL=y CONFIG_PROFILING=y CONFIG_RAMFS=y +CONFIG_RELAY=y CONFIG_REPLACE_EMULATED_ACCESS_TO_THREAD_POINTER=y CONFIG_RESOURCES_64BIT=y CONFIG_ROOT_NFS=y +CONFIG_RPCSEC_GSS_KRB5=y CONFIG_RTC_CLASS=y CONFIG_RTC_DEBUG=y CONFIG_RTC_DRV_DS1307=y @@ -222,11 +313,17 @@ CONFIG_RT_MUTEXES=y CONFIG_RWSEM_GENERIC_SPINLOCK=y CONFIG_SATA_MV=y +CONFIG_SATA_PMP=y CONFIG_SATA_SIL24=y +CONFIG_SATA_SIL=y CONFIG_SCHED_NO_NO_OMIT_FRAME_POINTER=y CONFIG_SCSI=y +CONFIG_SCSI_DMA=y CONFIG_SCSI_PROC_FS=y CONFIG_SCSI_SAS_ATTRS=y +CONFIG_SECCOMP=y +CONFIG_SECURITY=y +CONFIG_SECURITY_NETWORK=y CONFIG_SELECT_MEMORY_MODEL=y CONFIG_SERIAL_8250=y CONFIG_SERIAL_8250_CONSOLE=y @@ -239,16 +336,21 @@ CONFIG_SERIO_LIBPS2=y CONFIG_SERIO_SERPORT=y CONFIG_SHMEM=y +CONFIG_SIGNALFD=y CONFIG_SKY2=y CONFIG_SLAB=y +CONFIG_SLABINFO=y CONFIG_SMP=y CONFIG_SPARSEMEM=y CONFIG_SPARSEMEM_MANUAL=y CONFIG_SPARSEMEM_STATIC=y +CONFIG_SSB_POSSIBLE=y CONFIG_STACKTRACE_SUPPORT=y CONFIG_STANDALONE=y CONFIG_STOP_MACHINE=y +CONFIG_STP=y CONFIG_SUNRPC=y +CONFIG_SUNRPC_GSS=y CONFIG_SWAP=y CONFIG_SWAP_IO_SPACE=y CONFIG_SYN_COOKIES=y @@ -256,18 +358,25 @@ CONFIG_SYSCTL_SYSCALL=y CONFIG_SYSFS=y CONFIG_SYSFS_DEPRECATED=y +CONFIG_SYSFS_DEPRECATED_V2=y CONFIG_SYSVIPC=y CONFIG_SYSVIPC_COMPAT=y CONFIG_SYSVIPC_SYSCTL=y +CONFIG_SYS_HAS_EARLY_PRINTK=y +CONFIG_SYS_R4K_CVET_HWTIMER=y CONFIG_SYS_SUPPORTS_32BIT_KERNEL=y CONFIG_SYS_SUPPORTS_64BIT_KERNEL=y CONFIG_SYS_SUPPORTS_ARBIT_HZ=y CONFIG_SYS_SUPPORTS_BIG_ENDIAN=y +CONFIG_SYS_SUPPORTS_DYNAMIC_FTRACE=y CONFIG_SYS_SUPPORTS_HIGHMEM=y CONFIG_SYS_SUPPORTS_KGDB=y CONFIG_SYS_SUPPORTS_SMP=y CONFIG_TCP_CONG_CUBIC=y +CONFIG_THERMAL=y +CONFIG_TIMERFD=y CONFIG_TMPFS=y +CONFIG_TRACEPOINTS=y CONFIG_TRACE_IRQFLAGS_SUPPORT=y CONFIG_UNIX98_PTYS=y CONFIG_UNIX=y @@ -276,21 +385,29 @@ CONFIG_USB_ARCH_HAS_HCD=y CONFIG_USB_ARCH_HAS_OHCI=y CONFIG_USB_DEVICEFS=y +CONFIG_USB_DEVICE_CLASS=y CONFIG_USB_DWC_OTG=y +CONFIG_USB_DYNAMIC_MINORS=y CONFIG_USB_EHCI_HCD=y CONFIG_USB_HID=y +CONFIG_USB_LIBUSUAL=y CONFIG_USB_MON=y CONFIG_USB_OHCI_HCD=y CONFIG_USB_OHCI_LITTLE_ENDIAN=y CONFIG_USB_STORAGE=y +CONFIG_USB_SUPPORT=y CONFIG_USB_UHCI_HCD=y +CONFIG_USER_SCHED=y +CONFIG_USE_GENERIC_SMP_HELPERS=y CONFIG_USE_RI_XI_PAGE_BITS=y CONFIG_VFAT_FS=y CONFIG_VIA_RHINE=y CONFIG_VIA_RHINE_MMIO=y +CONFIG_VIRT_TO_BUS=y CONFIG_VM_EVENT_COUNTERS=y CONFIG_VT=y CONFIG_VT_CONSOLE=y +CONFIG_WATCHDOG=y CONFIG_WEAK_ORDERING=y CONFIG_XFRM=y CONFIG_XOR_BLOCKS=y
