Hi,i've just read through the whole log files from the last few days, when i started these massive tests.
just to let you know in case i haven't done so before.... i'm running two types of tests.
1. sending to the mail system several emails per second, postfixs just keep delivering mails every second. A part from that, a script checks several mailboxes via imap and pop3 every minute.
2. running some disk tests, like bonnie.anyway... a part from that issue with only 1 cpu used and also running out of ram memory... i was wondering if anything would be logged. And actually it is. The thing is that nodes haven't reported any error or warning at all in days, nor have dovecots.
But postfixs keep logging this from time to time. It doesn't happen all the time, but when it does, it keeps logging this for several minutes, every second.
*************2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785301: /dummymassiu/new/1205166062.V12I514f6M451321.postfix01gluster01.cdmon.com,S=1322 => 333046 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785307: /dummymassiu/new/1205169576.V12I528b6M828341.postfix01gluster01.cdmon.com,S=2113 => 338102 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785308: /dummymassiu/new/1205165435.V12I5115eM368089.postfix01gluster01.cdmon.com,S=1322 => 332126 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785310: /dummymassiu/new/1205163678.V12I50696M779037.postfix01gluster01.cdmon.com,S=2110 => 329366 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785311: /dummymassiu/new/1205166503.V12I517e8M525608.postfix02gluster01.cdmon.com,S=1321 => 333800 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785312: /dummymassiu/new/1205168060.V12I521beM376767.postfix01gluster01.cdmon.com,S=1323 => 336318 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785313: /dummymassiu/new/1205172223.V12I52adaM551011.postfix02gluster01.cdmon.com,S=1321 => 338650 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785316: /dummymassiu/new/1205163863.V12I507c2M877288.postfix01gluster01.cdmon.com,S=2113 => 329666 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785320: /dummymassiu/new/1205163951.V12I50862M599096.postfix01gluster01.cdmon.com,S=1323 => 329826 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785322: /dummymassiu/new/1205169551.V12I52898M543504.postfix03gluster01.cdmon.com,S=2110 => 338072 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785324: /dummymassiu/new/1205167646.V12I51f74M976792.postfix03gluster01.cdmon.com,S=2110 => 335732 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:27 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785326: /dummymassiu/new/1205167981.V12I52149M809051.postfix04gluster01.cdmon.com,S=1323 => 336201 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785330: /dummymassiu/new/1205166685.V12I51916M157626.postfix01gluster01.cdmon.com,S=2112 => 334102 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785332: /dummymassiu/new/1205164950.V12I50e5eM93799.postfix01gluster01.cdmon.com,S=2112 => 331358 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785333: /dummymassiu/new/1205164539.V12I50c5aM252840.postfix01gluster01.cdmon.com,S=2112 => 330842 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785336: /dummymassiu/new/1205166068.V12I51505M733823.postfix04gluster01.cdmon.com,S=1323 => 333061 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785339: /dummymassiu/new/1205164143.V12I5098dM237821.postfix04gluster01.cdmon.com,S=2113 => 330125 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785340: /dummymassiu/new/1205169554.V12I5289bM892195.postfix02gluster01.cdmon.com,S=2110 => 338075 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785341: /dummymassiu/new/1205169041.V12I52639M355241.postfix04gluster01.cdmon.com,S=1322 => 337465 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785344: /dummymassiu/new/1205168969.V12I525f9M212765.postfix01gluster01.cdmon.com,S=2112 => 337401 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785345: /dummymassiu/new/1205165036.V12I50f0bM828482.postfix03gluster01.cdmon.com,S=1321 => 331531 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785347: /dummymassiu/new/1205167618.V12I51f40M129843.postfix03gluster01.cdmon.com,S=2110 => 335680 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785348: /dummymassiu/new/1205165356.V12I510f0M733837.postfix02gluster01.cdmon.com,S=2111 => 332016 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785349: /dummymassiu/new/1205166260.V12I51656M140344.postfix01gluster01.cdmon.com,S=1323 => 333398 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785350: /dummymassiu/new/1205166740.V12I5197bM561623.postfix01gluster01.cdmon.com,S=1322 => 334203 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785353: /dummymassiu/new/1205165612.V12I51257M218424.postfix02gluster01.cdmon.com,S=1321 => 332375 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785359: /dummymassiu/new/1205166335.V12I516d7M906967.postfix03gluster01.cdmon.com,S=2110 => 333527 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785360: /dummymassiu/new/1205166706.V12I5193bM376041.postfix02gluster01.cdmon.com,S=2111 => 334139 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785362: /dummymassiu/new/1205167774.V12I52045M925044.postfix04gluster01.cdmon.com,S=1322 => 335941 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785363: /dummymassiu/new/1205163717.V12I506d3M585524.postfix01gluster01.cdmon.com,S=2111 => 329427 Rehashing because st_nlink less than dentry maps 2008-03-13 08:18:28 W [fuse-bridge.c:391:fuse_entry_cbk] glusterfs-fuse: 2785366: /dummymassiu/new/1205166212.V12I5160dM479271.postfix04gluster01.cdmon.com,S=1322 => 333325 Rehashing because st_nlink less than dentry maps
************* En/na Jordi Moles ha escrit:
Hi, yes, sorry, that is a typo, i mean: dom0 is NOT running any server or client daemon. En/na Basavanagowda Kanur ha escrit:Jordi, I have one more question, inline with your the message below. On Wed, Mar 12, 2008 at 10:25 PM, Jordi Moles <jordi@xxxxxxxxx> wrote:Hi,I'm using Xen 3.2.0 on debian lenny machines, xen installed from apt-getrepositories. All machines, nodes, postfixs and dovecots are virtuals.nodes have 2GB of ram and 4GB of swap, dovecots and postfixs 512M of ramand also 4GB of swap. dom0 is running any server or client gluster daemon.Did you mean 'dom0 is not running any server or client glusterfs daemon'?I've attached a picture from cacti, which i use to monitor all parts ofgluster. There are two points at which the node runs out of ram memory. The first one is when i perform a postmark test, and the second one is abonnie test. The other readings which are close to 1GB refers to "normal" use of my mail system, receiving mails, people checking their mailboxes and so on. no errors or warnings are reported in the gluster log files. here's the xen config files from any kind of machine: NODES ************* # # Configuration file for the Xen instance node01gluster01, created # by xen-tools 3.9 on Thu Feb 28 16:39:27 2008. # # # Kernel + memory size # kernel = '/boot/vmlinuz-2.6.18-6-xen-amd64' ramdisk = '/boot/initrd.img-2.6.18-6-xen-amd64' memory = '2048' vcpus = '2' # # Disk device(s). # root = '/dev/sda2 ro' disk = ['file:/home/xen/domains/node01gluster01/swap.img,sda1,w', 'file:/home/xen/domains/node01gluster01/disk.img,sda2,w',] # # Hostname # name = 'node01gluster01' # # Networking # vif = ['bridge=xenbr0,ip=192.168.1.204,mac=00:16:3E:B3:2F:18','bridge=xenbr1,ip=192.168.2.204,mac=00:16:3E:B3:2F:19' ] # # Behaviour # on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' ************* DOVECOTS ************* # Configuration file for the Xen instance dovecot02gluster01.cdmon.com, created # by xen-tools 3.9 on Thu Feb 28 16:51:32 2008. # # # Kernel + memory size # kernel = '/boot/vmlinuz-2.6.18-6-xen-amd64' ramdisk = '/boot/initrd.img-2.6.18-6-xen-amd64' memory = '512' # # Disk device(s). # root = '/dev/sda2 ro' disk = [ 'file:/home/xen/domains/dovecot02gluster01.cdmon.com/swap.img,sda1,w', 'file:/home/xen/domains/dovecot02gluster01.cdmon.com/disk.img,sda2,w', ] # # Hostname # name = 'dovecot02gluster01.cdmon.com' # # Networking # vif = [ 'bridge=xenbr0,ip=212.x.x.x,mac=00:16:3E:B2:E7:69','bridge=xenbr1,ip= 192.168.1.36,mac=00:16:3E:B2:E7:70' ] # # Behaviour # on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' ************** POSTFIXS ************** # # Configuration file for the Xen instance postfix02gluster01.cdmon.com, created # by xen-tools 3.9 on Thu Feb 28 16:40:20 2008. # # # Kernel + memory size # kernel = '/boot/vmlinuz-2.6.18-6-xen-amd64' ramdisk = '/boot/initrd.img-2.6.18-6-xen-amd64' memory = '512' # # Disk device(s). # root = '/dev/sda2 ro' disk = [ 'file:/home/xen/domains/postfix02gluster01.cdmon.com/swap.img,sda1,w', 'file:/home/xen/domains/postfix02gluster01.cdmon.com/disk.img,sda2,w', ] # # Hostname # name = 'postfix02gluster01.cdmon.com' # # Networking # vif = [ 'bridge=xenbr0,ip=212.x.x.x,mac=00:16:3E:F9:47:0D','bridge=xenbr1,ip= 192.168.1.212,mac=00:16:3E:F9:47:0E' ] # # Behaviour # on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart' ************** En/na Basavanagowda Kanur ha escrit:Jordi, We will be trying to reproduce your problem at our setup. To gofurther,we have few questions for you: 1. Where are glusterfs servers running? On virtual machines or real machines? 2. Where are glusterfs clients running? On virtual machines or real machines? 3. Is the Xen dom-0 also using glusterfs client? If yes, what allprocessesare accessing the glusterfs mount. 4. If Xen dom-0 hosts the block device images for Xen guest OSes onglusterfs mount, then what is the access method for guest OSes? loopback(disk=['file:']) or blktp (disk=['tap:aio:']) or nfs rexport? Detailed description of your Xen setup in addition to answers to above questions will help us debug the problem. -- Gowda_______________________________________________ Gluster-devel mailing list Gluster-devel@xxxxxxxxxx http://lists.nongnu.org/mailman/listinfo/gluster-devel
