SOLVED: Problem was caused by bug affecting servers with more than 40 CPUs visible - https://review.gluster.org/#/c/15062/ Rebalance works after upgrading to 3.10.4. Many thanks to Nithya for help! Best regards, Szymon Miotk On Thu, Jul 13, 2017 at 10:32 AM, Szymon Miotk <szymon.miotk@xxxxxxxxx> wrote: > Hi Nithya, > I see index in context: > [2017-07-07 10:07:18.230202] E [MSGID: 106062] > [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict] > 0-glusterd: failed to get index > > I wonder if there is anything I can do to fix it. > I was trying to strace gluster process but still have no clue what > exactly is gluster index. > > Best regards, > Szymon Miotk > > On Thu, Jul 13, 2017 at 10:12 AM, Nithya Balachandran > <nbalacha@xxxxxxxxxx> wrote: >> Hi Szymon, >> >> I have received the files and will take a look and get back to you. >> >> In what context are you seeing index? >> >> Thanks, >> Nithya >> >> On 11 July 2017 at 01:15, Szymon Miotk <szymon.miotk@xxxxxxxxx> wrote: >>> >>> Hi Nithya, >>> >>> the files were sent to priv to avoid spamming the list with large >>> attachments. >>> Could someone explain what is index in Gluster? >>> Unfortunately index is popular word, so googling is not very helpful. >>> >>> Best regards, >>> Szymon Miotk >>> >>> On Sun, Jul 9, 2017 at 6:37 PM, Nithya Balachandran <nbalacha@xxxxxxxxxx> >>> wrote: >>> > >>> > On 7 July 2017 at 15:42, Szymon Miotk <szymon.miotk@xxxxxxxxx> wrote: >>> >> >>> >> Hello everyone, >>> >> >>> >> >>> >> I have problem rebalancing Gluster volume. >>> >> Gluster version is 3.7.3. >>> >> My 1x3 replicated volume become full, so I've added three more bricks >>> >> to make it 2x3 and wanted to rebalance. >>> >> But every time I start rebalancing, it fails immediately. >>> >> Rebooting Gluster nodes doesn't help. >>> >> >>> >> # gluster volume rebalance gsae_artifactory_cluster_storage start >>> >> volume rebalance: gsae_artifactory_cluster_storage: success: Rebalance >>> >> on gsae_artifactory_cluster_storage has been started successfully. Use >>> >> rebalance status command to check status of the rebalance process. >>> >> ID: b22572ff-7575-4557-8317-765f7e52d445 >>> >> >>> >> # gluster volume rebalance gsae_artifactory_cluster_storage status >>> >> Node Rebalanced-files >>> >> size scanned failures skipped status >>> >> run time in secs >>> >> --------- ----------- >>> >> ----------- ----------- ----------- ----------- >>> >> ------------ -------------- >>> >> localhost 0 >>> >> 0Bytes 0 0 0 failed >>> >> 0.00 >>> >> 10.239.40.9 0 >>> >> 0Bytes 0 0 0 failed >>> >> 0.00 >>> >> 10.239.40.8 0 >>> >> 0Bytes 0 0 0 failed >>> >> 0.00 >>> >> volume rebalance: gsae_artifactory_cluster_storage: success: >>> >> >>> >> The messages in logfiles mention 'failed to get index': >>> >> [2017-07-07 10:07:18.230202] E [MSGID: 106062] >>> >> [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict] >>> >> 0-glusterd: failed to get index >>> >> >>> >> and then the rebalance process crashes: >>> >> pending frames: >>> >> frame : type(0) op(0) >>> >> frame : type(0) op(0) >>> >> frame : type(0) op(0) >>> >> patchset: git://git.gluster.com/glusterfs.git >>> >> signal received: 4 >>> >> time of crash: >>> >> 2017-07-07 10:07:23 >>> >> configuration details: >>> >> argp 1 >>> >> backtrace 1 >>> >> dlfcn 1 >>> >> libpthread 1 >>> >> llistxattr 1 >>> >> setfsid 1 >>> >> spinlock 1 >>> >> epoll.h 1 >>> >> xattr.h 1 >>> >> st_atim.tv_nsec 1 >>> >> package-string: glusterfs 3.7.3 >>> >> >>> >> >>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f24de214502] >>> >> >>> >> >>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f24de23059d] >>> >> /lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f24dd612d40] >>> >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x11f6b)[0x7f24dd9b2f6b] >>> >> >>> >> >>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_ref+0x19)[0x7f24de234e69] >>> >> >>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_copy+0x4a)[0x7f24de21291a] >>> >> >>> >> >>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_local_init+0x4b)[0x7f24d851f51b] >>> >> >>> >> >>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_lookup+0x91)[0x7f24d8550521] >>> >> >>> >> >>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(syncop_lookup+0x1a2)[0x7f24de258fc2] >>> >> >>> >> >>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_fix_layout+0x87)[0x7f24d85289e7] >>> >> >>> >> >>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_start_crawl+0x6d3)[0x7f24d8529ce3] >>> >> >>> >> >>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_wrap+0x12)[0x7f24de255822] >>> >> /lib/x86_64-linux-gnu/libc.so.6(+0x498b0)[0x7f24dd6258b0] >>> >> >>> >> >>> >> Anybody has a clue how to fix 'failed to get index' error? >>> > >>> > >>> > Do you have the coredump? If yes, please send that and the rebalance >>> > logs >>> > across. >>> > >>> > Regards, >>> > Nithya >>> >> >>> >> >>> >> Thank you in advance! >>> >> Szymon Miotk >>> >> _______________________________________________ >>> >> Gluster-users mailing list >>> >> Gluster-users@xxxxxxxxxxx >>> >> http://lists.gluster.org/mailman/listinfo/gluster-users >>> > >>> > >> >> _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
