Re: Rebalance task fails

[Nithya Balachandran <nbalacha@xxxxxxxxxx>]

On 13 July 2017 at 14:02, Szymon Miotk <szymon.miotk@xxxxxxxxx> wrote:

Hi Nithya,
I see index in context:
[2017-07-07 10:07:18.230202] E [MSGID: 106062]
[glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict]
0-glusterd: failed to get index

I wonder if there is anything I can do to fix it.
I was trying to strace gluster process but still have no clue what
exactly is gluster index.

This message looks like a glusterd log message (not rebalance) and glusterd_volume_rebalance_use_rsp_dict possibly failed because the rebalance process had already crashed. The index here is simply the value of the number of nodes on which the rebalance process should be running - it is used to track the rebalance status on all nodes.

Best regards,
Szymon Miotk

On Thu, Jul 13, 2017 at 10:12 AM, Nithya Balachandran
<nbalacha@xxxxxxxxxx> wrote:
> Hi Szymon,
>
> I have received the files and will take a look and get back to you.
>
> In what context are you seeing index?
>
> Thanks,
> Nithya
>
> On 11 July 2017 at 01:15, Szymon Miotk <szymon.miotk@xxxxxxxxx> wrote:
>>
>> Hi Nithya,
>>
>> the files were sent to priv to avoid spamming the list with large
>> attachments.
>> Could someone explain what is index in Gluster?
>> Unfortunately index is popular word, so googling is not very helpful.
>>
>> Best regards,
>> Szymon Miotk
>>
>> On Sun, Jul 9, 2017 at 6:37 PM, Nithya Balachandran <nbalacha@xxxxxxxxxx>
>> wrote:
>> >
>> > On 7 July 2017 at 15:42, Szymon Miotk <szymon.miotk@xxxxxxxxx> wrote:
>> >>
>> >> Hello everyone,
>> >>
>> >>
>> >> I have problem rebalancing Gluster volume.
>> >> Gluster version is 3.7.3.
>> >> My 1x3 replicated volume become full, so I've added three more bricks
>> >> to make it 2x3 and wanted to rebalance.
>> >> But every time I start rebalancing, it fails immediately.
>> >> Rebooting Gluster nodes doesn't help.
>> >>
>> >> # gluster volume rebalance  gsae_artifactory_cluster_storage start
>> >> volume rebalance: gsae_artifactory_cluster_storage: success: Rebalance
>> >> on gsae_artifactory_cluster_storage has been started successfully. Use
>> >> rebalance status command to check status of the rebalance process.
>> >> ID: b22572ff-7575-4557-8317-765f7e52d445
>> >>
>> >> # gluster volume rebalance  gsae_artifactory_cluster_storage status
>> >>                                     Node Rebalanced-files
>> >> size       scanned      failures       skipped               status
>> >> run time in secs
>> >>                                ---------      -----------
>> >> -----------   -----------   -----------   -----------
>> >> ------------     --------------
>> >>                                localhost                0
>> >> 0Bytes             0             0             0               failed
>> >>              0.00
>> >>                              10.239.40.9                0
>> >> 0Bytes             0             0             0               failed
>> >>              0.00
>> >>                              10.239.40.8                0
>> >> 0Bytes             0             0             0               failed
>> >>              0.00
>> >> volume rebalance: gsae_artifactory_cluster_storage: success:
>> >>
>> >> The messages in logfiles mention 'failed to get index':
>> >> [2017-07-07 10:07:18.230202] E [MSGID: 106062]
>> >> [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict]
>> >> 0-glusterd: failed to get index
>> >>
>> >> and then the rebalance process crashes:
>> >> pending frames:
>> >> frame : type(0) op(0)
>> >> frame : type(0) op(0)
>> >> frame : type(0) op(0)
>> >> patchset: git://git.gluster.com/glusterfs.git
>> >> signal received: 4
>> >> time of crash:
>> >> 2017-07-07 10:07:23
>> >> configuration details:
>> >> argp 1
>> >> backtrace 1
>> >> dlfcn 1
>> >> libpthread 1
>> >> llistxattr 1
>> >> setfsid 1
>> >> spinlock 1
>> >> epoll.h 1
>> >> xattr.h 1
>> >> st_atim.tv_nsec 1
>> >> package-string: glusterfs 3.7.3
>> >>
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f24de214502]
>> >>
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f24de23059d]
>> >> /lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f24dd612d40]
>> >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x11f6b)[0x7f24dd9b2f6b]
>> >>
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_ref+0x19)[0x7f24de234e69]
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_copy+0x4a)[0x7f24de21291a]
>> >>
>> >>
>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_local_init+0x4b)[0x7f24d851f51b]
>> >>
>> >>
>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_lookup+0x91)[0x7f24d8550521]
>> >>
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(syncop_lookup+0x1a2)[0x7f24de258fc2]
>> >>
>> >>
>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_fix_layout+0x87)[0x7f24d85289e7]
>> >>
>> >>
>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_start_crawl+0x6d3)[0x7f24d8529ce3]
>> >>
>> >>
>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_wrap+0x12)[0x7f24de255822]
>> >> /lib/x86_64-linux-gnu/libc.so.6(+0x498b0)[0x7f24dd6258b0]
>> >>
>> >>
>> >> Anybody has a clue how to fix 'failed to get index' error?
>> >
>> >
>> > Do you have the coredump? If yes, please send that and the rebalance
>> > logs
>> > across.
>> >
>> > Regards,
>> > Nithya
>> >>
>> >>
>> >> Thank you in advance!
>> >> Szymon Miotk
>> >> _______________________________________________
>> >> Gluster-users mailing list
>> >> Gluster-users@xxxxxxxxxxx
>> >> http://lists.gluster.org/mailman/listinfo/gluster-users
>> >
>> >
>
>

_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
http://lists.gluster.org/mailman/listinfo/gluster-users