Re: "Mismatching layouts" in glusterfs client logs after new brick addition and rebalance

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

I think it'd be extremely helpful if gluster had a feature to grab all the necessary logs/debug info (maybe a few variations depending on the bug) so that all the user would have to do is enter a simple command and have gluster generate the whole bug report, ready to be sent to to the gluster team.

Sincerely,
Artem

--
Founder, Android Police, APK Mirror, Illogical Robot LLC


On Tue, Jul 7, 2020 at 1:47 AM Shreyansh Shah <shreyansh.shah@xxxxxxxxxxxxxx> wrote:
Sounds good, thank you. 

On Tue, Jul 7, 2020 at 2:12 PM Barak Sason Rofman <bsasonro@xxxxxxxxxx> wrote:
Thanks Shreyansh,

I'll look into it, however I'll likely need some help from more senior team members to perform RCA.
I'll update once I have new insights.

My regards,

On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah <shreyansh.shah@xxxxxxxxxxxxxx> wrote:
Hi Barak,
Thanks for looking into this and helping me out,
The fix-layout was successful, and I ran a rebalance after completion of fix-layout.
The rebalance status though did show failure for 3 nodes.

On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro@xxxxxxxxxx> wrote:
Greetings again Shreyansh,

I'm indeed seeing a lot of errors in the log file - still unsure about the RC.
You mentioned that prior to running rebalance you ran fix-layout, was the fix-layout successful?
Another question - did you wait until fix-layout was completed before running rebalance?

My thanks,

On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah <shreyansh.shah@xxxxxxxxxxxxxx> wrote:
Hi,
Attaching rebalance logs
FYI, we ran "gluster rebalance fix-layout" followed by "gluster rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout"


PFA

On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <bsasonro@xxxxxxxxxx> wrote:
I think it would be best.
As I can't say at this point where the problem is originating from, brick logs might also be necessary (I assume I would have a better picture once I have the rebalance logs).

Cheers,

On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah <shreyansh.shah@xxxxxxxxxxxxxx> wrote:
Hi Barak,
Can provide the rebalance logs. Do you require all the brick logs (14 in total)?

On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman <bsasonro@xxxxxxxxxx> wrote:
Greetings Shreyansh,

Off-hand I can't come up with a reason for these failures.
In order to start looking into this, access to the full rebalance logs is required (possibly brick logs as well).
Can you provide those?

My regards,


On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <shreyansh.shah@xxxxxxxxxxxxxx> wrote:
Hi, 
Did anyone get a chance to look into this?

On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <shreyansh.shah@xxxxxxxxxxxxxx> wrote:
Hi All,

We are facing "Mismatching layouts for <directory>,gfid = <id>" errors.

We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to the existing setup.
Post that we did a rebalance fix-layout and then a rebalance (which is currently still in progress). The status shows "failed" on certain bricks but "in progress" for others. Adding output for gluster rebalance status below.

The glusterfs client logs are flooded with "Mismatching layouts for <directory>,gfid = <id>"
The performance too seems to have degraded due to this, even basic commands like `cd` and `ls` are taking more than a minute compared to sub-second number before brick addition.
Apart from that we also experienced many binaries and files giving error stale file handle error even though the files were present.


gluster rebalance status :

Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
localhost              176         3.5GB         12790             0          8552          in progress       21:36:01
10.132.0.72             8232       394.8GB         19995            21            26               failed       14:50:30
10.132.0.44            12625         1.0TB         50023             1         10202          in progress       21:36:00
10.132.0.3            21982       956.8GB         79145             1         34571          in progress       21:36:00
10.132.0.9             7975       355.8GB         20157             6          1522               failed       14:51:45
10.132.0.73             6293       394.5GB         26414           151          8085               failed       14:51:45
10.132.0.70             6480       477.1GB         21058            27          1787               failed       14:50:32
Estimated time left for rebalance to complete :      130:56:28


Logs from one of the clients below:

[2020-07-02 12:30:14.971916] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk layout - 2761060380 - 3067813815 - 4159036738
[2020-07-02 12:30:14.971935] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.032013] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk layout - 3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:15.032059] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.032107] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:15.032153] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.093329] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk layout - 2454306944 - 2761060379 - 4159036738
[2020-07-02 12:30:15.093373] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.093460] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk layout - 2761060380 - 3067813815 - 4159036738
[2020-07-02 12:30:15.093515] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.151063] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk layout - 3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:15.151108] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.151149] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:15.151162] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.424321] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk layout - 920400036 - 1227153471 - 4159036738
[2020-07-02 12:30:15.424380] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424456] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk layout - 1840730208 - 2147483643 - 4159036738
[2020-07-02 12:30:15.424484] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424525] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk layout - 1533976772 - 1840730207 - 4159036738
[2020-07-02 12:30:15.424542] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424596] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk layout - 613646600 - 920400035 - 4159036738
[2020-07-02 12:30:15.424607] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:16.004482] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on data-client-7 (hashed subvol is data-client-17)
[2020-07-02 12:30:16.005523] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_1_DATA.dat
[2020-07-02 12:30:16.531047] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat on data-client-9 (hashed subvol is data-client-19)
[2020-07-02 12:30:16.532086] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_1_METADATA.dat
[2020-07-02 12:30:18.733229] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on data-client-17 (hashed subvol is data-client-9)
[2020-07-02 12:30:18.734421] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_2_DATA.dat
[2020-07-02 12:30:19.171930] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat on data-client-9 (hashed subvol is data-client-18)
[2020-07-02 12:30:19.172901] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_2_METADATA.dat
[2020-07-02 12:30:21.028495] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on data-client-6 (hashed subvol is data-client-15)
[2020-07-02 12:30:21.029836] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_2_DATA.dat
[2020-07-02 12:30:21.127648] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat on data-client-11 (hashed subvol is data-client-3)
[2020-07-02 12:30:21.128713] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_2_METADATA.dat
[2020-07-02 12:30:21.201126] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on data-client-15 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.201928] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_3_DATA.dat
[2020-07-02 12:30:21.566158] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat on data-client-7 (hashed subvol is data-client-16)
[2020-07-02 12:30:21.567123] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_3_METADATA.dat
[2020-07-02 12:30:21.649357] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:21.661381] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_4_DATA.dat
[2020-07-02 12:30:21.748937] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat on data-client-15 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.749481] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_4_METADATA.dat
[2020-07-02 12:30:21.898593] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on data-client-14 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.899442] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_6_DATA.dat
[2020-07-02 12:30:22.039337] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat on data-client-10 (hashed subvol is data-client-2)
[2020-07-02 12:30:22.040086] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_6_METADATA.dat
[2020-07-02 12:30:22.501877] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat on data-client-15 (hashed subvol is data-client-8)
[2020-07-02 12:30:22.502712] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECDS1_DATA.dat
[2020-07-02 12:30:22.782577] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11 (hashed subvol is data-client-6)
[2020-07-02 12:30:22.783777] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECDS1_METADATA.dat
[2020-07-02 12:30:23.146847] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on data-client-17 (hashed subvol is data-client-9)
[2020-07-02 12:30:23.148009] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM1_DATA.dat
[2020-07-02 12:30:23.229290] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed subvol is data-client-6)
[2020-07-02 12:30:23.230151] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM1_METADATA.dat
[2020-07-02 12:30:23.889520] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:23.896618] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM2_DATA.dat
[2020-07-02 12:30:24.093017] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed subvol is data-client-15)
[2020-07-02 12:30:24.094117] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM2_METADATA.dat
[2020-07-02 12:30:24.345257] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on data-client-17 (hashed subvol is data-client-10)
[2020-07-02 12:30:24.346234] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM3_DATA.dat
[2020-07-02 12:30:24.425835] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed subvol is data-client-15)
[2020-07-02 12:30:24.426880] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM3_METADATA.dat
[2020-07-02 12:30:25.158718] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat on data-client-9 (hashed subvol is data-client-19)
[2020-07-02 12:30:25.159619] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO1_DATA.dat
[2020-07-02 12:30:25.531479] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed subvol is data-client-10)
[2020-07-02 12:30:25.540569] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
[2020-07-02 12:30:25.771692] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat on data-client-11 (hashed subvol is data-client-3)
[2020-07-02 12:30:25.772610] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO2_DATA.dat
[2020-07-02 12:30:25.866118] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15 (hashed subvol is data-client-8)
[2020-07-02 12:30:25.866917] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
[2020-07-02 12:30:26.424386] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat on data-client-9 (hashed subvol is data-client-18)
[2020-07-02 12:30:26.425309] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO3_DATA.dat
[2020-07-02 12:30:26.818852] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10 (hashed subvol is data-client-2)
[2020-07-02 12:30:26.819890] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
[2020-07-02 12:30:27.352405] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat on data-client-10 (hashed subvol is data-client-2)
[2020-07-02 12:30:27.352914] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO4_DATA.dat
[2020-07-02 12:30:27.521286] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed subvol is data-client-18)
[2020-07-02 12:30:27.522325] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
[2020-07-02 12:30:28.566634] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat on data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:28.579295] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO5_DATA.dat
[2020-07-02 12:30:28.958028] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat on data-client-7 (hashed subvol is data-client-16)
[2020-07-02 12:30:28.959102] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO6_DATA.dat
[2020-07-02 12:30:29.012429] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed subvol is data-client-15)
[2020-07-02 12:30:29.013416] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
[2020-07-02 12:30:29.396716] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on data-client-17 (hashed subvol is data-client-10)
[2020-07-02 12:30:29.397740] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSEFO_BSE_TSDATA.dat
[2020-07-02 12:30:29.556312] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed subvol is data-client-18)
[2020-07-02 12:30:29.557197] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
[2020-07-02 12:30:30.605354] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9 (hashed subvol is data-client-19)
[2020-07-02 12:30:30.606117] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
[2020-07-02 12:30:31.559206] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout - 613576736 - 920330171 - 4159036738
[2020-07-02 12:30:31.559255] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
[2020-07-02 12:30:31.569025] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout - 920330172 - 1227083607 - 4159036738
[2020-07-02 12:30:31.569067] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
[2020-07-02 12:30:31.701849] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:31.701895] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373
[2020-07-02 12:30:31.738464] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout - 3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:31.738507] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373
[2020-07-02 12:30:31.857102] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk layout - 3067883680 - 3374637115 - 4159036738
[2020-07-02 12:30:31.857147] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.857180] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:31.857197] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.917705] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0 - 306753435 - 4159036738
[2020-07-02 12:30:31.917781] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.917855] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk layout - 3988213852 - 4294967295 - 4159036738
[2020-07-02 12:30:31.917874] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:32.390945] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout - 3681460416 - 3988213851 - 4159036738
[2020-07-02 12:30:32.390998] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/NIFTY, gfid = b2d4deb7-c58c-4046-b6f2-7c7f44d71311
[2020-07-02 12:30:32.391056] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout - 3988213852 - 4294967295 - 4159036738
[2020-07-02 12:30:32.391075] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/NIFTY, gfid = b2d4deb7-c58c-4046-b6f2-7c7f44d71311
[2020-07-02 12:33:50.915279] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 4-data-dht: renaming /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2 (2cb54500-814d-4e85-83e7-e33d9440b18d) (hash=data-client-6/cache=data-client-18) => /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null)) (hash=data-client-6/cache=<nul>)
[2020-07-02 12:34:09.799586] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 4-data-dht: renaming /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k (99938ee6-6986-4123-9d72-ec09e2310b4f) (hash=data-client-17/cache=data-client-18) => /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null)) (hash=data-client-17/cache=<nul>)
....


Please look into this at top-priority if possible.
Let me know if anything else is required.


--
Regards,
Shreyansh Shah


--
Regards,
Shreyansh Shah
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users


--
Barak Sason Rofman

Gluster Storage Development

Red Hat Israel

34 Jerusalem rd. Ra'anana, 43501

bsasonro@redhat.com    T: +972-9-7692304
M: +972-52-4326355



--
Regards,
Shreyansh Shah


--
Barak Sason Rofman

Gluster Storage Development

Red Hat Israel

34 Jerusalem rd. Ra'anana, 43501

bsasonro@redhat.com    T: +972-9-7692304
M: +972-52-4326355



--
Regards,
Shreyansh Shah


--
Barak Sason Rofman

Gluster Storage Development

Red Hat Israel

34 Jerusalem rd. Ra'anana, 43501

bsasonro@redhat.com    T: +972-9-7692304
M: +972-52-4326355



--
Regards,
Shreyansh Shah


--
Barak Sason Rofman

Gluster Storage Development

Red Hat Israel

34 Jerusalem rd. Ra'anana, 43501

bsasonro@redhat.com    T: +972-9-7692304
M: +972-52-4326355



--
Regards,
Shreyansh Shah
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux