Also include gluster v status, I want to check the status of your bricks and SHD processes. -b ----- Original Message ----- > From: "Ben Turner" <bturner@xxxxxxxxxx> > To: "Christoph Schäbel" <christoph.schaebel@xxxxxxxxxxxx> > Cc: gluster-users@xxxxxxxxxxx > Sent: Tuesday, August 29, 2017 12:35:05 AM > Subject: Re: GFID attir is missing after adding large amounts of data > > This is strange, a couple of questions: > > 1. What volume type is this? What tuning have you done? gluster v info > output would be helpful here. > > 2. How big are your bricks? > > 3. Can you write me a quick reproducer so I can try this in the lab? Is it > just a single multi TB file you are untarring or many? If you give me the > steps to repro, and I hit it, we can get a bug open. > > 4. Other than this are you seeing any other problems? What if you untar a > smaller file(s)? Can you read and write to the volume with say DD without > any problems? > > It sounds like you have some other issues affecting things here, there is no > reason why you shouldn't be able to untar and write multiple TBs of data to > gluster. Go ahead and answer those questions and I'll see what I can do to > help you out. > > -b > > ----- Original Message ----- > > From: "Christoph Schäbel" <christoph.schaebel@xxxxxxxxxxxx> > > To: gluster-users@xxxxxxxxxxx > > Sent: Monday, August 28, 2017 3:55:31 AM > > Subject: GFID attir is missing after adding large amounts > > of data > > > > Hi Cluster Community, > > > > we are seeing some problems when adding multiple terrabytes of data to a 2 > > node replicated GlusterFS installation. > > > > The version is 3.8.11 on CentOS 7. > > The machines are connected via 10Gbit LAN and are running 24/7. The OS is > > virtualized on VMWare. > > > > After a restart of node-1 we see that the log files are growing to multiple > > Gigabytes a day. > > > > Also there seem to be problems with the replication. > > The setup worked fine until sometime after we added the additional data > > (around 3 TB in size) to node-1. We added the data to a mountpoint via the > > client, not directly to the brick. > > What we did is add tar files via a client-mount and then untar them while > > in > > the client-mount folder. > > The brick (/mnt/brick1/gv0) is using the XFS filesystem. > > > > When checking the file attributes of one of the files mentioned in the > > brick > > logs, i can see that the gfid attribute is missing on node-1. On node-2 the > > file does not even exist. > > > > getfattr -m . -d -e hex > > mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet > > > > # file: > > mnt/brick1/gv0/.glusterfs/40/59/40598e46-9868-4d7c-b494-7b978e67370a/type=type1/part-r-00002-4846e211-c81d-4c08-bb5e-f22fa5a4b404.gz.parquet > > security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a756e6c6162656c65645f743a733000 > > > > We repeated this scenario a second time with a fresh setup and got the same > > results. > > > > Does anyone know what we are doing wrong ? > > > > Is there maybe a problem with glusterfs and tar ? > > > > > > Log excerpts: > > > > > > glustershd.log > > > > [2017-07-26 15:31:36.290908] I [MSGID: 108026] > > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: > > performing entry selfheal on fe5c42ac-5fda-47d4-8221-484c8d826c06 > > [2017-07-26 15:31:36.294289] W [MSGID: 114031] > > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote > > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No > > data available] > > [2017-07-26 15:31:36.298287] I [MSGID: 108026] > > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: > > performing entry selfheal on e31ae2ca-a3d2-4a27-a6ce-9aae24608141 > > [2017-07-26 15:31:36.300695] W [MSGID: 114031] > > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote > > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No > > data available] > > [2017-07-26 15:31:36.303626] I [MSGID: 108026] > > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: > > performing entry selfheal on 2cc9dafe-64d3-454a-a647-20deddfaebfe > > [2017-07-26 15:31:36.305763] W [MSGID: 114031] > > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote > > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No > > data available] > > [2017-07-26 15:31:36.308639] I [MSGID: 108026] > > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: > > performing entry selfheal on cbabf9ed-41be-4d08-9cdb-5734557ddbea > > [2017-07-26 15:31:36.310819] W [MSGID: 114031] > > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote > > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No > > data available] > > [2017-07-26 15:31:36.315057] I [MSGID: 108026] > > [afr-self-heal-entry.c:833:afr_selfheal_entry_do] 0-gv0-replicate-0: > > performing entry selfheal on 8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69 > > [2017-07-26 15:31:36.317196] W [MSGID: 114031] > > [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv0-client-1: remote > > operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [No > > data available] > > > > > > > > bricks/mnt-brick1-gv0.log > > > > 2017-07-26 15:31:36.287831] E [MSGID: 115050] > > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153546: LOOKUP > > <gfid:d99930df-6b47-4b55-9af3-c767afd6584c>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > (d99930df-6b47-4b55-9af3-c767afd6584c/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) > > ==> (No data available) [No data available] > > [2017-07-26 15:31:36.294202] E [MSGID: 113002] [posix.c:266:posix_lookup] > > 0-gv0-posix: buf->ia_gfid is null for > > /mnt/brick1/gv0/.glusterfs/e7/2d/e72d9005-b958-432b-b4a9-37aaadd9d2df/type=type1/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > [No data available] > > [2017-07-26 15:31:36.294235] E [MSGID: 115050] > > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153564: LOOKUP > > <gfid:fe5c42ac-5fda-47d4-8221-484c8d826c06>/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > (fe5c42ac-5fda-47d4-8221-484c8d826c06/part-r-00001-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) > > ==> (No data available) [No data available] > > [2017-07-26 15:31:36.300611] E [MSGID: 113002] [posix.c:266:posix_lookup] > > 0-gv0-posix: buf->ia_gfid is null for > > /mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > [No data available] > > [2017-07-26 15:31:36.300645] E [MSGID: 115050] > > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153582: LOOKUP > > <gfid:e31ae2ca-a3d2-4a27-a6ce-9aae24608141>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > (e31ae2ca-a3d2-4a27-a6ce-9aae24608141/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) > > ==> (No data available) [No data available] > > [2017-07-26 15:31:36.305671] E [MSGID: 113002] [posix.c:266:posix_lookup] > > 0-gv0-posix: buf->ia_gfid is null for > > /mnt/brick1/gv0/.glusterfs/33/d4/33d47146-bc30-49dd-ada8-475bb75435bf/type=type1/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > [No data available] > > [2017-07-26 15:31:36.305711] E [MSGID: 115050] > > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153600: LOOKUP > > <gfid:2cc9dafe-64d3-454a-a647-20deddfaebfe>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > (2cc9dafe-64d3-454a-a647-20deddfaebfe/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) > > ==> (No data available) [No data available] > > [2017-07-26 15:31:36.310735] E [MSGID: 113002] [posix.c:266:posix_lookup] > > 0-gv0-posix: buf->ia_gfid is null for > > /mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type2/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > [No data available] > > [2017-07-26 15:31:36.310767] E [MSGID: 115050] > > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153618: LOOKUP > > <gfid:cbabf9ed-41be-4d08-9cdb-5734557ddbea>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > (cbabf9ed-41be-4d08-9cdb-5734557ddbea/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) > > ==> (No data available) [No data available] > > [2017-07-26 15:31:36.317113] E [MSGID: 113002] [posix.c:266:posix_lookup] > > 0-gv0-posix: buf->ia_gfid is null for > > /mnt/brick1/gv0/.glusterfs/df/71/df715321-3078-47c8-bf23-dec47abe46d7/type=type3/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > [No data available] > > [2017-07-26 15:31:36.317146] E [MSGID: 115050] > > [server-rpc-fops.c:156:server_lookup_cbk] 0-gv0-server: 6153636: LOOKUP > > <gfid:8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69>/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet > > (8a3c1c16-8edf-40f0-b2ea-8e70c39e1a69/part-r-00002-becc67f0-1665-47b6-8566-fa0245f560ad.gz.parquet) > > ==> (No data available) [No data available] > > > > > > Regards, > > Christoph > > _______________________________________________ > > Gluster-users mailing list > > Gluster-users@xxxxxxxxxxx > > http://lists.gluster.org/mailman/listinfo/gluster-users > > > _______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx http://lists.gluster.org/mailman/listinfo/gluster-users
