Just found this in the tierd.log Not sure what it means or how to fix it tho, but I assume it may be the cause of my problem with files not being demoted from the hot tier..
[2018-09-30 12:25:56.438821] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> [Transport endpoint is not connected]
[2018-09-30 12:25:56.440940] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.444633] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.448347] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.451919] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test-march-3-bc-file-501 (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.456198] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //krishna (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.459702] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test1 (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.463164] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.466601] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.470129] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.473758] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.477237] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.480696] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.484354] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.487884] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:56.490973] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.494618] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.498059] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.501428] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //DBTest (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.504736] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //testCliff (cac8d0c8-2a10-4fa1-8b94-bf47981c01ff). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
The message "I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)" repeated 4 times between [2018-09-30 12:25:53.164873] and [2018-09-30 12:25:57.515127]
[2018-09-30 12:25:57.520933] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:57.521545] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:57.521854] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> [Transport endpoint is not connected]
[2018-09-30 12:25:57.524079] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.527993] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.531829] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.535529] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test-march-3-bc-file-501 (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.539852] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //krishna (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.543468] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test1 (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.547051] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.550694] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.554498] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.558086] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.561568] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.565094] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.568562] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.571974] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:57.574925] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.578639] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.582118] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.585547] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //DBTest (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
The message "E [MSGID: 109037] [tier.c:2532:tier_run] 0-FFPrimary-tier-dht: Demotion failed" repeated 3 times between [2018-09-30 12:25:54.323716] and [2018-09-30 12:25:57.592361]
[2018-09-30 12:25:58.002724] I [MSGID: 0] [dht-rebalance.c:4194:gf_defrag_total_file_size] 0-FFPrimary-tier-dht: local subvol: FFPrimary-cold-dht,cnt = 272288612352
[2018-09-30 12:25:58.018840] I [MSGID: 0] [dht-rebalance.c:4198:gf_defrag_total_file_size] 0-FFPrimary-tier-dht: Total size files = 272288612352
[2018-09-30 12:25:58.018854] I [dht-rebalance.c:4325:dht_file_counter_thread] 0-dht: tmp data size =272288612352
The message "E [MSGID: 109037] [tier.c:720:tier_lookup_file] 0-FFPrimary-tier-dht: Failed to lookup file 6bed3d1e-17b8-11e8-a5dc-902b3450f388
[No such file or directory]" repeated 4 times between [2018-09-30 12:25:54.247352] and [2018-09-30 12:25:58.599151]
[2018-09-30 12:25:58.599162] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)
[2018-09-30 12:25:58.604808] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:58.605428] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:58.605750] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> [Transport endpoint is not connected]
[2018-09-30 12:25:58.607865] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.611433] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.615059] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.618669] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test-march-3-bc-file-501 (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.622667] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //krishna (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.626029] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test1 (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.629459] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.632994] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.636669] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.640155] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.643516] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.648787] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.652106] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.655577] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:58.658482] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.661703] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.665051] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
The message "W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e8a6aba1-e5ce-4ded-b474-1c5bf49b1285 (edffea55-296b-4d1b-8114-b4f8dd10920a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]" repeated 3 times between [2018-09-30 12:25:55.406782] and [2018-09-30 12:25:58.674842]
[2018-09-30 12:25:58.675351] E [MSGID: 109037] [tier.c:2532:tier_run] 0-FFPrimary-tier-dht: Demotion failed
[root@Glus1 FFPrimary]#
On Sat, Sep 29, 2018 at 3:54 PM David Brown <dbccemtp@xxxxxxxxx> wrote:
4 hours later, no files have been demoted[root@Glus1 ~]# gluster volume status FFPrimary detailStatus of volume: FFPrimaryHot Bricks:------------------------------------------------------------------------------Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3TCP Port : 49155RDMA Port : 0Online : YPid : 24177File System : xfsDevice : /dev/nvme0n1Mount Options : rw,seclabel,relatime,attr2,inode64,noquotaInode Size : 512Disk Space Free : 24.3GBTotal Disk Space : 476.7GBInode Count : 50877088Free Inodes : 50874696------------------------------------------------------------------------------Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2TCP Port : 49155RDMA Port : 0Online : YPid : 17994File System : xfsDevice : /dev/nvme0n1Mount Options : rw,seclabel,relatime,attr2,inode64,noquotaInode Size : 512Disk Space Free : 15.5GBTotal Disk Space : 476.7GBInode Count : 32560288Free Inodes : 32557896------------------------------------------------------------------------------Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1TCP Port : 49154RDMA Port : 0Online : YPid : 23573File System : xfsDevice : /dev/nvme0n1Mount Options : rw,seclabel,relatime,attr2,inode64,noquotaInode Size : 512Disk Space Free : 13.8GBTotal Disk Space : 476.7GBInode Count : 29027000Free Inodes : 29024515Cold Bricks:------------------------------------------------------------------------------Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5TCP Port : 49152RDMA Port : 0Online : YPid : 23442File System : xfsDevice : /dev/sdb1Mount Options : rw,seclabel,relatime,attr2,inode64,noquotaInode Size : 512Disk Space Free : 2.5TBTotal Disk Space : 2.7TBInode Count : 292971904Free Inodes : 292969488------------------------------------------------------------------------------Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6TCP Port : 49153RDMA Port : 0Online : YPid : 17856File System : xfsDevice : /dev/sdb1Mount Options : rw,seclabel,relatime,attr2,inode64,noquotaInode Size : 512Disk Space Free : 2.5TBTotal Disk Space : 2.7TBInode Count : 292971904Free Inodes : 292969489------------------------------------------------------------------------------Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7TCP Port : 49153RDMA Port : 0Online : YPid : 24018File System : xfsDevice : /dev/sdb1Mount Options : rw,seclabel,relatime,attr2,inode64,noquotaInode Size : 512Disk Space Free : 2.5TBTotal Disk Space : 2.7TBInode Count : 292971904Free Inodes : 292969488------------------------------------------------------------------------------Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8TCP Port : 49153RDMA Port : 0Online : YPid : 23518File System : xfsDevice : /dev/sdc1Mount Options : rw,seclabel,relatime,attr2,inode64,noquotaInode Size : 512Disk Space Free : 2.7TBTotal Disk Space : 2.7TBInode Count : 292971904Free Inodes : 292969607------------------------------------------------------------------------------Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9TCP Port : 49154RDMA Port : 0Online : YPid : 17943File System : xfsDevice : /dev/sdc1Mount Options : rw,seclabel,relatime,attr2,inode64,noquotaInode Size : 512Disk Space Free : 2.7TBTotal Disk Space : 2.7TBInode Count : 292971904Free Inodes : 292969607------------------------------------------------------------------------------Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10TCP Port : 49154RDMA Port : 0Online : YPid : 24108File System : xfsDevice : /dev/sdc1Mount Options : rw,seclabel,relatime,attr2,inode64,noquotaInode Size : 512Disk Space Free : 2.7TBTotal Disk Space : 2.7TBInode Count : 292971904Free Inodes : 292969604[root@Glus1 ~]# gluster volume infoVolume Name: FFPrimaryType: TierVolume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61eStatus: StartedSnapshot Count: 0Number of Bricks: 9Transport-type: tcpHot Tier :Hot Tier Type : ReplicateNumber of Bricks: 1 x 3 = 3Brick1: Glus3:/data/glusterfs/FFPrimary/brick3Brick2: Glus2:/data/glusterfs/FFPrimary/brick2Brick3: Glus1:/data/glusterfs/FFPrimary/brick1Cold Tier:Cold Tier Type : Distributed-ReplicateNumber of Bricks: 2 x 3 = 6Brick4: Glus1:/data/glusterfs/FFPrimary/brick5Brick5: Glus2:/data/glusterfs/FFPrimary/brick6Brick6: Glus3:/data/glusterfs/FFPrimary/brick7Brick7: Glus1:/data/glusterfs/FFPrimary/brick8Brick8: Glus2:/data/glusterfs/FFPrimary/brick9Brick9: Glus3:/data/glusterfs/FFPrimary/brick10Options Reconfigured:cluster.tier-promote-frequency: 1800cluster.tier-demote-frequency: 120cluster.watermark-low: 60cluster.watermark-hi: 80performance.flush-behind: onperformance.cache-max-file-size: 128MBperformance.cache-size: 25GBdiagnostics.count-fop-hits: offdiagnostics.latency-measurement: offcluster.tier-mode: cachefeatures.ctr-enabled: ontransport.address-family: inetnfs.disable: onperformance.client-io-threads: off[root@Glus1 ~]# gluster volume tier FFPrimary statusNode Promoted files Demoted files Status run time in h:m:s--------- --------- --------- --------- ---------localhost 49 0 in progress 5159:15:50Glus2 0 0 in progress 5159:15:50Glus3 0 2075 in progress 5159:15:52Tiering Migration Functionality: FFPrimary: success[root@Glus1 ~]#On Sat, Sep 29, 2018 at 11:56 AM David Brown <dbccemtp@xxxxxxxxx> wrote:Thank you Hari,I have set:cluster.tier-promote-frequency: 1800cluster.tier-demote-frequency: 120I will let you know if it makes a difference after some time. So far (10 minutes), nothing has changed.I would agree with you, that by looking at the result of 'gluster volume tier FFPrimary status' it would seem that demoting is happening. However, for the last 24hrs, nothing has changed in the tier status report except the time. Could it be stuck? How would I know? Is there a way to restart it without restarting the cluster?On Sat, Sep 29, 2018 at 11:08 AM Hari Gowtham <hgowtham@xxxxxxxxxx> wrote:Hi,
I can see that the demotion is happening from the status provided by you.
Do verify it.
I would recommend you to change the cluster.tier-demote-frequency to 120
and cluster.tier-promote-frequency to 1800 to increase the demotions until the
hot tier is emptied to a certain extent. Later you can use the values
existing now.
On Sat, Sep 29, 2018 at 5:39 PM David Brown <dbccemtp@xxxxxxxxx> wrote:
>
> Hey Everyone,
>
> I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD cold tier.
> I recently ran into some problems when the hot tier became full with df-h showing 100%.
>
> I did not have a watermark-hi set, but it is my understanding that 90% is the default. In an attempt to get the cluster to demote some files, I set cluster.watermark-hi: 80 but it is still not demoting.
>
>
> [root@Glus1 ~]# gluster volume info
>
> Volume Name: FFPrimary
> Type: Tier
> Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 9
> Transport-type: tcp
> Hot Tier :
> Hot Tier Type : Replicate
> Number of Bricks: 1 x 3 = 3
> Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
> Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
> Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
> Cold Tier:
> Cold Tier Type : Distributed-Replicate
> Number of Bricks: 2 x 3 = 6
> Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
> Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
> Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
> Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
> Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
> Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
> Options Reconfigured:
> cluster.tier-promote-frequency: 120
> cluster.tier-demote-frequency: 1800
> cluster.watermark-low: 60
> cluster.watermark-hi: 80
> performance.flush-behind: on
> performance.cache-max-file-size: 128MB
> performance.cache-size: 25GB
> diagnostics.count-fop-hits: off
> diagnostics.latency-measurement: off
> cluster.tier-mode: cache
> features.ctr-enabled: on
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
> [root@Glus1 ~]# gluster volume tier FFPrimary status
> Node Promoted files Demoted files Status run time in h:m:s
> --------- --------- --------- --------- ---------
> localhost 49 0 in progress 5151:30:45
> Glus2 0 0 in progress 5151:30:45
> Glus3 0 2075 in progress 5151:30:47
> Tiering Migration Functionality: FFPrimary: success
> [root@Glus1 ~]#
>
> What can cause GlusterFS to stop demoting files and allow it to completely fill the Hot Tier?
>
> Thank you!
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> https://lists.gluster.org/mailman/listinfo/gluster-users
--
Regards,
Hari Gowtham.
_______________________________________________ Gluster-users mailing list Gluster-users@xxxxxxxxxxx https://lists.gluster.org/mailman/listinfo/gluster-users
