Re: Hot Tier exceeding watermark-hi

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 

Hi David,

Just for your information, as a project, we are not currently taking any development focus on Tiering feature!

Please refer to email thread @ https://lists.gluster.org/pipermail//gluster-devel/2018-July/055017.html

It is recommended to use 'dmcache' on your disks to get best performance out of your backend instead! Also note that this may get retired in next upcoming release: https://review.gluster.org/21331

Hope this email will save you lot of time!

Regards,
Amar


On Sun, Sep 30, 2018 at 6:04 PM David Brown <dbccemtp@xxxxxxxxx> wrote:
Just found this in the tierd.log Not sure what it means or how to fix it tho, but I assume it may be the cause of my problem with files not being demoted from the hot tier.. 


[2018-09-30 12:25:56.438821] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> [Transport endpoint is not connected]
[2018-09-30 12:25:56.440940] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.444633] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.448347] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.451919] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test-march-3-bc-file-501 (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.456198] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //krishna (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.459702] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test1 (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.463164] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.466601] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.470129] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.473758] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.477237] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.480696] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.484354] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.487884] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:56.490973] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.494618] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.498059] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.501428] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //DBTest (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:56.504736] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //testCliff (cac8d0c8-2a10-4fa1-8b94-bf47981c01ff). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
The message "I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)" repeated 4 times between [2018-09-30 12:25:53.164873] and [2018-09-30 12:25:57.515127]
[2018-09-30 12:25:57.520933] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:57.521545] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:57.521854] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> [Transport endpoint is not connected]
[2018-09-30 12:25:57.524079] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.527993] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.531829] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.535529] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test-march-3-bc-file-501 (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.539852] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //krishna (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.543468] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test1 (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.547051] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.550694] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.554498] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.558086] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.561568] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.565094] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.568562] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.571974] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:57.574925] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.578639] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.582118] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:57.585547] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //DBTest (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
The message "E [MSGID: 109037] [tier.c:2532:tier_run] 0-FFPrimary-tier-dht: Demotion failed" repeated 3 times between [2018-09-30 12:25:54.323716] and [2018-09-30 12:25:57.592361]
[2018-09-30 12:25:58.002724] I [MSGID: 0] [dht-rebalance.c:4194:gf_defrag_total_file_size] 0-FFPrimary-tier-dht: local subvol: FFPrimary-cold-dht,cnt = 272288612352
[2018-09-30 12:25:58.018840] I [MSGID: 0] [dht-rebalance.c:4198:gf_defrag_total_file_size] 0-FFPrimary-tier-dht: Total size files = 272288612352
[2018-09-30 12:25:58.018854] I [dht-rebalance.c:4325:dht_file_counter_thread] 0-dht: tmp data size =272288612352
The message "E [MSGID: 109037] [tier.c:720:tier_lookup_file] 0-FFPrimary-tier-dht: Failed to lookup file 6bed3d1e-17b8-11e8-a5dc-902b3450f388
 [No such file or directory]" repeated 4 times between [2018-09-30 12:25:54.247352] and [2018-09-30 12:25:58.599151]
[2018-09-30 12:25:58.599162] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)
[2018-09-30 12:25:58.604808] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:58.605428] W [MSGID: 114031] [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:58.605750] E [MSGID: 114031] [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473> [Transport endpoint is not connected]
[2018-09-30 12:25:58.607865] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388 (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.611433] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.615059] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9 (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.618669] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test-march-3-bc-file-501 (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.622667] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //krishna (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.626029] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //test1 (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.629459] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316 (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.632994] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.636669] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.640155] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20 (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.643516] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.648787] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135 (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.652106] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.655577] I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:58.658482] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.661703] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8 (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
[2018-09-30 12:25:58.665051] W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]
The message "W [MSGID: 114031] [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8: remote operation failed. Path: //e8a6aba1-e5ce-4ded-b474-1c5bf49b1285 (edffea55-296b-4d1b-8114-b4f8dd10920a). Key: trusted.glusterfs.node-uuid [Transport endpoint is not connected]" repeated 3 times between [2018-09-30 12:25:55.406782] and [2018-09-30 12:25:58.674842]
[2018-09-30 12:25:58.675351] E [MSGID: 109037] [tier.c:2532:tier_run] 0-FFPrimary-tier-dht: Demotion failed
[root@Glus1 FFPrimary]#

On Sat, Sep 29, 2018 at 3:54 PM David Brown <dbccemtp@xxxxxxxxx> wrote:
4 hours later, no files have been demoted 

[root@Glus1 ~]# gluster volume status  FFPrimary detail
Status of volume: FFPrimary
Hot Bricks:
------------------------------------------------------------------------------
Brick                : Brick Glus3:/data/glusterfs/FFPrimary/brick3
TCP Port             : 49155
RDMA Port            : 0
Online               : Y
Pid                  : 24177
File System          : xfs
Device               : /dev/nvme0n1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 24.3GB
Total Disk Space     : 476.7GB
Inode Count          : 50877088
Free Inodes          : 50874696
------------------------------------------------------------------------------
Brick                : Brick Glus2:/data/glusterfs/FFPrimary/brick2
TCP Port             : 49155
RDMA Port            : 0
Online               : Y
Pid                  : 17994
File System          : xfs
Device               : /dev/nvme0n1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 15.5GB
Total Disk Space     : 476.7GB
Inode Count          : 32560288
Free Inodes          : 32557896
------------------------------------------------------------------------------
Brick                : Brick Glus1:/data/glusterfs/FFPrimary/brick1
TCP Port             : 49154
RDMA Port            : 0
Online               : Y
Pid                  : 23573
File System          : xfs
Device               : /dev/nvme0n1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 13.8GB
Total Disk Space     : 476.7GB
Inode Count          : 29027000
Free Inodes          : 29024515
Cold Bricks:
------------------------------------------------------------------------------
Brick                : Brick Glus1:/data/glusterfs/FFPrimary/brick5
TCP Port             : 49152
RDMA Port            : 0
Online               : Y
Pid                  : 23442
File System          : xfs
Device               : /dev/sdb1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 2.5TB
Total Disk Space     : 2.7TB
Inode Count          : 292971904
Free Inodes          : 292969488
------------------------------------------------------------------------------
Brick                : Brick Glus2:/data/glusterfs/FFPrimary/brick6
TCP Port             : 49153
RDMA Port            : 0
Online               : Y
Pid                  : 17856
File System          : xfs
Device               : /dev/sdb1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 2.5TB
Total Disk Space     : 2.7TB
Inode Count          : 292971904
Free Inodes          : 292969489
------------------------------------------------------------------------------
Brick                : Brick Glus3:/data/glusterfs/FFPrimary/brick7
TCP Port             : 49153
RDMA Port            : 0
Online               : Y
Pid                  : 24018
File System          : xfs
Device               : /dev/sdb1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 2.5TB
Total Disk Space     : 2.7TB
Inode Count          : 292971904
Free Inodes          : 292969488
------------------------------------------------------------------------------
Brick                : Brick Glus1:/data/glusterfs/FFPrimary/brick8
TCP Port             : 49153
RDMA Port            : 0
Online               : Y
Pid                  : 23518
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 2.7TB
Total Disk Space     : 2.7TB
Inode Count          : 292971904
Free Inodes          : 292969607
------------------------------------------------------------------------------
Brick                : Brick Glus2:/data/glusterfs/FFPrimary/brick9
TCP Port             : 49154
RDMA Port            : 0
Online               : Y
Pid                  : 17943
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 2.7TB
Total Disk Space     : 2.7TB
Inode Count          : 292971904
Free Inodes          : 292969607
------------------------------------------------------------------------------
Brick                : Brick Glus3:/data/glusterfs/FFPrimary/brick10
TCP Port             : 49154
RDMA Port            : 0
Online               : Y
Pid                  : 24108
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 2.7TB
Total Disk Space     : 2.7TB
Inode Count          : 292971904
Free Inodes          : 292969604

[root@Glus1 ~]# gluster volume info

Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
Options Reconfigured:
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

[root@Glus1 ~]# gluster volume tier FFPrimary status
Node                 Promoted files       Demoted files        Status               run time in h:m:s
---------            ---------            ---------            ---------            ---------
localhost            49                   0                    in progress          5159:15:50
Glus2 0                    0                    in progress          5159:15:50
Glus3 0                    2075                 in progress          5159:15:52
Tiering Migration Functionality: FFPrimary: success
[root@Glus1 ~]#



On Sat, Sep 29, 2018 at 11:56 AM David Brown <dbccemtp@xxxxxxxxx> wrote:
Thank you Hari, 

I have set:
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120

I will let you know if it makes a difference after some time. So far (10 minutes), nothing has changed. 
I would agree with you, that by looking at the result of 'gluster volume tier FFPrimary status' it would seem that demoting is happening. However, for the last 24hrs, nothing has changed in the tier status report except the time. Could it be stuck? How would I know? Is there a way to restart it without restarting the cluster?





On Sat, Sep 29, 2018 at 11:08 AM Hari Gowtham <hgowtham@xxxxxxxxxx> wrote:
Hi,

I can see that the demotion is happening from the status provided by you.
Do verify it.
I would recommend you to change the cluster.tier-demote-frequency to 120
and cluster.tier-promote-frequency to 1800 to increase the demotions until the
hot tier is emptied to a certain extent. Later you can use the values
existing now.
On Sat, Sep 29, 2018 at 5:39 PM David Brown <dbccemtp@xxxxxxxxx> wrote:
>
> Hey Everyone,
>
> I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD cold tier.
> I recently ran into some problems when the hot tier became full with df-h showing 100%.
>
> I did not have a watermark-hi set, but it is my understanding that 90% is the default. In an attempt to get the cluster to demote some files, I set cluster.watermark-hi: 80 but it is still not demoting.
>
>
> [root@Glus1 ~]# gluster volume info
>
> Volume Name: FFPrimary
> Type: Tier
> Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 9
> Transport-type: tcp
> Hot Tier :
> Hot Tier Type : Replicate
> Number of Bricks: 1 x 3 = 3
> Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
> Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
> Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
> Cold Tier:
> Cold Tier Type : Distributed-Replicate
> Number of Bricks: 2 x 3 = 6
> Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
> Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
> Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
> Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
> Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
> Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
> Options Reconfigured:
> cluster.tier-promote-frequency: 120
> cluster.tier-demote-frequency: 1800
> cluster.watermark-low: 60
> cluster.watermark-hi: 80
> performance.flush-behind: on
> performance.cache-max-file-size: 128MB
> performance.cache-size: 25GB
> diagnostics.count-fop-hits: off
> diagnostics.latency-measurement: off
> cluster.tier-mode: cache
> features.ctr-enabled: on
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
> [root@Glus1 ~]# gluster volume tier FFPrimary status
> Node                 Promoted files       Demoted files        Status               run time in h:m:s
> ---------            ---------            ---------            ---------            ---------
> localhost            49                   0                    in progress          5151:30:45
> Glus2 0                    0                    in progress          5151:30:45
> Glus3 0                    2075                 in progress          5151:30:47
> Tiering Migration Functionality: FFPrimary: success
> [root@Glus1 ~]#
>
> What can cause GlusterFS to stop demoting files and allow it to completely fill the Hot Tier?
>
> Thank you!
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users@xxxxxxxxxxx
> https://lists.gluster.org/mailman/listinfo/gluster-users



--
Regards,
Hari Gowtham.
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users


--
Amar Tumballi (amarts)
_______________________________________________
Gluster-users mailing list
Gluster-users@xxxxxxxxxxx
https://lists.gluster.org/mailman/listinfo/gluster-users
[Index of Archives]     [Gluster Development]     [Linux Filesytems Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux