Hi all,
we encountered some strange behavior when using storage classes for S3
protocol. Some objects end up in a different pool than we would expect.
Below is a list of commands used for create an account with replicated
storage class, upload some files to the bucket and checked that they
were uploaded to the correct location. Most of the files were uploaded
to the correct pool, but some were uploaded to the erasure-code pool.
Specifically, certain multipart objects (but they have zero size) and
stub. The multipart objects probably wouldn't mind so much, but stub is
important because we tried to delete it and without it the operations
will not go through.
Are there any errors in our setting (see zonegroup and zone setttings
below)?
Can someone please send us a setup that is functional so we can build it
accordingly?
We have a Pacific version.
=== Create the account ===
radosgw-admin user create --tenant vo_du_test --uid
s3_test_tomash_ec_replicated --display-name s3_test_tomash_ec_replicated
--storage-class "" --placement-id "EC_replicated" --tags ""
radosgw-admin user info --tenant vo_du_test --uid
s3_test_tomash_ec_replicated | jq
'.default_placement,.default_storage_class,.placement_tags'
"EC_replicated"
""
[]
=== Create the bucket with replicated storage class ===
s3cmd mb s3://b7-user-ec-stcl-replicated --storage-class=replicated
Bucket 's3://b7-user-ec-stcl-replicated/' created
s3cmd -c ~/.s3cfg/.s3cfg_clx_vo_du_test_s3_test_tomash_ec_replicated put
20MiB.dat s3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat
--storage-class=replicated
upload: '20MiB.dat' ->
's3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat' [part 1 of 2, 15MB]
[1 of 1]
15728640 of 15728640 100% in 0s 20.86 MB/s done
upload: '20MiB.dat' ->
's3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat' [part 2 of 2, 5MB]
[1 of 1]
5242880 of 5242880 100% in 0s 18.34 MB/s done
s3cmd -c ~/.s3cfg/.s3cfg_clx_vo_du_test_s3_test_tomash_ec_replicated
info s3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat
s3://b7-user-ec-stcl-replicated/20MiB_b7_repl.dat (object):
File size: 20971520
Storage: replicated
...
ACL: s3_test_tomash_ec_replicated: FULL_CONTROL
...
=== Checking where objects are stored ===
rados -p storage-clx.rgw.replicated.data ls | grep _b7_repl
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_1
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_3
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_2
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2_1
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1
ceph_mon:# rados -p storage-clx.rgw.EC.data ls | grep _b7_repl
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1_20MiB_b7_repl.dat
ceph_mon:# rados stat -p storage-clx.rgw.EC.data
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1_20MiB_b7_repl.dat
storage-clx.rgw.EC.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1_20MiB_b7_repl.dat
mtime 2023-01-24T12:46:48.000000+0100, size 0
ceph_mon:# rados stat -p storage-clx.rgw.EC.data
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1
storage-clx.rgw.EC.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1
mtime 2023-01-24T12:46:48.000000+0100, size 0
ceph_mon:# rados stat -p storage-clx.rgw.EC.data
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
storage-clx.rgw.EC.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
mtime 2023-01-24T12:46:48.000000+0100, size 0
ceph_mon:# rados stat -p storage-clx.rgw.replicated.data
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1
mtime 2023-01-24T12:46:47.000000+0100, size 4194304
rados stat -p storage-clx.rgw.replicated.data
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_1
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_1
mtime 2023-01-24T12:46:48.000000+0100, size 4194304
ceph_mon:# rados stat -p storage-clx.rgw.replicated.data
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_2
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_2
mtime 2023-01-24T12:46:48.000000+0100, size 4194304
ceph_mon:# rados stat -p storage-clx.rgw.replicated.data
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_3
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.1_3
mtime 2023-01-24T12:46:48.000000+0100, size 3145728
ceph_mon:# rados stat -p storage-clx.rgw.replicated.data
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__multipart_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2
mtime 2023-01-24T12:46:48.000000+0100, size 4194304
ceph_mon:# rados stat -p storage-clx.rgw.replicated.data
68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2_1
storage-clx.rgw.replicated.data/68618cb9-d006-4b86-b8f0-3722d952fd33.69137019.1__shadow_20MiB_b7_repl.dat.2~AooZ4xMGNV7oOzIQbNdo9fV5qw3vEr4.2_1
mtime 2023-01-24T12:46:48.000000+0100, size 1048576
=== Zonegroup setting ===
radosgw-admin zonegroup get
{
"id": "kda7dadb-7bbc-4197-b070-147f76e74717",
"name": "storage",
"api_name": "storage",
"is_master": "true",
"endpoints": [
"https://s3.clx.domain.cz";
],
"hostnames": [],
"hostnames_s3website": [],
"master_zone": "69922cb9-d006-4b86-b8f0-3722d952fd33",
"zones": [
{
"id": "24654cb9-d006-4b86-b8f0-3722d952fd33",
"name": "storage-clx",
"endpoints": [
"https://s3.clx.domain.cz";
],
"log_meta": "false",
"log_data": "false",
"bucket_index_max_shards": 11,
"read_only": "false",
"tier_type": "",
"sync_from_all": "true",
"sync_from": [],
"redirect_zone": ""
}
],
"placement_targets": [
{
"name": "EC_replicated",
"tags": [],
"storage_classes": [
"EC",
"STANDARD",
"replicated"
]
},
{
"name": "default-placement",
"tags": [],
"storage_classes": [
"STANDARD"
]
},
{
"name": "replicated",
"tags": [],
"storage_classes": [
"STANDARD"
]
}
],
"default_placement": "default-placement",
"realm_id": "87a9217-7e8d-4a1b-b508-eb634cc55647",
"sync_policy": {
"groups": []
}
}
=== Zone setting ===
radosgw-admin zone get
{
"id": "24654cb9-d006-4b86-b8f0-3722d952fd33",
"name": "storage-clx",
"domain_root": "storage-clx.rgw.meta:root",
"control_pool": "storage-clx.rgw.control",
"gc_pool": "storage-clx.rgw.log:gc",
"lc_pool": "storage-clx.rgw.log:lc",
"log_pool": "storage-clx.rgw.log",
"intent_log_pool": "storage-clx.rgw.log:intent",
"usage_log_pool": "storage-clx.rgw.log:usage",
"roles_pool": "storage-clx.rgw.meta:roles",
"reshard_pool": "storage-clx.rgw.log:reshard",
"user_keys_pool": "storage-clx.rgw.meta:users.keys",
"user_email_pool": "storage-clx.rgw.meta:users.email",
"user_swift_pool": "storage-clx.rgw.meta:users.swift",
"user_uid_pool": "storage-clx.rgw.meta:users.uid",
"otp_pool": "storage-clx.rgw.otp",
"system_key": {
"access_key": "",
"secret_key": ""
},
"placement_pools": [
{
"key": "EC_replicated",
"val": {
"index_pool": "storage-clx.rgw.EC.index",
"storage_classes": {
"STANDARD": {
"data_pool": "storage-clx.rgw.EC.data"
},
"replicated": {
"data_pool": "storage-clx.rgw.replicated.data"
}
},
"data_extra_pool": "storage-clx.rgw.EC.non-ec",
"index_type": 0
}
},
{
"key": "default-placement",
"val": {
"index_pool": "storage-clx.rgw.EC.index",
"storage_classes": {
"STANDARD": {
"data_pool": "storage-clx.rgw.EC.data"
}
},
"data_extra_pool": "storage-clx.rgw.EC.non-ec",
"index_type": 0
}
},
{
"key": "replicated",
"val": {
"index_pool": "storage-clx.rgw.replicated.index",
"storage_classes": {
"STANDARD": {
"data_pool": "storage-clx.rgw.replicated.data"
}
},
"data_extra_pool": "",
"index_type": 0
}
}
],
"realm_id": "87a9217-7e8d-4a1b-b508-eb634cc55647",
"notif_pool": "storage-clx.rgw.log:notif"
}
Thank you
Regards,
Michal Strnad
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
