Hi Frederic
For your point 3, the default_storage_class from the user info is apparently ignored.
Setting it on Nautilus 14.2.15 had no impact and objects were still stored with STANDARD.
Another issue is that some clients like s3cmd are per default explicitly using STANDARD.
And even after modifying s3cmd code to avoid sending x-amz-storage-class:STANDARD,
the default_storage_class from the user info was still ignored.
So basically you need all your clients to set the correct new storage class for new PUT's.
The second issue is that all objects in the new storage class will have a small head object in the pool of the STANDARD class.
I am unsure if they are important or not...
I also tested a strategy using a new placement target, where you need 3 new pools for data, index and data_extra.
In this case no head objects are stored in the former pool, you get a clean and complete pool separation.
The new problem is that you can only set the placement target (aka region_name:placement_name) during bucket creation.
If you change it later (by replacing the bucket instance and changing .data.bucket_info.placement_rule) it works for new objects but you loose access to former ones...
Maybe this strategy could be used to migrate each bucket to a new one by using something like "mc mirror --watch" or similar. Then delete the old bucket and rename the new one.
A third strategy would involve cache tiering:
#overview:
#to migrate from a replicated pool:
#1) flush all objects from $cachepool to $storagepool:
cachepool=srcpool
storagepool=dstpool
#to migrate from an EC pool:
#1) create a temporary replicated pool
#2) promote all objects from $storagepool to $cachepool:
storagepool=srcpool
cachepool=tmppool
#3) then flush all objects from $cachepool to $storagepool:
cachepool=tmppool
storagepool=dstpool
This "seems" to work. I can share more details if you want.
We are going with the fourth strategy, which is to use multi-site to sync all buckets, then do a fail over.
Please share your experience and results, it is certainly interesting. ;-)
I think many are facing the same challenges with pool migration.
Cheers
Francois
https://lists.ceph.io/hyperkitty/list/ceph-users@xxxxxxx/thread/UZLA7XU2RKUC5NBZALJGOIZIWLUGELFO/
--
EveryWare AG
François Scheurer
Senior Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: francois.scheurer@xxxxxxxxxxxx
web: http://www.everyware.ch
________________________________
From: Irek Fasikhov <malmyzh@xxxxxxxxx>
Sent: Wednesday, January 26, 2022 8:14 PM
To: Frédéric Nass
Cc: Ceph Users
Subject: [ceph-users] Re: Moving all s3 objects from an ec pool to a replicated pool using storage classes.
Hi.
Basic logic:
1.bucket policy transition
2.radosgw-admin gc process --include-all
3.1.rados ls -p pool | grep <bucket_marker_id> >bucket_objects.txt
3.2.rados listxattr -p pool objname | xargs -L1 echo rados getattr -p pool
objname >> objname.txt
3.3.rados create -p pool objname
3.4.cat objname.txt | xargs -L1 echo rados setattr -p pool objname attr
value
4.radosgw-admin metadata get bucket.instance.<bucket>.<marker_id> | tee
bucket.json bucket_backup.json
5.change placement_rule to "default-placement/NEW_CLASS",
6.radosgw-admin metadata rm bucket.instance.<bucket>.<marker_id>
7.radosgw-admin metadata put bucket.instance.<bucket>.<marker_id> <
bucker.json
вт, 25 янв. 2022 г. в 21:31, Frédéric Nass <frederic.nass@xxxxxxxxxxxxxxxx>:
>
> Le 25/01/2022 à 18:28, Casey Bodley a écrit :
> > On Tue, Jan 25, 2022 at 11:59 AM Frédéric Nass
> > <frederic.nass@xxxxxxxxxxxxxxxx> wrote:
> >>
> >> Le 25/01/2022 à 14:48, Casey Bodley a écrit :
> >>> On Tue, Jan 25, 2022 at 4:49 AM Frédéric Nass
> >>> <frederic.nass@xxxxxxxxxxxxxxxx> wrote:
> >>>> Hello,
> >>>>
> >>>> I've just heard about storage classes and imagined how we could use
> them
> >>>> to migrate all S3 objects within a placement pool from an ec pool to a
> >>>> replicated pool (or vice-versa) for data resiliency reasons, not to
> save
> >>>> space.
> >>>>
> >>>> It looks possible since ;
> >>>>
> >>>> 1. data pools are associated to storage classes in a placement pool
> >>>> 2. bucket lifecycle policies can take care of moving data from a
> storage
> >>>> class to another
> >>>> 3. we can set a user's default_storage_class to have all new objects
> >>>> written by this user reach the new storage class / data pool.
> >>>> 4. after all objects have been transitioned to the new storage class,
> we
> >>>> can delete the old storage class, rename the new storage class to
> >>>> STANDARD so that it's been used by default and unset any user's
> >>>> default_storage_class setting.
> >>> i don't think renaming the storage class will work the way you're
> >>> hoping. this storage class string is stored in each object and used to
> >>> locate its data, so renaming it could render the transitioned objects
> >>> unreadable
> >> Hello Casey,
> >>
> >> Thanks for pointing that out.
> >>
> >> Do you believe this scenario would work if stopped at step 3.? (keeping
> >> default_storage_class set on users's profiles and not renaming the new
> >> storage class to STANDARD. Could we delete the STANDARD storage class
> >> btw since we would not use it anymore?).
> >>
> >> If there is no way to define the default storage class of a placement
> >> pool without naming it STANDARD could we imaging transitioning all
> >> objects again by:
> >>
> >> 4. deleting the storage class named STANDARD
> >> 5. creating a new one named STANDARD (using a ceph pool of the same data
> >> placement scheme than the one used by the temporary storage class
> >> created above)
> > instead of deleting/recreating STANDARD, you could probably just
> > modify it's data pool. only do this once you're certain that there are
> > no more objects in the old data pool. you might need to wait for
> > garbage collection to clean up the tail objects there too (or force it
> > with 'radosgw-admin gc process --include-all')
>
> Interesting scenario. So in the end we'd have objects named after both
> storage classes in the same ceph pool, the old ones named after the new
> storage class name and the new ones being written after the STANDARD
> storage class, right?
>
> >
> >> 6. transitioning all objects again to the new STANDARD storage class.
> >> Then delete the temporary storage class.
> > i think this step 6 would run into the
> > https://tracker.ceph.com/issues/50974 that Konstantin shared - if the
> > two storage classes have the same pool name, the transition doesn't
> > actually take effect. you might consider leaving this 'temporary'
> > storage class around, but pointing the defaults back at STANDARD
>
> Well, in step 6., I'd thought about using another new pool for the
> recreated STANDARD storage class (to avoid the issue shared by
> Konstantin , thanks to him btw) and move all objects to this new pool
> again in a new global transition.
>
> But, I understand you'd recommend avoiding deleting/recreating STANDARD
> and just modify the STANDARD data pool after GC execution, am I right?
>
> Frédéric.
>
> >
> >> ?
> >>
> >> Best regards,
> >>
> >> Frédéric.
> >>
> >>>> Would that work?
> >>>>
> >>>> Anyone tried this with success yet?
> >>>>
> >>>> Best regards,
> >>>>
> >>>> Frédéric.
> >>>>
> >>>> --
> >>>> Cordialement,
> >>>>
> >>>> Frédéric Nass
> >>>> Direction du Numérique
> >>>> Sous-direction Infrastructures et Services
> >>>>
> >>>> Tél : 03.72.74.11.35
> >>>>
> >>>> _______________________________________________
> >>>> ceph-users mailing list -- ceph-users@xxxxxxx
> >>>> To unsubscribe send an email to ceph-users-leave@xxxxxxx
> _______________________________________________
> ceph-users mailing list -- ceph-users@xxxxxxx
> To unsubscribe send an email to ceph-users-leave@xxxxxxx
>
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
_______________________________________________
ceph-users mailing list -- ceph-users@xxxxxxx
To unsubscribe send an email to ceph-users-leave@xxxxxxx
