Re: RGW lifecycle not expiring objects

[Graham Allan <gta@xxxxxxx>]

I upgraded my test cluster to Luminous 12.1.0, and a separate problem 
made me realize a possible cause for the lifecycle failure.

After the upgrade, radosgw spammed its logfile with "failed to list 
reshard log entries" errors. I realized that the radosgw user only had 
authorization to access specific osd pools - and not the newly-created 
"default.rgw.reshard" pool.

Which made me also realize, it had no access to the new-for-kraken 
"default.rgw.lc" pool. That may explain the failure to process lifecycle 
below...

The behavior has now changed. Listing lifecycle on my test bucket gives 
different output:

% aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \
    get-bucket-lifecycle-configuration --bucket=testgta
{
    "Rules": [
        {
            "Status": "Enabled",
            "Prefix": "",
            "Expiration": {
                "ExpiredObjectDeleteMarker": true
            },
            "ID": "test"
        }
    ]
}

Editing this lifecycle to add back the now-missing "Days" parameter 
results in a NotImplemented error:

% aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \
    put-bucket-lifecycle-configuration --bucket=testgta \
    --lifecycle-configuration  file://lifecycle.json

An error occurred (NotImplemented) when calling the 
PutBucketLifecycleConfiguration operation: Unknown

and the lifecycle is not updated.

So, I think I understand why it may not have worked before, but the 
goalposts seem to have changed to a new problem.

Would appreciate any ideas...

Graham

On 06/08/2017 11:54 AM, Graham Allan wrote:
> Sorry I didn't get to reply until now. The thing is I believe I *do* 
> have a lifecycle configured on at least one bucket. As noted in that 
> issue, I get an error returned when trying to set the lifecycle, but it 
> does appear to get stored:
>
> % aws --endpoint-url https://xxx.xxx.xxx.xxx s3api \
>    get-bucket-lifecycle-configuration --bucket=testgta
> {
>      "Rules": [
>          {
>              "Status": "Enabled",
>              "Prefix": "",
>              "Expiration": {
>                  "Days": 3
>              },
>              "ID": "test"
>          }
>      ]
> }
>
> Does that look valid? Is there some other configuration which is needed 
> besides this? The rgw config values for lifecycle look as expected.
>
> Thanks,
>
> Graham
>
> On 06/06/2017 01:43 PM, Ben Hines wrote:
> > If you have nothing listed in 'lc list', you probably need to add a 
> > lifecycle configuration using the S3 API. It's not automatic and has 
> > to be added per-bucket.
> >
> >
> > Here's some sample code for doing so: 
> > http://tracker.ceph.com/issues/19587
> >
> > -Ben
> >
> > On Tue, Jun 6, 2017 at 9:07 AM, Graham Allan <gta@xxxxxxx 
> > <mailto:gta@xxxxxxx>> wrote:
> >
> >     I still haven't seen anything get expired from our kraken (11.2.0)
> >     system.
> >
> >     When I run "radosgw-admin lc list" I get no output, besides debug
> >     output (I have "debug rgw = 10" at present):
> >
> >     # radosgw-admin lc list
> >     2017-06-06 10:57:49.319576 7f2b26ffd700  2
> >     RGWDataChangesLog::ChangesRenewThread: start
> >     2017-06-06 10:57:49.350646 7f2b49558c80 10 Cannot find current
> >     period zone using local zone
> >     2017-06-06 10:57:49.379065 7f2b49558c80  2 all 8 watchers are set,
> >     enabling cache
> >     []
> >     2017-06-06 10:57:49.399538 7f2b49558c80  2 removed watcher,
> >     disabling cache
> >
> >     Unclear to me whether the debug message about "Cannot find current
> >     period zone using local zone" is related or indicates a problem.
> >
> >     Currently all the lc config is more or less default, eg a few values:
> >
> >     # ceph --show-config|grep rgw_|grep lifecycle
> >     rgw_lifecycle_enabled = true
> >     rgw_lifecycle_thread = 1
> >     rgw_lifecycle_work_time = 00:00-06:00
> >
> >
> >     Graham
> >
> >     On 06/05/2017 01:07 PM, Ben Hines wrote:
> >
> >         FWIW lifecycle is working for us. I did have to research to find
> >         the appropriate lc config file settings, the documentation for
> >         which is found in a git pull request (waiting for another
> >         release?) rather than on the Ceph docs site.
> >         https://github.com/ceph/ceph/pull/13990
> >         <https://github.com/ceph/ceph/pull/13990>
> >
> >
> >         Try these:
> >         debug rgw = 20
> >         rgw lifecycle work time = 00:01-23:59
> >
> >
> >         and see if you have lifecycles listed when you run:
> >
> >
> >         radosgw-admin lc list
> >
> >
> >         2017-06-05 10:58:00.473957 7f3429f77c80  0 System already 
> > converted
> >         [
> >               {
> >                   "bucket": ":bentest:default.653959.6",
> >                   "status": "COMPLETE"
> >               },
> >               {
> >                   "bucket": ":<redacted>:default.24713983.1",
> >                   "status": "PROCESSING"
> >               },
> >               {
> >                   "bucket": ":<redacted>:default.24713983.2",
> >                   "status": "PROCESSING"
> >               },
> >
> >         ....
> >
> >
> >         At 10 loglevel, the lifecycle processor logs 'DELETED' each time
> >         it deletes something:
> >         https://github.com/ceph/ceph/blob/master/src/rgw/rgw_lc.cc#L388
> >         <https://github.com/ceph/ceph/blob/master/src/rgw/rgw_lc.cc#L388>
> >
> >            grep --text DELETED client.<hostname>.log | wc -l
> >         121853
> >
> >
> >         -Ben
> >
> >         On Mon, Jun 5, 2017 at 6:16 AM, Daniel Gryniewicz
> >         <dang@xxxxxxxxxx <mailto:dang@xxxxxxxxxx>
> >         <mailto:dang@xxxxxxxxxx <mailto:dang@xxxxxxxxxx>>> wrote:
> >
> >              Kraken has lifecycle, Jewel does not.
> >
> >              Daniel
> >
> >
> >              On 06/04/2017 07:16 PM, ceph.novice@xxxxxxxxxxxxxxxx
> >         <mailto:ceph.novice@xxxxxxxxxxxxxxxx>
> >              <mailto:ceph.novice@xxxxxxxxxxxxxxxx
> >         <mailto:ceph.novice@xxxxxxxxxxxxxxxx>> wrote:
> >
> >
> >                  grrr... sorry && and again as text :|
> >
> >
> >                  Gesendet: Montag, 05. Juni 2017 um 01:12 Uhr
> >                  Von: ceph.novice@xxxxxxxxxxxxxxxx
> >         <mailto:ceph.novice@xxxxxxxxxxxxxxxx>
> >                  <mailto:ceph.novice@xxxxxxxxxxxxxxxx
> >         <mailto:ceph.novice@xxxxxxxxxxxxxxxx>>
> >                  An: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx
> >         <mailto:yehuda@xxxxxxxxxx>
> >                  <mailto:yehuda@xxxxxxxxxx <mailto:yehuda@xxxxxxxxxx>>>
> >                  Cc: "ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>
> >                  <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>" <ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>
> >                  <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>>, ceph-devel@xxxxxxxxxxxxxxx
> >         <mailto:ceph-devel@xxxxxxxxxxxxxxx>
> >                  <mailto:ceph-devel@xxxxxxxxxxxxxxx
> >         <mailto:ceph-devel@xxxxxxxxxxxxxxx>>
> >                  Betreff: Re:  RGW lifecycle not expiring
> >         objects
> >
> >
> >
> >                  Hi (again) Yehuda.
> >
> >                  Looping in ceph-devel...
> >
> >                  Could it be that lifecycle is still not implemented
> >         neither in
> >                  Jewel nor in Kraken, even if release notes and other
> >         places say so?
> >
> >         https://www.spinics.net/lists/ceph-devel/msg34492.html
> >         <https://www.spinics.net/lists/ceph-devel/msg34492.html>
> >                  <https://www.spinics.net/lists/ceph-devel/msg34492.html
> >         <https://www.spinics.net/lists/ceph-devel/msg34492.html>>
> >         
> > https://github.com/ceph/ceph-ci/commit/7d48f62f5c86913d8f00b44d46a04a52d338907c 
> >
> >         
> > <https://github.com/ceph/ceph-ci/commit/7d48f62f5c86913d8f00b44d46a04a52d338907c> 
> >
> >         
> > <https://github.com/ceph/ceph-ci/commit/7d48f62f5c86913d8f00b44d46a04a52d338907c 
> >
> >         
> > <https://github.com/ceph/ceph-ci/commit/7d48f62f5c86913d8f00b44d46a04a52d338907c>> 
> >
> >         
> > https://github.com/ceph/ceph-ci/commit/9162bd29594d34429a09562ed60a32a0703940ea 
> >
> >         
> > <https://github.com/ceph/ceph-ci/commit/9162bd29594d34429a09562ed60a32a0703940ea> 
> >
> >         
> > <https://github.com/ceph/ceph-ci/commit/9162bd29594d34429a09562ed60a32a0703940ea 
> >
> >         
> > <https://github.com/ceph/ceph-ci/commit/9162bd29594d34429a09562ed60a32a0703940ea>> 
> >
> >
> >                  Thanks & regards
> >                    Anton
> >
> >
> >                  Gesendet: Sonntag, 04. Juni 2017 um 21:34 Uhr
> >                  Von: ceph.novice@xxxxxxxxxxxxxxxx
> >         <mailto:ceph.novice@xxxxxxxxxxxxxxxx>
> >                  <mailto:ceph.novice@xxxxxxxxxxxxxxxx
> >         <mailto:ceph.novice@xxxxxxxxxxxxxxxx>>
> >                  An: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx
> >         <mailto:yehuda@xxxxxxxxxx>
> >                  <mailto:yehuda@xxxxxxxxxx <mailto:yehuda@xxxxxxxxxx>>>
> >                  Cc: "ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>
> >                  <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>" <ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>
> >                  <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>>
> >                  Betreff: Re:  RGW lifecycle not expiring
> >         objects
> >                  Hi Yahuda.
> >
> >                  Well, here we go:
> >         
> > http://tracker.ceph.com/issues/20177[http://tracker.ceph.com/issues/20177] 
> >
> >         
> > <http://tracker.ceph.com/issues/20177%5Bhttp://tracker.ceph.com/issues/20177%5D> 
> >
> >         
> > <http://tracker.ceph.com/issues/20177%5Bhttp://tracker.ceph.com/issues/20177%5D 
> >
> >         
> > <http://tracker.ceph.com/issues/20177%5Bhttp://tracker.ceph.com/issues/20177%5D>> 
> >
> >
> >                  As it's my first one, hope it's ok as it is...
> >
> >                  Thanks & regards
> >                  Anton
> >
> >
> >                  Gesendet: Samstag, 03. Juni 2017 um 00:14 Uhr
> >                  Von: "Yehuda Sadeh-Weinraub" <yehuda@xxxxxxxxxx
> >         <mailto:yehuda@xxxxxxxxxx>
> >                  <mailto:yehuda@xxxxxxxxxx <mailto:yehuda@xxxxxxxxxx>>>
> >                  An: ceph.novice@xxxxxxxxxxxxxxxx
> >         <mailto:ceph.novice@xxxxxxxxxxxxxxxx>
> >                  <mailto:ceph.novice@xxxxxxxxxxxxxxxx
> >         <mailto:ceph.novice@xxxxxxxxxxxxxxxx>>
> >                  Cc: "Graham Allan" <gta@xxxxxxx <mailto:gta@xxxxxxx>
> >         <mailto:gta@xxxxxxx <mailto:gta@xxxxxxx>>>,
> >                  "ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>
> >         <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>"
> >                  <ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>
> >         <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>>
> >                  Betreff: Re:  RGW lifecycle not expiring
> >         objects
> >                  Have you opened a ceph tracker issue, so that we don't
> >         lose track of
> >                  the problem?
> >
> >                  Thanks,
> >                  Yehuda
> >
> >                  On Fri, Jun 2, 2017 at 3:05 PM,
> >         <ceph.novice@xxxxxxxxxxxxxxxx 
> > <mailto:ceph.novice@xxxxxxxxxxxxxxxx>
> >                  <mailto:ceph.novice@xxxxxxxxxxxxxxxx
> >         <mailto:ceph.novice@xxxxxxxxxxxxxxxx>>> wrote:
> >
> >                      Hi Graham.
> >
> >                      We are on Kraken and have the same problem with
> >         "lifecycle".
> >                      Various (other) tools like s3cmd or CyberDuck do
> >         show the
> >                      applied "expiration" settings, but objects seem
> >         never to be
> >                      purged.
> >
> >                      If you should have new findings, hints,... PLEASE
> >         share/let
> >                      me know.
> >
> >                      Thanks a lot!
> >                      Anton
> >
> >
> >                      Gesendet: Freitag, 19. Mai 2017 um 22:44 Uhr
> >                      Von: "Graham Allan" <gta@xxxxxxx
> >         <mailto:gta@xxxxxxx> <mailto:gta@xxxxxxx <mailto:gta@xxxxxxx>>>
> >                      An: ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>
> >         <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>
> >
> >                      Betreff:  RGW lifecycle not expiring
> >         objects
> >                      I've been having a hard time getting the s3 object
> >         lifecycle
> >                      to do
> >                      anything here. I was able to set a lifecycle on a 
> > test
> >                      bucket. As others
> >                      also seem to have found, I do get an EACCES error
> >         on setting the
> >                      lifecycle, but it does however get stored:
> >
> >                          % aws --endpoint-url
> >         https://xxx.xxx.xxx.xxx[https://xxx.xxx.xxx.xxx
> >         <https://xxx.xxx.xxx.xxx>
> >         <https://xxx.xxx.xxx.xxx>][https://xxx.xxx.xxx.xxx
> >         <https://xxx.xxx.xxx.xxx>
> >         <https://xxx.xxx.xxx.xxx>[https://xxx.xxx.xxx.xxx
> >         <https://xxx.xxx.xxx.xxx>
> >                          <https://xxx.xxx.xxx.xxx>]] s3api
> >                          get-bucket-lifecycle-configuration 
> > --bucket=testgta
> >                          {
> >                          "Rules": [
> >                          {
> >                          "Status": "Enabled",
> >                          "Prefix": "",
> >                          "Expiration": {
> >                          "Days": 3
> >                          },
> >                          "ID": "test"
> >                          }
> >                          ]
> >                          }
> >
> >
> >                      but many days later I have yet to see any object
> >         actually
> >                      get expired.
> >                      There are some hints in the rgw log that the expiry
> >         thread
> >                      does run
> >                      periodically:
> >
> >                          2017-05-19 03:49:03.281347 7f74f1134700 2
> >                          RGWDataChangesLog::ChangesRenewThread: start
> >                          2017-05-19 03:49:16.356022 7f74ef931700 2 object
> >                          expiration: start
> >                          2017-05-19 03:49:16.356036 7f74ef931700 20
> >         proceeding
> >                          shard = obj_delete_at_hint.0000000000
> >                          2017-05-19 03:49:16.359785 7f74ef931700 20
> >         proceeding
> >                          shard = obj_delete_at_hint.0000000001
> >                          2017-05-19 03:49:16.364667 7f74ef931700 20
> >         proceeding
> >                          shard = obj_delete_at_hint.0000000002
> >                          2017-05-19 03:49:16.369636 7f74ef931700 20
> >         proceeding
> >                          shard = obj_delete_at_hint.0000000003
> >
> >                      ...
> >
> >                          2017-05-19 03:49:16.803270 7f74ef931700 20
> >         proceeding
> >                          shard = obj_delete_at_hint.0000000126
> >                          2017-05-19 03:49:16.806423 7f74ef931700 2 object
> >                          expiration: stop
> >
> >
> >                      "radosgw-admin lc process" gives me no output 
> > unless I
> >                      enable debug, then:
> >
> >                          ]# radosgw-admin lc process
> >                          2017-05-19 15:28:46.383049 7fedb9ffb700 2
> >                          RGWDataChangesLog::ChangesRenewThread: start
> >                          2017-05-19 15:28:46.421806 7feddc240c80 10
> >         Cannot find
> >                          current period zone using local zone
> >                          2017-05-19 15:28:46.453431 7feddc240c80 2 all 8
> >         watchers
> >                          are set, enabling cache
> >                          2017-05-19 15:28:46.614991 7feddc240c80 2 
> > removed
> >                          watcher, disabling cache
> >
> >
> >                      "radosgw-admin lc list" seems to return "empty" 
> > output:
> >
> >                          # radosgw-admin lc list
> >                          []
> >
> >
> >                      Is there anything obvious that I might be missing?
> >
> >                      Graham
> >                      --
> >                      Graham Allan
> >                      Minnesota Supercomputing Institute - gta@xxxxxxx
> >         <mailto:gta@xxxxxxx>
> >                      <mailto:gta@xxxxxxx <mailto:gta@xxxxxxx>>
> >                      _______________________________________________
> >                      ceph-users mailing list
> >         ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> >         <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>
> >         
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]]] 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5D> 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5D 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D%5D>> 
> >
> >
> >
> >                      _______________________________________________
> >                      ceph-users mailing list
> >         ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> >         <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>
> >         
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]]] 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D> 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D%5D>> 
> >
> >
> >                  _______________________________________________
> >                  ceph-users mailing list
> >         ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> >         <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>
> >         
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com][http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com]] 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D> 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D%5D>> 
> >
> >
> >                    _______________________________________________
> >         ceph-users
> >                  mailing list ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>
> >                  <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>
> >         
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com[http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com] 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D> 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D 
> >
> >         
> > <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5Bhttp://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com%5D>> 
> >
> >                  _______________________________________________
> >                  ceph-users mailing list
> >         ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> >         <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>
> >         http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >         <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> >                  <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >         <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>
> >
> >
> >              _______________________________________________
> >              ceph-users mailing list
> >         ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> >         <mailto:ceph-users@xxxxxxxxxxxxxx
> >         <mailto:ceph-users@xxxxxxxxxxxxxx>>
> >         http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >         <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> >              <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >         <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>>
> >
> >
> >
> >
> >         _______________________________________________
> >         ceph-users mailing list
> >         ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> >         http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >         <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
> >
> >
> >     --     Graham Allan
> >     Minnesota Supercomputing Institute - gta@xxxxxxx <mailto:gta@xxxxxxx>
> >     _______________________________________________
> >     ceph-users mailing list
> >     ceph-users@xxxxxxxxxxxxxx <mailto:ceph-users@xxxxxxxxxxxxxx>
> >     http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >     <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>

--
Graham Allan
Minnesota Supercomputing Institute - gta@xxxxxxx
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com