for a long time, our s3tests for lifecycle expiration have been disabled in teuthology, because they were very sensitive to timing, and relied on rgw's rgw_lc_debug_interval configuration to run lifecycle processing more regularly than once per day. test cases that passed reliably in local testing would fail randomly in the teuthology environment Soumya recently revived these expiration tests in a new rgw/lifecycle suite, which sets rgw_lc_debug_interval to 10 seconds, and scales the sleeps in test cases by a multiple of rgw_lc_debug_interval. this has worked fairly well, but we've still seen some failures there separating these tests into their own rgw/lifecycle suite helps a lot, because we can continue to increase these sleeps without causing all of the other s3tests in rgw/verify to take longer. the runtime of the rgw/lifecycle job by itself is around 30-40 minutes. this will continue to grow as we add new test cases, or extend the sleeps that sometimes lead to failures. in the short/medium term, maybe this is good enough but if our goal was the minimize the runtime of these tests, i would consider a model more like our https://github.com/ceph/ragweed/ tests, where each test case is split up into a 'prepare' step and 'check' step. we use ragweed in upgrade tests, which run the prepare step against an older radosgw version, and the check step against an upgraded radosgw for lifecycle testing, the 'prepare' step would create buckets and objects and set lifecycle policies. then we could run 'radosgw-admin lc process' to completion once, then run all of the 'check' steps to verify that the expected expirations/transitions took place instead of relying on rgw_lc_debug_interval to scale the 'Days' in lifecycle rules, 'radosgw-admin lc process' could take a new argument like --rgw-lc-debug-offset=86400 that adds a day to the 'current' time used in lifecycle time comparisons. without relying on sleeps and rgw_lc_debug_interval, this should make the tests much more reliable. if there are test cases that check progress over multiple expiration cycles, the rgw/lifecycle suite would need to loop over more steps instead of just prepare/check _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
