This is very kind of you. Hopefully noone will mind ;-) On 05/07/2014 16:39, Yuri Weinstein wrote: > I killed several runs that had been running for 2-3 days, hopefully it > will speed up your runs. > > Thx > YuriW > > On Sat, Jul 5, 2014 at 6:46 AM, Loic Dachary <loic@xxxxxxxxxxx> wrote: >> >> Hi, >> >> It looks like there is a shortage of VPS for some reason: >> >> http://pulpito.ceph.com/loic-2014-07-03_11:24:33-upgrade:firefly-x:stress-split-wip-8475-testing-basic-vps/ >> >> has a number of tests scheduled since ~48h and not making progress. >> >> Cheers >> >> On 04/07/2014 00:39, Loic Dachary wrote: >>> Hi Ceph, >>> >>> The firefly-x test upgrade suite is designed to check that upgrading from Firefly to a newer version (master or a branch) works as expected. It was created it by copying dumpling-x and can be browsed at https://github.com/ceph/ceph-qa-suite/tree/master/suites/upgrade/firefly-x >>> >>> To establish a baseline, a run was scheduled to upgrade from firefly to firefly (i.e. no upgrade really ;-) and it should therefore show that when nothing happens all is well. It however fails in various ways as can be seen here. >>> >>> ./virtualenv/bin/teuthology-suite --suite upgrade/firefly-x/stress-split --suite-dir ~/software/ceph/ceph-qa-suite --ceph firefly --machine-type vps --email loic@xxxxxxxxxxx http://pulpito.ceph.com/loic-2014-07-02_23:05:05-upgrade:firefly-x:stress-split-firefly-testing-basic-vps/ >>> >>> * Command failed on vpm105 with status 1: 'sudo yum install -y http://gitbuilder.ceph.com/kernel-rpm-redhatenterpriseserver6-x86_64-basic/sha1/8102ce7556a99f6348067c60583320d308f36362/kernel.x86_64.rpm' >>> Does that mean kernels are not ready yet for this distribution and the tests should be skipped ? >>> * Command failed on vpm058 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" >>> http://pulpito.ceph.com/loic-2014-07-02_23:05:05-upgrade:firefly-x:stress-split-firefly-testing-basic-vps/338941 >>> >>> Although it looks like http://tracker.ceph.com/issues/7808 which is a duplicate of http://tracker.ceph.com/issues/7799 it is slightly different and http://tracker.ceph.com/issues/8735 was created to keep track of it. >>> >>> * Command failed on vpm070 with status 1: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f -i 1' http://pulpito.ceph.com/loic-2014-07-02_23:05:05-upgrade:firefly-x:stress-split-firefly-testing-basic-vps/338904/ >>> >>> Although the root of the error seems to be that osd 1 cannot be killed by the thrasher, I don't see meaningfull error messages. http://tracker.ceph.com/issues/8736 was filed to keep track of this condition. >>> >>> * timed out waiting for admin_socket to appear after osd.1 restart http://pulpito.ceph.com/loic-2014-07-02_23:05:05-upgrade:firefly-x:stress-split-firefly-testing-basic-vps/338908/ >>> >>> It looks like a race : the osd is killed at the same time it is restarted by the thrasher and http://tracker.ceph.com/issues/8737 was opened for this >>> >>> * hang on "INFO:teuthology.task.rados:joining rados" >>> http://pulpito.ceph.com/loic-2014-07-02_23:05:05-upgrade:firefly-x:stress-split-firefly-testing-basic-vps/338915/ >>> >>> It looks like a bug and http://tracker.ceph.com/issues/8740 was filed >>> >>> When the same suite is run to upgrade from firefly to master it gives http://pulpito.ceph.com/loic-2014-07-02_22:04:23-upgrade:firefly-x:stress-split-master-testing-basic-vps/ which shows the following errors: >>> >>> * Command failed on vpm105 with status 1: 'sudo yum install -y http://gitbuilder.ceph.com/kernel-rpm-redhatenterpriseserver6-x86_64-basic/sha1/8102ce7556a99f6348067c60583320d308f36362/kernel.x86_64.rpm' (same as above) >>> >>> * Could not reconnect to ubuntu@xxxxxxxxxxxxxxxxxxxxxxxxxxx : it looks like a transient timeout problem that can be ignored >>> http://pulpito.ceph.com/loic-2014-07-02_22:04:23-upgrade:firefly-x:stress-split-master-testing-basic-vps/338891/ >>> 2014-07-02T18:52:24.546 INFO:teuthology.orchestra.connection:{'username': u'ubuntu', 'hostname': u'vpm042.front.sepia.ceph.com', 'timeout': 60} >>> >>> * Command failed on vpm017 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" >>> One of which looks exactly as http://tracker.ceph.com/issues/7799 which was re-opened >>> >>> * hang on "INFO:teuthology.task.rados:joining rados" (same as above) >>> >>> Cheers >>> >> >> -- >> Loïc Dachary, Artisan Logiciel Libre >> -- Loïc Dachary, Artisan Logiciel Libre
Attachment:
signature.asc
Description: OpenPGP digital signature
