On Mon, Jun 29, 2020 at 11:58 AM Patrick Donnelly <pdonnell@xxxxxxxxxx> wrote: > > On Mon, Jun 29, 2020 at 11:38 AM Patrick Donnelly <pdonnell@xxxxxxxxxx> wrote: > > > > On Mon, Jun 29, 2020 at 10:35 AM Yuri Weinstein <yweinste@xxxxxxxxxx> wrote: > > > fs - approved Patrick? > > > > There's a new upgrade test failure: > > https://pulpito.ceph.com/yuriw-2020-06-25_22:31:00-fs-octopus-distro-basic-smithi/5180149 > > > > 2020-06-26T07:13:13.444 INFO:teuthology.orchestra.run.smithi087:> sudo > > adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage > > timeout 120 ceph --cluster ceph --log-early osd dump --format=json > > ... > > 2020-06-26T07:13:13.709 > > INFO:teuthology.orchestra.run.smithi087.stderr:Error EINVAL: invalid > > command > > > > I vaguely recall seeing this before but can't find the tracker ticket. > > The last successful run was on the 24th (the day before you scheduled > > tests!!): > > > > https://pulpito.ceph.com/teuthology-2020-06-24_04:10:03-fs-octopus-distro-basic-smithi/5174869/ > > > > The difference in commits > > 5c9eee5a7ad1bb8b161f33b4560d5dfc4a000b6d..upstream/heads/octopus does > > not shine light on why this would happen. > > > > Any ideas? > > Looks like this is caused by an older version of Mimic, v13.2.8. The > successful run from June 24th: > > 2020-06-25T06:10:56.756 DEBUG:teuthology.packaging:Querying > https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=mimic > 2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Pulling from > https://4.chacra.ceph.com/r/ceph/mimic/8e9675a830b7e7a7164659a2a8f8ad08b8f61358/ubuntu/bionic/flavors/default/ > 2020-06-25T06:10:56.938 INFO:teuthology.task.install.deb:Package > version is 13.2.10-15-g8e9675a830-1bionic > > vs. the failure on June 25th: > > 2020-06-26T07:11:31.659 DEBUG:teuthology.packaging:Querying > https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=mimic > 2020-06-26T07:11:31.820 INFO:teuthology.task.install.deb:Pulling from > https://chacra.ceph.com/r/ceph/mimic/5579a94fafbc1f9cc913a0f5d362953a5d9c3ae0/ubuntu/bionic/flavors/default/ > 2020-06-26T07:11:31.821 INFO:teuthology.task.install.deb:Package > version is 13.2.8-1bionic > > This is probably caused by the LRC data loss. Looks like we need to > get new builds of Mimic. yes, the bug we are hitting was fixed by https://github.com/ceph/ceph/pull/33130. > > -- > Patrick Donnelly, Ph.D. > He / Him / His > Senior Software Engineer > Red Hat Sunnyvale, CA > GPG: 19F28A586F808C2402351B93C3301A3E258DD79D > _______________________________________________ Dev mailing list -- dev@xxxxxxx To unsubscribe send an email to dev-leave@xxxxxxx
