<sorry for double-send — forgot to make plain text for ceph-devel>
Hi Shinobu,
Thanks for the response.
> On Jul 23, 2015, at 5:05 PM, Shinobu Kinjo <shinobu.kj@xxxxxxxxx> wrote:
>
> Hi Travis,
>
> Is this that you are talking about:
>
> ``dnf [options] list obsoletes [<package-name-specs>...]``
> ``dnf [options] repository-packages <repoid> info obsoletes [<package-name-spec>...]``
> ``dnf [options] repository-packages <repoid> list obsoletes [<package-name-spec>...]``
Partially. Those queries either list packages already installed that are obsoleted by packages available in any enabled repos, or list packages in the repos that obsolete packages that are already installed (the man page says reposository-packages info/list are the same thing?). Such commands would show the issue I am trying to solve - namely that for certain releases of Ceph, older packages of Ceph from EPEL take priority over newer ones, due to package obsoletions.
The exact scenario we hit was a problem most commonly on CentOS - I have not confirmed whether it is an issue in Fedora. But here is a specific example. Starting with a CentOS 7 machine that has EPEL installed/enabled, and Ceph rpm-firefly enabled by default, and the yum priorities plugin installed (with check_obsoletes=0):
# yum info --disablerepo=Ceph --disablerepo=epel --disableplugin=priorities ceph
Error: No matching Packages to list
^ Expected - ceph packages aren’t reachable
# yum info --disablerepo=Ceph --disableplugin=priorities ceph
Available Packages
Name : ceph
Arch : x86_64
Epoch : 1
Version : 0.80.7
Release : 0.4.el7
^ EPEL includes Ceph 0.80.7.
# yum info --disablerepo=epel --disableplugin=priorities ceph
Available Packages
Name : ceph
Arch : x86_64
Epoch : 1
Version : 0.80.10
^ Ceph repos have 0.80.10.
# yum info --disableplugin=priorities ceph
Available Packages
Name : ceph
Arch : x86_64
Epoch : 1
Version : 0.80.7
^ With priorities disabled, but both repos enabled, yum resolves the lower version number, 0.80.7 from EPEL.
# yum info ceph
24 packages excluded due to repository priority protections
Available Packages
Name : ceph
Arch : x86_64
Version : 0.80.10
^ With priorities enabled, we now get 0.80.10. The ceph.repo file has priority=1 in it.
Great! With priorities enabled, we now see 0.80.10. Let’s install:
# yum install -v ceph
….
….
Error: Package: 1:python-rbd-0.80.7-2.el7.x86_64 (base)
Requires: librbd1 = 1:0.80.7-2.el7
Available: librbd1-0.80-0.el7.x86_64 (Ceph)
librbd1 = 0.80-0.el7
Available: librbd1-0.80.1-0.el7.x86_64 (Ceph)
librbd1 = 0.80.1-0.el7
Available: librbd1-0.80.3-0.el7.x86_64 (Ceph)
librbd1 = 0.80.3-0.el7
Available: librbd1-0.80.4-0.el7.x86_64 (Ceph)
librbd1 = 0.80.4-0.el7
Available: librbd1-0.80.5-0.el7.x86_64 (Ceph)
librbd1 = 0.80.5-0.el7
Available: librbd1-0.80.6-0.el7.x86_64 (Ceph)
librbd1 = 0.80.6-0.el7
Available: librbd1-0.80.7-0.el7.x86_64 (Ceph)
librbd1 = 0.80.7-0.el7
Available: librbd1-0.80.8-0.el7.x86_64 (Ceph)
librbd1 = 0.80.8-0.el7
Available: librbd1-0.80.9-0.el7.x86_64 (Ceph)
librbd1 = 0.80.9-0.el7
Installing: librbd1-0.80.10-0.el7.x86_64 (Ceph)
librbd1 = 0.80.10-0.el7
So why did the install fail? See [1] for full output, but the short version is at this step:
--> Processing Dependency: python-ceph for package: ceph-0.80.10-0.el7.x86_64
Searching pkgSack for dep: python-ceph
Not Updating Package that is obsoleted: python-ceph-0.80.10-0.el7.x86_64
TSINFO: Marking 1:python-ceph-compat-0.80.7-0.4.el7.x86_64 as install for ceph-0.80.10-0.el7.x86_64
When yum looks for python-ceph, it sees that it has been marked as obsoleted by python-ceph-compat, which is available from EPEL. Pulling in that python-ceph-compat causes all kinds of problems and the install ultimately fails. The solution is set check_obsoletes = 1 in /etc/yum/pluginconf.d/priorities.conf, which forces Yum to override the the obsoletion of a package from a lower priority repo. This is what we are doing today.
This problem still exists for our Firefly and Giant packages, even though the EPEL package added versions to their obsoletes over 4 months ago: [2]. It is this “check_obsoletes” behavior that I am unsure of in DNF. Granted, I haven’t tried it. It may be that it will read the same config file, even (/etc/yum/pluginconf.d/priorities.conf). I’d have to install DNF in a Fedora 20 VM (the last Fedora we built production packages of Firefly on) to see how DNF behaves here.
Even if check_obsoletes wasn’t a consideration, I’d really like to get away from setting priority values in repo files and installing an additional plugin for Yum to make things work. It feels like it is using magic numbers to make things resolve correctly. Maybe this is entirely invalid, since having the priorities plugin rolled into DNF gives it an air of legitimacy. I wanted to reach out to the Ceph community to see if a two phased install with a disabled-by-default EPEL made sense. I also wanted to see if any Yum/DNF experts said I was crazy. :)
As the yum examples in here show, having EPEL enabled can lead to unexpected resolutions. It is not immediately obvious to most people why 0.80.7 from EPEL would be picked by Yum instead of 0.80.10 that’s on ceph.com. And even when the resolution of a top-level package produces the correction version, that doesn’t mean that all the dependencies are going to resolve well.
[1] http://fpaste.org/247700/14377137/
[2] http://pkgs.fedoraproject.org/cgit/ceph.git/commit/?h=epel7&id=c9a91bad2f3c3083b8dad7a1feb9f84994c2f35c
- Travis
>
> - Shinobu
>
>
> On Fri, Jul 24, 2015 at 8:12 AM, Travis Rhoden <trhoden@xxxxxxxxxx> wrote:
>> HI Everyone,
>>
>> I’m working on ways to improve Ceph installation with ceph-deploy, and a common hurdle we have hit involves dependency issues between ceph.com hosted RPM repos, and packages within EPEL. For a while we were able to managed this with the priorities plugin, but then EPEL shipped packages that included changes that weren’t available on the ceph.com packages, and the EPEL packages “obsoleted” the ceph.com ones. This caused EPEL packages to take priority over ceph.com packages even when ceph.com packages had greater version numbers. The solution to this was to enable the “check_obsoletes” feature of the priorities plugin. That’s where we are today.
>>
>> Recently when working with DNF, I observed that the priorities feature got pulled natively into DNF, but I cannot find anything about whether “check_obsoletes” is still necessary or even an option. Regardless, I would like move away from this workflow as it is generally seen as poor. [1] [2]
>>
>> What I’d like to propose instead of ceph-deploy’s current workflow is to:
>>
>> (1) install epel-release on nodes that need it
>> (2) disable EPEL by default (using yum-config-manager)
>> (3) When installing Ceph, break the install into two parts
>> (3)(a) Explicitly install Ceph’s dependencies from EPEL by name, using yum —enablerepo=epel
>> (3)(b) Proceed normally with Ceph installation, but adding a —disablerepo=epel flag as well
>>
>> Note: the disabling of EPEL in 3b seems redundant with 2, but it would cover cases when a user/admin chooses to enable EPEL by default. We are mostly concerned with nodes that are dedicated to Ceph and therefore ceph-deploy is free to do things like disabling EPEL, but that’s certainly not always ideal. We could disable it by default *only* if we were the ones to install it. If it’s already there, we leave it along but then still do our two-phase install and explicitly disable it when doing the second phase of install.
>>
>> I think this workflow would allow us to no longer need to use repo priorities, but I might be missing something. A secondary motive to this is to end up with systems that EPEL disabled by default because it has caused issues with Calamari, where EPEL has newer packages of certain things than what gets installed initially and then breaks Calamari. Having EPEL disabled will prevent that, and will also prevent things like “yum update” from breaking things.
>>
>> Potential downsides I see are what happens when there are updates in EPEL that we want, say for a security fix?
>>
>> - Travis
>>
>>
>> [1] http://wiki.centos.org/PackageManagement/Yum/Priorities#head-38b91468cc607d0243f463489c2334bf40bfaaee
>> [2] http://wiki.centos.org/PackageManagement/Yum/Priorities#head-6601a4937d4b099e6d46eea0bdb54241d51c7277
>
>
>
> --
>>
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@xxxxxxxxxxxxxxx
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Life w/ Linux
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
