On Thu, Apr 27, 2017 at 2:32 AM, Nick Coghlan <ncoghlan@xxxxxxxxx> wrote: > On 27 April 2017 at 11:47, Nico Kadel-Garcia <nkadel@xxxxxxxxx> wrote: >> On Wed, Apr 26, 2017 at 7:13 AM, Charalampos Stratakis >>> At the present time, running sudo pip3 in Fedora is not safe. >>> Pip shares its installation directory with dnf, can remove >>> dnf-managed files and generally break the Python 3 interpreter. >> >> This is true of every version of pip, not merely pip3. It was also >> true of CPAN, and of many gradle, maven, and ant working environments >> that semi-randomly collate the very latest versions of indeterminate >> components and spray them on top of your system workspace with their >> own quite distinct ideas about packaging and versioning. >> >> There is no solution to this problem, except to use tools like >> "pyvenv" to set aside secluded workspaces for Python modules and their >> dependencies assembly. So, for most developers, they *should not* use >> "sudo pip". They should remain in user space, or possibly in shared >> workspaces, to avoid overwriting system components. > > Nothing is changing in terms of *recommended* practices. This change > proposal is entirely about harm mitigation for the cases where users > *do* run "sudo pip ...", either because that's their instinctive > reaction to a permissions error, or because some misguided > installation instructions for a 3rd party package advised them to do > it. > > Debian and derivatives already mitigate the potential harm for these > cases by requiring the "--install-layout=deb" option to be passed to > get distutils to install into the system directories rather than doing > it by default: https://wiki.debian.org/Python#Deviations_from_upstream > > Their approach means that any harm caused by "sudo pip install X" can > subsequently be fully reversed by doing "sudo pip uninstall X". > > At this moment, this is NOT true for Fedora and derivatives. Instead, > the remediation step here is "sudo pip uninstall X && sudo dnf > reinstall <something>" where you have to: > > 1. Figure out what "<something>" needs to be > 2. Hope that whatever you broke didn't affect your ability to run > "sudo dnf reinstall" > >>> Our first attempt to make sudo pip safe on Fedora [0] was >>> based on using a different binary (/usr/libexec/system-python) >>> for RPM packaging purposes and changing the behavior >>> of /usr/bin/python3 (and pip that uses its settings) to install >>> under /usr/local by default. Switching all the Python 3 packages >> >> Oh, Oh my. The extent to which this unworkable is... fascinating. > > Nico, the contemptuous tone you have adopted in this post is entirely > inappropriate. Please give other list participants the courtesy of > assuming expertise at least equivalent to your own, rather than > engaging in attempted displays of assumed superiority. I'll try to scale back. But it's a very familiar issue. It's difficult, even impossible, to completely isolate the results of one package management from the other without completely segregating their work spaces. It's been tried with CPAN, ant, gradle, maven, and other build systems. I actually tried similar approaches to isolate pip insalles for airflow, and for awscli. It was very painful. The dependencies themselves dependencies, including 3 distinct versions of the "Sphinx" document processing system, one version of which broke compilation for other components because the current version was incompatible with older software on the system, none of which were reasonable to go back and introduce locked software versions for. And the problem is a frequent one. Pip published modules are neither required, nor particularly expected, to remain compatible with older versions of software which may be dependent on it. And yes, I'd had "pip install" break RPM by upgrading its modules. It even breaks other tools that are already working, even pip installed tools, because of the tendency to upgrade to the latest version of every component. It's possible to package Python modules consistently. "py2pack" has proven an invaluable tool for me to actually package Python modules as RPM's for system deployment, and I do recommend it for general use. "pyvenv" can and does effectively isolate pip install from the base Fedora system for the safety of the base Fedora and of the particular pip environment. >> You >> will inevitably install incompatible future versions of critical >> packages which are incompatible with the Fedora system packages. And >> as soon as you install one that breaks RPM, if /usr/local/ if ahead of >> /usr in your search path for modules, you are *screwed*. > > This isn't true, as long as the fix is as simple as using pip to > uninstall whatever problematic package you just installed. > > At the moment, that isn't the case on Fedora and derivatives, since > "sudo pip install ..." can actually *corrupt* a system package > installation, rather than merely shadowing it at import time the way > it does on Debian et al. It's not merely that it can corrupt a system package, which it can do. It can introduce dependencies which replace functionality for tested system versions of those packages. Such an approach leaving the locally installed version of a module as the system default can actually *break pip", and even brak RPM, or cause flapping of version and successful functionality for the desired user application as the pip module and a later installed RPM version, fight it out out for version supremacy. Been there, done that, had to yank out the modules manually and take back sudo access from the developer who kept using "sudo pip install" on a shared system. That developer was also confident that pip did something like version control. It's not normally configured to do so: dependencies are usually published with no version stated. When a version is stated, it's usually a ">=" version number, and is generally not set to lock down the current version of modules that may have major and incompatible API changes. >> And if >> /usr/local is first, well, an RPM update may introduce a component >> that is incompatible with the pip installed modules at arbitrary >> times. > > The goal of this change isn't to allow co-existence of arbitrary > upstream and downstream versions as supported combinations - it's to > make sure that forced installation of an incompatible upstream version > is an easily reversible error, rather than a potentially > system-breaking mistake. I'm on board with the goal. I just think it's much safer to segregate them further from system utilities. pyvenv does a reasonably good job, and is much less likely to attempt to install an inappropriately upgraded system module in the default Python module paths, breaking other critical system tools. >>> We decided to try a different approach. The main idea is to detect >>> an ongoing RPM build and modify the behavior of the Python 3 >>> executable only when in normal user environment so that RPM builds >>> won't be affected at all. >> >> See https://xkcd.com/1172/ . You're inventing a workflow that isn't >> supported by *anyone*. > > The problem is that the default behaviour needs to change (to mitigate > the potential harm caused by a relatively common end user error), but > unlike Debian package builds (which know they need to set > "--install-layout=deb"), most RPM builds for Python packages currently > rely on that default behaviour as part of the build process. Changing > the default indiscriminately would thus be a breaking change that not > only affects Fedora RPM builds, but also COPR and other third party > RPM builds. > > So we need a solution that retains the existing behaviour for RPM > builds, but changes the behaviour to be less potentially destructive > when run interactively at a shell prompt, or as part of a > configuration management script. I'd call the approach you describe less *likely* destructive. Not less potentially destructive. Since "sudo pip install" can, through no malice of anyone involved, install packages that have never been tested together, run their "build" procedures, and do an accidental "rm -rf" of your entire filesystem with just such a component, the potential for destruction remains as high as it is for any software run as the root user. >>> The adjustment of the behavior can be done on different levels. >>> The first option is to set the sys.prefix variable to /usr/local >>> when the interpreter is initialized. This will affect >>> all the install methods, but the solution can cause some >>> problems in applications that rely on the value of sys.prefix. >>> A prototype of this implementation can be seen here [1]. >> >> I'm sorry, but.... ouch. Please stop trying to implement "pip install" >> as a system installation tool. > > That is not the goal of the change. If you're putting the modules in /usr/local/, for default use by all system users, then it looks very much like a system installation tool >> The effort would be better spent >> upgrading and enhancing "py2pack". > > Improving the level of commonality and collaboration between Fedora & > Mageia's pyp2rpm and openSUSE's py2pack would indeed be desirable, but > it's entirely orthogonal to the harm mitigation exercise being > discussed here. Allow me to respectfully. Easing the path to getting the modules into RPM would profoundly improve compatibility, and package dependency testing and control, far more than investing the effort into working around the compatibility issues and risks of a "sudo pip install" operations. Moreover, it would help provide extremely valuable version control of the build and install time dependencies of the desired pip module. >>> The other possibility is to limit the pip install location change >>> to distutils and pip [2]. This is the "safer" option, but does >>> not cover all corner cases. For example, Python software built >>> locally using cmake or similar tools will be installed into >>> /usr/lib and can conflict with system tools. Debian chose to >>> implement similar solution. >> >> It's local environment specific, unstable software sensitive to >> arbitrary and unmanaged upstream dependency updates, and extremely >> sensitive to local system pip modules. It *does not belong* in >> /usr/lib. Because the components are modular and bundled in a non-RPM >> compatible fashion, it behooves developers to use a tool to segregate >> the tools as much as feasible from the Fedora underlying >> infrastructure. i.e., use pyvenv to build them in a contained >> environment segregated from the system files. > > Helping users that are already doing the right thing isn't the problem > at hand - this is about making it easier for users to recover from > doing the *wrong* thing. > > Cheers, > Nick. Wouldn't it be more effective, more safe, and more efficient to improve the existing support for doing "the right thing", rather than trying to outsmart a build and deployment system that is essentially outside your control? To encourage the use of "pyvenv" for local such modules, in a local userspace, rather than as a default system version? Frankly, I've learned a very real distrust for "pip install". It's not a loathing for their attempt to create such a system. It's hard won experience from the last 5 or so such systems I've had to deal with personally and professionally. Version control and component compatibility have repeatedly bitten me *very* hard, especially when a component upgrade breaks other previously stable components. And debugging them can be *nightmarish*. _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
