[Nix-dev] Python: getting rid of PYTHONPATH in Nixpkgs

[Domen Kožar]

You can do something like:

(python3.withPackages (ps: [ ps.mrbob ])).override(foo: {ignoreCollisions =
true;})

On Wed, Nov 2, 2016 at 11:33 AM, Thomas Hunger <tehunger at gmail.com> wrote:

> I played with `withPackages` and it's rather nice. The only problem I had
> was "collision between A and B" style errors. Is there any plan to allow
> for collisions?
>
> On 2 November 2016 at 06:05, Dmitry Kalinkin <dmitry.kalinkin at gmail.com>
> wrote:
>
>>
>> > On 01 Nov 2016, at 11:56, Freddy Rietdijk <freddyrietdijk at fridh.nl>
>> wrote:
>> >
>> >
>> > > To solve 2) you could also make a wrapper that combines selected
>> outputs of the multiple output package without python itself. I think, this
>> is what texlive does. That way all the burden of wrapping will be limited
>> only to the multiple output packages.
>> >
>> > I'm not sure whether I understand what you mean. What kind of wrapper
>> do you mean? What does it wrap, and what does it set? The problem with the
>> multiple outputs is that when you would add each to PYTHONPATH, the extra
>> outputs won't be importable unless the 'main' output is first. I don't see
>> how that can be solved. One that sets PYTHONPATH? Or some kind of package
>> I had in mind a simple ```combine [blah.foo blah.bar]``` that would
>> produce a derivation with symlinks to both $foo/lib/python2.7/blah/foo and
>> $foo/lib/python2.7/blah/bar, so multiple outputs are combined back and can
>> be importable with the current envHook setting the PYTHONPATH. This is
>> similar to the current python wrapper that additionally throws in the
>> python interpreter itself along with wrapper to set PYTHONHOME.
>> >
>> > > As for the first point, I couldn’t very well understand what the
>> problem is. Are you talking about conflict between instances of a package
>> that were built against different version of the python? Then shouldn’t
>> `--prefix` override the incompatible version, since, as you said, order
>> matters?
>> >
>> > Example with discussion: https://github.com/NixOS/nixpkgs/pull/17749
>> >
>> >
>> > On Tue, Nov 1, 2016 at 4:12 PM, Dmitry Kalinkin <
>> dmitry.kalinkin at gmail.com> wrote:
>> >
>> > > On 01 Nov 2016, at 06:22, Freddy Rietdijk <freddyrietdijk at fridh.nl>
>> wrote:
>> > >
>> > > Hi,
>> > >
>> > > Currently we use PYTHONPATH a lot in Nixpkgs to let applications and
>> the interpreter find Python modules. This typically works fine, but there
>> are problems with this method and so I would like to get rid of it and use
>> only `python.withPackages` which uses `python.buildEnv`.
>> > >
>> > > The two main issues with the use of PYTHONPATH:
>> > >       • Before we used `--prefix $PYTHONPATH`, but this would leak
>> PYTHONPATH into subprocesses, which is especially problematic when both
>> parent and child depend on Python but of different versions.  `--set
>> $PYTHONPATH` would solve that issue, but that makes it impossible to add
>> other (impure) paths, which is an important feature. This also breaks
>> alternative shells like `ipython`. This issue is currently solved by
>> extending `sys.path` in the Python applications.
>> > >       • Limits the use of multiple outputs. When moving a module of a
>> package into a separate output it becomes problematic to load this again,
>> since just adding the module to PYTHONPATH typically doesn't work because
>> the order matters. While Python modules are typically small, and build
>> fast, rebuilding can take a lot of time in cases like `matplotlib` which
>> supports multiple backends and is depended on by quite some packages.
>> > > A method that is more reliable is building an environment with
>> symbolic links to all the modules, and wrapping the applications with a
>> wrapper that sets PYTHONHOME. This is exactly what `python.buildEnv` does,
>> and it solves both 1) and 2).
>> > >
>> > > `PYTHONPATH` is mainly constructed with the Python interpreter
>> setupHook. It is used in `buildPythonPackage` for building the package, and
>> after installing it is extended so the tests can run. Furthermore,
>> `PYTHONPATH` is set by the `setupHook` when using `nix-shell` like
>> `nix-shell -p pythonPackages.numpy`.
>> > >
>> > > I think we can get rid of the setupHook. For the building we can
>> create an env. This would be able to support multiple outputs as inputs,
>> but will be more expensive than setting PYTHONPATH. For the tests we do add
>> the newly constructed package to PYTHONPATH; there's no way around it and
>> it doesn't cause any problems either.
>> > >
>> > > The biggest impact will be on how `nix-shell` is used. It won't be
>> possible anymore to use it as shown before, instead `nix-shell -p
>> 'python.withPackages(ps:[ps.numpy])'` would have to be used. While it is
>> possible to keep the setupHook (or use it as a shellHook) it will be
>> unreliable in the case of multiple outputs.
>> > >
>> > > What do you think about this change? Do you see any problems with it?
>> Any other suggestions?
>> > >
>> > > Freddy
>> > > _______________________________________________
>> > > nix-dev mailing list
>> > > nix-dev at lists.science.uu.nl
>> > > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>> >
>> > Hi,
>> >
>> > I’m far from being an expert in python, but here are my thoughts:
>> >
>> > To solve 2) you could also make a wrapper that combines selected
>> outputs of the multiple output package without python itself. I think, this
>> is what texlive does. That way all the burden of wrapping will be limited
>> only to the multiple output packages. As for the first point, I couldn’t
>> very well understand what the problem is. Are you talking about conflict
>> between instances of a package that were built against different version of
>> the python? Then shouldn’t `--prefix` override the incompatible version,
>> since, as you said, order matters?
>> >
>> > Also, if you remove setupHook, won’t you also lose ability to specify
>> python dependencies directly in buildInputs? Put that inside mkDerivation?
>> >
>> > Dmitry
>> >
>>
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20161102/10cfc6e6/attachment.html>