[Nix-dev] [***SPAM***] Dynamic Dependency Management by Calling External Programs from Nix

[Marc Weber]

First:
What is your goal? Have a kernel derivation expose a list of modules it
has built so that configuration.nix can fail because the kernel does not
support nvidia?
Thus you have to build the kernel before evaluation of nixos system
can finish?
What was your goal? Saving time ? :-) (Don't take this too serious)

>     a single manifest file that contains only the info that cabal2nix
>     would have pulled from the network (hashes+urls of .cabal files, and
You already have this: its called hack-nix. It works fine. The manifest
is a huge .nix db used by the solver which was written in nix.
All you have to do is install it.
I did the same for Ruby (which is working fine but causes you to have
20.000 files about packages you never want to use.)

So how would your proposal look like?


== building darcs sample ==

    hackageIndex = fetchurl { .. } # keep binary format

    haskellDependencyTree = { hackageIndex, ghc_version, flags, target_packages}:
      mkDerivation = {
        name = "run cabal2nix dynamically";
        hash_of_resulting_default_nix = "xx...";
        buildPhase = ''
          # (2) run cabal2nix on hackageindex creating dependency tree for target_packages
          # (3) write result to $out/auto-nix-files/default.nix
        '';
      }

    Then you can do

        darcs = import (haskellDependencyTree { ghc-2.8, "+old-base", "darcs" }+"/auto-nix-files/default.nix" )

==

and you have to make sure that nix-build and nix-env cannot exploit your system!
Eg default.nix could contains strange loops causing segfaults ..
Thus never ever run nix-build as root again ?

Funnier: cross compiling: first you have to wait for compiling cabal2nix
for arm, then you will have to build haskellDependencyTree on arm
architecture copying back the dependency/package information you want to
use to create .drv files for the haskell packages.
Thus without "arm" architecture you can't evaluate anything anymore!
Good thing?


==> future: ruby forge, cpan, eclipse packages, vim, emacs, haxe,
java/scala libraries, python, [ .. endless list ]

How to cope with them? Do you want to dump cpan? (maybe there is no
other way, could those package databases be split into pieces? .. )

python is a very funny case: pypi (the package index) exists, but its
dependency information is not complete. You can get dependencies by 
./setup.py --show-dependendencies or such.

Thus you have to download and fetch a source of a package in order to
know about its dependencies so that you can build and pass them.

Thus there is no nice way to statically fetch this information in
advance. (I still have to test whether you can run setup.py --show-deps
or such on each package)
Again nix-build would take ages (fetching all python sources)..

If a mirror is unavailable evaluation of nix-env would break.
Thus doing syntax checks would become very hard.

Let's hope that output does not depend on time or kernel version or
you'll get different package sets using same nixpgks *shrug* :)

While it one step towards "managing scala, java, .. " libraries you also
give up some control.

But yes: I think its the only way to manage sbt, maven, ivy builds
unless Java/Scala community join nixos (unlikely).

It would be terrific if nixos could share eclipse plugins.

but you'll also give up about "nix, tell me how long installation will
take: show me how many bytes will be downloaded, estimate build time
based on recorded build times found on hydra and my cpu power".
It does not exist yet - but could be implemented. It still can,
but the ETA will change the same way it does when running find on /.

I don't have the energy to think about all implications ..

Sincerly
Marc Weber

The related mail I never sent. But maybe its time to do so:


From: Marc Weber <marco-oweber at gmx.de>
To: nix-dev at cs.uu.nl
Cc: 
Bcc: 
Subject: java/scala (maven,ivy,sbt); ruby (gem); python (easyinstall,pip); perl (cpan); eclipse (p2) => future?
Date: Tue, 20 Sep 2011 03:14:19 +0200
Message-Id: <1316476599-sup-1988 at nixos>

There are many universes..

1) Do we want to support them in the future?
2) How do we want to support them in the future?

They all have in common that there is kind of structure based on the
individual build systems:
- you have a package pool
- you can query dependencies (sometimes this can be done after
  installing a package only :-(, eg see all the python packgaes)
- the package pool is that large - that you don't want to have
  a full copy on your hard disk. Eg why have 20.000 package descriptions
  (ruby) or 5.000 (haskell) or probably much more (cpan) if you're only
  interested in 20 packages (and those may not have been packaged yet!)

nixpkgs still has "simpleBuildTool" version 0.7.x. Current versions are
0.10 and 0.11: https://github.com/harrah/xsbt/wiki/Setup

How to bootstrap?
- get a .jar
- run it
What does it do?
Its reusing existing maven/ivy/.. infrastructure: It fetches packages
and puts the result into ~/.ivy2/... the way maven/.. tools do.


We have three options:
(1) adopt Nix (allow derivations to be build dynamically depending on
  output of other packages)
(2) package everything manually - only that what we need. (Nice, but a
  lot of effort)
(3) try to prefetch all info we can get from all packages. Then use
server farms to keep things around - because in the long run nobody
wants to have a dump of cpan/hackage/... on his/her disk.

We can do (2) for packages we need only. Still its causing a lot of work
to keep up to date. (Eg see how much time Peter had to spend on Haskell)

proposal:
What about modifying nix so that depending packages may depend on the
outcome of a package?

Example:

python-lib-A is copied into $TMP (builddir).
Then you can run pyhton setup.py --show-dependencies >> $out/nix-support/deps
Then you nix can read that special file and generate derivations for
those deps on the fly also creating a new derivation for lib-A which
depends on "dynamic dependencies".

Yes - this will cause a lot of "please fix hash and store it somewhere".
But that's all you have to do - and that could be automated.

What does this mean?
When generating derivations you do no longer know the total amount of
derivations until some of the "dynamic" derivations have been built.
Eg python-lib-A may depend on python-lib-B which depends on
python-lib-C. However this all is known after running python setup.py
--show-dependencies .. of python-lib-A and -B.

I know that this suggestion is a huge step - and that we have to think
carefully about the implications.. However I don't see how we should be
able to use the power of nix for all those code with the current man
power we have.
 
I know what I'm talking about. Eg the zipped package descriptions of all
available ruby packages is about 27MB of size! Its great. It works.
I've been able to install 5 year old packages with little effort some
time ago. But I don't think its a good idea to have 141MB of unzipped
27884 files laying around because you want to install 10-20 ruby
libraries..
Neither is it an option to spend 2 hours on package things manually
if you can use existing ruby tools to install them (such as gem install,
easyinstall, pip, cabal, cpan ..)

How do you think about this?

If we made that one little change we could package all those universes
easily without causing mass downloads of package descriptions nobody
wants to have on his/her disk. That's fine for devs. But not for end
user or server systems (?)

But they all use their own "stateful" installation directories.
So I think that it would be best to patch them so that they use the Nix
store as storage engine only. Maybe we can get what nobody else has:
Shared storage of packages on multi user systems this way without
spending additional days / weeks / month on writing nix expressions
manually.

I could extend the list of the subject by el-get (emacs install system),
vim-addon-manager (vim install system), the haxe library installer
(haxelib), pear (php), and even zsh ships its own install system ..
And I'm pretty sure that when thinking about it you'll find more use
cases - even firefox plugins could be installed in /nix/store/..
(although I don't recommend this right now, talking about the future
here)

So why do the effort at all - why not using the individual package
solutions?

Because for admins its easy 
- to verify integrity of packages (using nix-store --verify --check-contents)
- because you can duplicate working setups without caring too much about
  the individual package systems
- you can share builds among users.
- you can still have the one /etc/nixos/configuration.nix file for
  everything.

Thoughts?

Marc Weber

Need a sample what I'm talking about? The new simple build tool
bootstrap .jar dependency list:

B
C
You package this once maybe two times. But at some time you want to code - not package.
(Yes - I want to face the real world)

downloading http://repo1.maven.org/maven2/net/java/dev/jna/jna/3.2.3/jna-3.2.3.jar ...
downloading http://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.8.1/scala-compiler-2.8.1.jar ...
downloading http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.8.1/scala-library-2.8.1.jar ...
downloading http://repo1.maven.org/maven2/jline/jline/0.9.91/jline-0.9.91.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/sbt_2.8.1/0.10.1/jars/sbt_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/main_2.8.1/0.10.1/jars/main_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/compiler-interface/0.10.1/jars/compiler-interface-bin.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/compiler-interface/0.10.1/jars/compiler-interface-src.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/precompiled-2_9_0-1/0.10.1/jars/compiler-interface-bin.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/actions_2.8.1/0.10.1/jars/actions_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/interface/0.10.1/jars/interface.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/io_2.8.1/0.10.1/jars/io_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/ivy_2.8.1/0.10.1/jars/ivy_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/launcher-interface_2.8.1/0.10.1/jars/launcher-interface_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/logging_2.8.1/0.10.1/jars/logging_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/process_2.8.1/0.10.1/jars/process_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/run_2.8.1/0.10.1/jars/run_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/classfile_2.8.1/0.10.1/jars/classfile_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/classpath_2.8.1/0.10.1/jars/classpath_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/incremental-compiler_2.8.1/0.10.1/jars/incremental-compiler_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/persist_2.8.1/0.10.1/jars/persist_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/compile_2.8.1/0.10.1/jars/compile_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/completion_2.8.1/0.10.1/jars/completion_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/discovery_2.8.1/0.10.1/jars/discovery_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/task-system_2.8.1/0.10.1/jars/task-system_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/tasks_2.8.1/0.10.1/jars/tasks_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/tracking_2.8.1/0.10.1/jars/tracking_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/testing_2.8.1/0.10.1/jars/testing_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/control_2.8.1/0.10.1/jars/control_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/collections_2.8.1/0.10.1/jars/collections_2.8.1.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/api_2.8.1/0.10.1/jars/api_2.8.1.jar ...
downloading http://repo1.maven.org/maven2/org/scala-tools/sbinary/sbinary_2.8.1/0.4.0/sbinary_2.8.1-0.4.0.jar ...
downloading http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar ...
downloading http://repo1.maven.org/maven2/com/jcraft/jsch/0.1.31/jsch-0.1.31.jar ...
downloading http://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar ...
downloading http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar ...
downloading http://repo1.maven.org/maven2/commons-codec/commons-codec/1.2/commons-codec-1.2.jar ...
downloading http://repo1.maven.org/maven2/jline/jline/0.9.94/jline-0.9.94.jar ...
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/cache_2.8.1/0.10.1/jars/cache_2.8.1.jar ...
downloading http://repo1.maven.org/maven2/org/scala-tools/testing/test-interface/0.5/test-interface-0.5.jar ...