[Nix-dev] cabal-install 1.24's Nix-style builds, and a new style of Nix + language package manager integration

[John Ericson]

After much pondering, I've decided the best way to work with
language-specific build managers is for them to come up with the end-to-end
build plan, and us to build each package in that build plan.

Our current method with Haskell of trying to replicate each package and all
the possible ways it can be configured just doesn't scale very well with
respect to configuaration options. Things like `if impl(ghcjs):` have long
remained unimplemented, and implementing them bakes more and more
build-plan resolution machinery into machine until we approach
re-implementing cabal-install. [Peti has discussed this in depth in
https://github.com/NixOS/cabal2nix/blob/master/doc/03-map-cabal-files-to-nix-without-information-loss.md
.]

At the same time, or current method isn't great for developing multiple
packages simultaneously. The default thing to do is make a package set with
each one, but if one doesn't handle the source very carefully, a change to
any of them will cause them all to be rebuilt. Furthermore, one needs
switch shells per each package---which is a pain in itself---and that
prevents the use of neat things like `stack repl` which interprets multiple
packages at once for seamless concurrent development. [I discussed part of
this in https://github.com/NixOS/nixpkgs/issues/10902 a bit.]

`stack --nix` at first glance seems to solve both of these areas, but in
fact comes with it's own set of downsides. Yes, since Stack manages the
build plan less knowledge is needed on the nix side, but stack also
*executes* the build plan. This means that both Nix's superior caching is
not utilized, and per-package patches (like foreign dependencies) are
unavailable. [See https://github.com/NixOS/nixpkgs/issues/15995.]

What I'm proposing should give us the best of both worlds in that language
package manager makes most of the decisions, but we still have an
opportunity to cache and patch, and development and deployment workflows
overlap as much as possible. This methodology could in principle work with
any language and its package manager, but I single out Cabal+Haskell
because the new Nix-style workflow (
http://cabal.readthedocs.io/en/latest/nix-local-build.html) makes it the
most ready.

For both deployment and development, the idea is we first run cabal-install
to compute a build plan. During this stage it in principle only needs
access to .cabal files / the Hackage index and a project config---certainly
not any already built binaries like today. Next we instruct cabal-install
to decompose the plan into "sub plans" for each dependency/would-be system
package---this is the biggest change needed for cabal-install, but one that
any system package manager could leverage, not just us. After that, using a
reworked cabal2nix, we convert each of those plans to a Nix
derivation---this may include turning hackage deps into local deps with
their source fetched by nix. Finally, each of those derivations can use
`cabal new-install` to install their root package, and the caches/package
DBs can be passed downstream and combined just like today.

There are two important concepts to note here: First, naively, the use of
entire sub plans in each Nix derivation would cause cabal-install to
rebuild each dependency for package in the plan that depends that needs it,
but since the Nix derivations will also depend on each other, `cabal
new-build`'s own caching should kick in so each Nix derivation does in fact
correspond to each Haskell package being built. Second, while cabal-install
can be instructed to build any package in a plan, the decomposition of
build plans is necessary so build plans that differ in ways that don't
affect common dependencies still end up sharing nix-executed builds. In
both cases, we rely on cabal-install's imitation of our methodology.

For development, we leverage the new nix-style build's preexisting notion
of a local vs external package. For convenience, it's probably best to let
cabal-install devise the build plan impurely (this is analogous to
non-restricted mode for Nix expr evaluation), and not copy any local
sources to the Nix store. Then, only external packages (and only those
which are dependencies of the current goal) should be built by Nix in
accordance with the plan above. In a shell with those built, cabal-install
can be run again building local packages normally and locally using its
more fine-grained per-file hashing and enabling tricks analogous to
`stack-repl`. While the default shell derivation will need tweaking if any
local packages have special requirements like foreign dependencies, this is
better than with `stack --nix` where the shell needs tweaking if *any*
dependency has such special requirements.
like today.

For deployment we don't need to worry about local vs external packages, and
on the cabal-install side `cabal new-install` should treat them all the
same IIUC. However, we do need to make sure that the build plan resolution
can be done in a pure environment: this means making sure all local
packages locations are in the Nix store and the Hackage index is
pre-downloaded by Nix. After that however no further complications arise.

There's a lot of work with nixpkgs and cabal2nix required for all this, but
comparatively little upstream for cabal-install. And the end result I
believe is hugely worthwhile, solving just about every pain point with our
existing infrastructure.

John

N.B. not quite sure on the division of labor between Cabal and
cabal-install with the new system, and erring on the side of cabal-install.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160918/67176f23/attachment.html>