[Nix-dev] Another approach at parallelizing in-build jobs

[Lluís Batlle i Rossell]

Hello,

Marc reverted his changes on stdenv for in-build parallelization, and I created
a nixpkgs branch for that. But some minutes later, Michael Raskin proposed the
idea of using substituters to achieve in-build parallelization.

As we have little means to achieve reproduceability once we have the parameters
"-j" with a number greater than one in 'make', substituters already have the
meaning of "you trust this? go on and accept the risk, if you want", instead of
affecting any package upstream.

I noticed that nix supports setting the substituters (NIX_SUBSTITUTERS env var),
but I think that the substituter lacks something to enable the in-build
parallelization: pass the '.drv' file for derivation output paths.

I've seen the nix code that, for manifest paths having a "Derive:" path, nix
does a "assertInStore(derivePath)". I propose the idea of giving the decision of
what to do with derive paths to the substituter (that very same check can be
in the download-from-manifests.pl file).

This way, even a normal nix build could be implemented as one of those new
substituters. The "multi-task-build" substituter should be able to build a store
path given a '.drv' that he may modify at will (change makeFlags, for example,
or check values in the .drv as "buildInParallel = false" - opt-out tricks for
that substituter).

What do you think?

Regards,
Lluís.