[Nix-dev] Re: [Nix-commits] SVN commit: nix - 22282 - MarcWeber - nixpkgs/branches/stdenv-updates/pkgs/stdenv/generic

[Ludovic Courtès]

Hello,

Peter Simons <simons at cryp.to> writes:

>  > 1. Hydra does not need the in-build parallelization, she has enough
>  > with the inter-build parallelization
>
> I don't use pre-built binaries from Hydra, I don't feel strongly about this
> topic. However, I would really like to see this claim backed up by data. I
> very much doubt that Hydra could not benefit from build parallelism.

My intuition is that there are a few bottlenecks, such as stdenv.
However, beyond that, I feel there are quite enough independent jobs to
keep the build farm busy.

(I’m actually starting to study this as part of my day job, so I hope to
have concrete data sometime this summer.)

>  > 2. There is no way for nix to manage the balance between in-build and
>  > inter-build parallelization (make -j, and nix max-jobs).
>
> I think that this claim is inaccurate. GNU Make's "-l" flag does provide
> means for ensuring a sensible system load, and there are other means to
> accomplish the same thing.

Right.

>  > 3. Nix developers, wanting a quick build for testing, can add the "make -j X"
>  > temporarily before commiting the expression.
>
> Adding "make -j X" temporarily does not solve the problem that certain
> builds take ages (OpenOffice, boost, gcc, qt, firefox etc.) when performed
> on a single core.

That builds take ages doesn’t matter as long as you’re getting pre-built
binaries.  For those who don’t use pre-built binaries, the problem is
the same as for Hydra (above).

>  > 4. The benefits of your change come at the expense of some clarity in
>  > stdenv, and some impurities, and in the nix world many don't like
>  > impurities much.
>
> I agree that impurities are undesirable in general. In this particular case,
> however, I don't see a problem. Passing the number of available cores to
> Make is a fairly insignificant impurity that allows for fairly significant
> benefits, i.e. significantly reduced build times.

It’s impossible to know with certainty whether a package is
parallel-build-safe, i.e., deterministic regardless of the level of
parallelism.  Thus, its build output can depend on build task
scheduling, in which case passing the number of cores is indeed an
impurity.

>  > 5. By default, generic builder expressions should not be built parallelizing at
>  > all.
>
> Yes, this is true. It was a mistake to implement this patch using an opt-out
> scheme. Clearly, opt-in is what we want.

+1

>  > I'm for the revert, considering what I wrote above.
>
> I disagree with most of the points you made. However, I am in favor of
> reverting the patch, too, because I'm not particularly fond of the
> implementation -- especially of the choice to implement opt-out.

+1

Thanks,
Ludo’.