[Nix-dev] On npm2nix and the NPM package set in Nixpkgs

[Sander van der Burg]

Hello Nix and Node.js users,

I have been absent for a while in this discussion, but as far as I know the
state of the NPM packages in Nixpkgs is still quite bad and despite some
discussions on the mailing list we have not really come to any consensus
yet.

As some of you may know, I have my own re-engineered version of npm2nix
that lives in a specific branch in my own personal fork (
https://github.com/svanderburg/npm2nix/tree/reengineering2). A few months
ago, I did some major efforts in getting npm 3.x's behaviour supported,
which I have documented in this blog post:
http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html

I have been using this reengineering2 branch for all my public and some of
my private projects since the beginning of this year, and for me it seems
to work quite well, despite the fact that some of npm 3.x's flat module
installation oddities are still not accurately supported yet.

I also received a couple of reports from other people claiming that their
projects work and even encountered some people saying that it should
replace the current npm2nix. :)

Obviously, I do not want to claim that my implementation is the perfect
solution as it (for example) is much slower than the vanilla npm2nix, and
it composes the entire set of dependencies in one derivation as opposed to
generating a Nix store path per NPM dependency. (I do this for a very good
reason. For more details, please read my blog post).

Furthermore, I have also spoken to people that suggested completely
different kinds of approaches in getting NPM supported in a Nix environment.

Something that I have not done yet is investigating whether this
reengineered solution could be a potential replacement for the NPM packages
set in Nixpkgs.

Today, I have been working on an integration pattern, and the good news is:
it seems that I was able to generate Nix expressions for almost all
packages that are in pkgs/top-level/node-packages.json. The only exceptions
were the node-xmpp-* and bip-* packages, but some of them seem to have
broken dependencies, which is not npm2nix's fault.

If we would proceed integrating, we have a number of practical implications:

- I believe it is desired to have both Node.js 4.x and Node.js 5.x, 6.x
supported (I actually need all of them). To support all of these, we need
two different sets of generated Nix expressions. The former uses npm 2.x
with the classic dependency addressing approach and the latter uses npm 3.x
with flat module installations.
- I think most library packages should be removed from node-packages.json:
as explained in my blog post: how a package gets composed and to which
version a range resolve depends on the state of the includer. When somebody
wants their own NPM project to be deployed, he should use npm2nix directly
on package.json, and not refer to any NPM libraries in Nixpkgs.
- Some NPM packages must be overridden to provide native dependencies. The
mechanisms that the reengineering2 branch use are different. It would
probably take a bit of effort to get these migrated.

For example, this is how I override the webdrvr package to provide
phantomjs and the Selenium webdriver:

{pkgs, system}:

let
  nodePackages = import ./composition-v4.nix {
    inherit pkgs system;
  };
in
nodePackages // {
  webdrvr = nodePackages.webdrvr.override (oldAttrs: {
    buildInputs = oldAttrs.buildInputs ++ [ pkgs.phantomjs ];

    preRebuild = ''
      mkdir $TMPDIR/webdrvr

      ln -s ${pkgs.fetchurl {
        url = "
https://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar
";
        sha1 = "ef1b5f8ae9c99332f99ba8794988a1d5b974d27b";
      }} $TMPDIR/webdrvr/selenium-server-standalone-2.43.1.jar
      ln -s ${pkgs.fetchurl {
        url = "
http://chromedriver.storage.googleapis.com/2.10/chromedriver_linux64.zip";
        sha1 = "26220f7e43ee3c0d714860db61c4d0ecc9bb3d89";
      }} $TMPDIR/webdrvr/chromedriver_linux64.zip

    '';
  });
}


Although we have some practical issues, I think none of them would impose a
serious problem.

Then about npm2nix itself: Obviously, we could say that my version replaces
the upstream npm2nix and gets "blessed" into the new "official" version,
but I don't know whether everybody likes it.

Alternatively, we could be a bit more pragmatic: I stop calling my
reengineering2 version npm2nix, I give it a different name and I release it
as a different package. This makes it possible for those who want it, to
still use the 'vanilla' npm2nix alongside my version.

Then in Nixpkgs we can decide to:

- to keep npm2nix the default and provide my tool as a package
- or to make the reengineering2 version the default, and provide npm2nix as
a package
- in theory: support both package sets, but this might be a bit overkill :)

For those who don't know: although my repository is a fork of npm2nix, the
reengineering2 version is basically a rewrite of npm2nix and quite
different than the upstream version. It is written in JavaScript (as
opposed to CoffeeScript), has a different modular structure and different
command-line interface, so that's why I'm very careful in proposing to
replace the upstream npm2nix.

Moreover, it also does not share any git revision history with the upstream
npm2nix. :)

As a final note: for those who do not know about this: the reengineering2
tool can already be used outside Nixpkgs and this is what I have been doing
for all my projects. The expressions that it generates are based on the
principles I have described in this blog post:
http://sandervanderburg.blogspot.com/2014/07/managing-private-nix-packages-outside.html

My apologies for this very long email, but I'd like to have your feedback
and I don't want my preferences to disrupt other people's workflows.

What do you think?

Best,

Sander
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160622/9ece29d8/attachment.html>