[Nix-dev] On npm2nix and the NPM package set in Nixpkgs

Sander van der Burg svanderburg at gmail.com
Thu Jul 7 23:11:18 CEST 2016 

Well, for me personally it does not matter that much.

So far, I have only seen one +1 vote for making my version the new npm2nix.
However, not so long ago, I noticed that there was another incoming pull
request for the vanilla npm2nix. Perhaps the person who filed it, did not
know about the existence of the reengineering2 branch or this discussion
thread. :)

Anyway, the reason why I still haven't integrated anything yet, is this
issue: https://github.com/svanderburg/npm2nix/issues/7

When using the master Nixpkgs branch, it seems that npm2nix no longer
computes the sha256 hashes for the git checkouts correctly. Apparently,
some change in the fetchgit {} infrastructure (or a dependency thereof)
causes this, but I don't know why. When using the latest stable branch of
Nixpkgs (release-16.03) everything seems to work just fine.

I guess this issue needs to be resolved first. IMO it's a bit useless to
have NPM packages set in Nixpkgs with broken Git dependencies, even though
the master Nixpkgs should not be considered stable.

Besides npm2nix, other generators are affected as well -- bower2nix also
seems to break in one of my use cases. I haven't checked any other lang2nix
generators though :(


On Tue, Jul 5, 2016 at 12:35 PM, Rok Garbas <rok at garbas.si> wrote:

> we can still keep and old version of npm2nix in nixpkgs for ppl who use it.
> and also a branch with old code could be created, for people that want
> pudh bugfixes or develop further (very unlikely).
>
>
> On Tue, Jul 5, 2016 at 11:16 AM, Tomasz Czyż <tomasz.czyz at gmail.com>
> wrote:
> > Rok,
> >
> > what about people who are already using previous solution? Why break
> their
> > workflows?
> >
> > 2016-07-05 7:36 GMT+01:00 Rok Garbas <rok at garbas.si>:
> >>
> >> +1 for just keeping the name npm2nix and bumping up the version.
> >>
> >> i'm not using it on any active project, but i'm going to in the near
> >> future.
> >>
> >>
> >>
> >> On Mon, Jul 4, 2016 at 10:11 PM, Tobias Pflug <tobias.pflug at gmx.net>
> >> wrote:
> >> > Hi Sander,
> >> >
> >> > sorry for my very late response. I'll make this one brief as I am
> sadly
> >> > on
> >> > my phone.
> >> >
> >> > I  belong to one of those who tried your new npm2nix and in fact am
> >> > already
> >> > using it regularly. I am very much in favor of having your
> >> > re-engineeering2
> >> > branch replacing npm2nix as the de-facto node integration tool.
> >> >
> >> > I also definitely want to see the current set of auto-generated node
> >> > packages removed from nix. They are almost exclusively *totally*
> >> > outdated.
> >> >
> >> > Thank you a lot for your continued efforts on this. Working with
> >> > npm/node is
> >> > annoying but we are better off with your contributions.
> >> >
> >> > cheers,
> >> > Tobi
> >> >
> >> > On 22 Jun 2016, at 20:24, Sander van der Burg <svanderburg at gmail.com>
> >> > wrote:
> >> >
> >> > Hello Nix and Node.js users,
> >> >
> >> > I have been absent for a while in this discussion, but as far as I
> know
> >> > the
> >> > state of the NPM packages in Nixpkgs is still quite bad and despite
> some
> >> > discussions on the mailing list we have not really come to any
> consensus
> >> > yet.
> >> >
> >> > As some of you may know, I have my own re-engineered version of
> npm2nix
> >> > that
> >> > lives in a specific branch in my own personal fork
> >> > (https://github.com/svanderburg/npm2nix/tree/reengineering2). A few
> >> > months
> >> > ago, I did some major efforts in getting npm 3.x's behaviour
> supported,
> >> > which I have documented in this blog post:
> >> >
> >> >
> http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html
> >> >
> >> > I have been using this reengineering2 branch for all my public and
> some
> >> > of
> >> > my private projects since the beginning of this year, and for me it
> >> > seems to
> >> > work quite well, despite the fact that some of npm 3.x's flat module
> >> > installation oddities are still not accurately supported yet.
> >> >
> >> > I also received a couple of reports from other people claiming that
> >> > their
> >> > projects work and even encountered some people saying that it should
> >> > replace
> >> > the current npm2nix. :)
> >> >
> >> > Obviously, I do not want to claim that my implementation is the
> perfect
> >> > solution as it (for example) is much slower than the vanilla npm2nix,
> >> > and it
> >> > composes the entire set of dependencies in one derivation as opposed
> to
> >> > generating a Nix store path per NPM dependency. (I do this for a very
> >> > good
> >> > reason. For more details, please read my blog post).
> >> >
> >> > Furthermore, I have also spoken to people that suggested completely
> >> > different kinds of approaches in getting NPM supported in a Nix
> >> > environment.
> >> >
> >> > Something that I have not done yet is investigating whether this
> >> > reengineered solution could be a potential replacement for the NPM
> >> > packages
> >> > set in Nixpkgs.
> >> >
> >> > Today, I have been working on an integration pattern, and the good
> news
> >> > is:
> >> > it seems that I was able to generate Nix expressions for almost all
> >> > packages
> >> > that are in pkgs/top-level/node-packages.json. The only exceptions
> were
> >> > the
> >> > node-xmpp-* and bip-* packages, but some of them seem to have broken
> >> > dependencies, which is not npm2nix's fault.
> >> >
> >> > If we would proceed integrating, we have a number of practical
> >> > implications:
> >> >
> >> > - I believe it is desired to have both Node.js 4.x and Node.js 5.x,
> 6.x
> >> > supported (I actually need all of them). To support all of these, we
> >> > need
> >> > two different sets of generated Nix expressions. The former uses npm
> 2.x
> >> > with the classic dependency addressing approach and the latter uses
> npm
> >> > 3.x
> >> > with flat module installations.
> >> > - I think most library packages should be removed from
> >> > node-packages.json:
> >> > as explained in my blog post: how a package gets composed and to which
> >> > version a range resolve depends on the state of the includer. When
> >> > somebody
> >> > wants their own NPM project to be deployed, he should use npm2nix
> >> > directly
> >> > on package.json, and not refer to any NPM libraries in Nixpkgs.
> >> > - Some NPM packages must be overridden to provide native dependencies.
> >> > The
> >> > mechanisms that the reengineering2 branch use are different. It would
> >> > probably take a bit of effort to get these migrated.
> >> >
> >> > For example, this is how I override the webdrvr package to provide
> >> > phantomjs
> >> > and the Selenium webdriver:
> >> >
> >> > {pkgs, system}:
> >> >
> >> > let
> >> >   nodePackages = import ./composition-v4.nix {
> >> >     inherit pkgs system;
> >> >   };
> >> > in
> >> > nodePackages // {
> >> >   webdrvr = nodePackages.webdrvr.override (oldAttrs: {
> >> >     buildInputs = oldAttrs.buildInputs ++ [ pkgs.phantomjs ];
> >> >
> >> >     preRebuild = ''
> >> >       mkdir $TMPDIR/webdrvr
> >> >
> >> >       ln -s ${pkgs.fetchurl {
> >> >         url =
> >> >
> >> > "
> https://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar
> ";
> >> >         sha1 = "ef1b5f8ae9c99332f99ba8794988a1d5b974d27b";
> >> >       }} $TMPDIR/webdrvr/selenium-server-standalone-2.43.1.jar
> >> >       ln -s ${pkgs.fetchurl {
> >> >         url =
> >> >
> >> > "
> http://chromedriver.storage.googleapis.com/2.10/chromedriver_linux64.zip";
> >> >         sha1 = "26220f7e43ee3c0d714860db61c4d0ecc9bb3d89";
> >> >       }} $TMPDIR/webdrvr/chromedriver_linux64.zip
> >> >
> >> >     '';
> >> >   });
> >> > }
> >> >
> >> >
> >> > Although we have some practical issues, I think none of them would
> >> > impose a
> >> > serious problem.
> >> >
> >> > Then about npm2nix itself: Obviously, we could say that my version
> >> > replaces
> >> > the upstream npm2nix and gets "blessed" into the new "official"
> version,
> >> > but
> >> > I don't know whether everybody likes it.
> >> >
> >> > Alternatively, we could be a bit more pragmatic: I stop calling my
> >> > reengineering2 version npm2nix, I give it a different name and I
> release
> >> > it
> >> > as a different package. This makes it possible for those who want it,
> to
> >> > still use the 'vanilla' npm2nix alongside my version.
> >> >
> >> > Then in Nixpkgs we can decide to:
> >> >
> >> > - to keep npm2nix the default and provide my tool as a package
> >> > - or to make the reengineering2 version the default, and provide
> npm2nix
> >> > as
> >> > a package
> >> > - in theory: support both package sets, but this might be a bit
> overkill
> >> > :)
> >> >
> >> > For those who don't know: although my repository is a fork of npm2nix,
> >> > the
> >> > reengineering2 version is basically a rewrite of npm2nix and quite
> >> > different
> >> > than the upstream version. It is written in JavaScript (as opposed to
> >> > CoffeeScript), has a different modular structure and different
> >> > command-line
> >> > interface, so that's why I'm very careful in proposing to replace the
> >> > upstream npm2nix.
> >> >
> >> > Moreover, it also does not share any git revision history with the
> >> > upstream
> >> > npm2nix. :)
> >> >
> >> > As a final note: for those who do not know about this: the
> >> > reengineering2
> >> > tool can already be used outside Nixpkgs and this is what I have been
> >> > doing
> >> > for all my projects. The expressions that it generates are based on
> the
> >> > principles I have described in this blog post:
> >> >
> >> >
> http://sandervanderburg.blogspot.com/2014/07/managing-private-nix-packages-outside.html
> >> >
> >> > My apologies for this very long email, but I'd like to have your
> >> > feedback
> >> > and I don't want my preferences to disrupt other people's workflows.
> >> >
> >> > What do you think?
> >> >
> >> > Best,
> >> >
> >> > Sander
> >> >
> >> > _______________________________________________
> >> > nix-dev mailing list
> >> > nix-dev at lists.science.uu.nl
> >> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >> >
> >> >
> >> > _______________________________________________
> >> > nix-dev mailing list
> >> > nix-dev at lists.science.uu.nl
> >> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >> >
> >>
> >>
> >>
> >> --
> >> Rok Garbas
> >> http://www.garbas.si
> >> rok at garbas.si
> >> _______________________________________________
> >> nix-dev mailing list
> >> nix-dev at lists.science.uu.nl
> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
> >
> >
> >
> > --
> > Tomasz Czyż
>
>
>
> --
> Rok Garbas
> http://www.garbas.si
> rok at garbas.si
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160707/8ee96256/attachment-0001.html>

More information about the nix-dev mailing list