[Nix-dev] On npm2nix and the NPM package set in Nixpkgs
Michael Fellinger
m.fellinger at gmail.com
Tue Jul 5 11:56:49 CEST 2016
For what it’s worth, I’m using the re-engineeering2 branch standalone for
projects with hundreds of npm dependencies. The way I use it right now is
like this:
https://gist.github.com/manveru/20d22586d9dceae90930be528cbc49ce
Having it as a part of nixpkgs would be nice, but won’t really change how I
build and use it, and having npm dependencies listed in nixpkgs isn’t very
productive, the ecosystem for JS libraries changes so fast, that we’d have
to automatically update the index every day and hope nothing breaks, people
depending on js libraries listed in nixpkgs will always be frustrated.
I prefer having the checksums generated by npm2nix, and giving each app
that is packaged in nixpkgs their own list of dependencies generated by
npm2nix.
So for now, I think adding it under a different name to nixpkgs and
gradually changing things to use the new approach might be the best
solution.
There are two things that I’m still hoping could be done better: one is
that the location to the package.json should be more flexible, while you
can specify where it is located when running npm2nix, it won’t be found
later when it’s in a directory above the one you put the npm2nix-generated
files in. I opted for putting npm-generated files into their own
subdirectory, and then run
`ln -s "$(nix-build ./nix/npm.nix -A
package)/lib/node_modules/myapp/node_modules" node_modules`
Maybe there’s a better way, but that’s what I figured out on my own.
On 5 July 2016 at 11:17:05, Tomasz Czyż (tomasz.czyz at gmail.com) wrote:
Rok,
what about people who are already using previous solution? Why break their
workflows?
2016-07-05 7:36 GMT+01:00 Rok Garbas <rok at garbas.si>:
> +1 for just keeping the name npm2nix and bumping up the version.
>
> i'm not using it on any active project, but i'm going to in the near
> future.
>
>
>
> On Mon, Jul 4, 2016 at 10:11 PM, Tobias Pflug <tobias.pflug at gmx.net>
> wrote:
> > Hi Sander,
> >
> > sorry for my very late response. I'll make this one brief as I am sadly
> on
> > my phone.
> >
> > I belong to one of those who tried your new npm2nix and in fact am
> already
> > using it regularly. I am very much in favor of having your
> re-engineeering2
> > branch replacing npm2nix as the de-facto node integration tool.
> >
> > I also definitely want to see the current set of auto-generated node
> > packages removed from nix. They are almost exclusively *totally*
> outdated.
> >
> > Thank you a lot for your continued efforts on this. Working with
> npm/node is
> > annoying but we are better off with your contributions.
> >
> > cheers,
> > Tobi
> >
> > On 22 Jun 2016, at 20:24, Sander van der Burg <svanderburg at gmail.com>
> wrote:
> >
> > Hello Nix and Node.js users,
> >
> > I have been absent for a while in this discussion, but as far as I know
> the
> > state of the NPM packages in Nixpkgs is still quite bad and despite some
> > discussions on the mailing list we have not really come to any consensus
> > yet.
> >
> > As some of you may know, I have my own re-engineered version of npm2nix
> that
> > lives in a specific branch in my own personal fork
> > (https://github.com/svanderburg/npm2nix/tree/reengineering2). A few
> months
> > ago, I did some major efforts in getting npm 3.x's behaviour supported,
> > which I have documented in this blog post:
> >
> http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html
> >
> > I have been using this reengineering2 branch for all my public and some
> of
> > my private projects since the beginning of this year, and for me it
> seems to
> > work quite well, despite the fact that some of npm 3.x's flat module
> > installation oddities are still not accurately supported yet.
> >
> > I also received a couple of reports from other people claiming that their
> > projects work and even encountered some people saying that it should
> replace
> > the current npm2nix. :)
> >
> > Obviously, I do not want to claim that my implementation is the perfect
> > solution as it (for example) is much slower than the vanilla npm2nix,
> and it
> > composes the entire set of dependencies in one derivation as opposed to
> > generating a Nix store path per NPM dependency. (I do this for a very
> good
> > reason. For more details, please read my blog post).
> >
> > Furthermore, I have also spoken to people that suggested completely
> > different kinds of approaches in getting NPM supported in a Nix
> environment.
> >
> > Something that I have not done yet is investigating whether this
> > reengineered solution could be a potential replacement for the NPM
> packages
> > set in Nixpkgs.
> >
> > Today, I have been working on an integration pattern, and the good news
> is:
> > it seems that I was able to generate Nix expressions for almost all
> packages
> > that are in pkgs/top-level/node-packages.json. The only exceptions were
> the
> > node-xmpp-* and bip-* packages, but some of them seem to have broken
> > dependencies, which is not npm2nix's fault.
> >
> > If we would proceed integrating, we have a number of practical
> implications:
> >
> > - I believe it is desired to have both Node.js 4.x and Node.js 5.x, 6.x
> > supported (I actually need all of them). To support all of these, we need
> > two different sets of generated Nix expressions. The former uses npm 2.x
> > with the classic dependency addressing approach and the latter uses npm
> 3.x
> > with flat module installations.
> > - I think most library packages should be removed from
> node-packages.json:
> > as explained in my blog post: how a package gets composed and to which
> > version a range resolve depends on the state of the includer. When
> somebody
> > wants their own NPM project to be deployed, he should use npm2nix
> directly
> > on package.json, and not refer to any NPM libraries in Nixpkgs.
> > - Some NPM packages must be overridden to provide native dependencies.
> The
> > mechanisms that the reengineering2 branch use are different. It would
> > probably take a bit of effort to get these migrated.
> >
> > For example, this is how I override the webdrvr package to provide
> phantomjs
> > and the Selenium webdriver:
> >
> > {pkgs, system}:
> >
> > let
> > nodePackages = import ./composition-v4.nix {
> > inherit pkgs system;
> > };
> > in
> > nodePackages // {
> > webdrvr = nodePackages.webdrvr.override (oldAttrs: {
> > buildInputs = oldAttrs.buildInputs ++ [ pkgs.phantomjs ];
> >
> > preRebuild = ''
> > mkdir $TMPDIR/webdrvr
> >
> > ln -s ${pkgs.fetchurl {
> > url =
> > "
> https://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar
> ";
> > sha1 = "ef1b5f8ae9c99332f99ba8794988a1d5b974d27b";
> > }} $TMPDIR/webdrvr/selenium-server-standalone-2.43.1.jar
> > ln -s ${pkgs.fetchurl {
> > url =
> > "
> http://chromedriver.storage.googleapis.com/2.10/chromedriver_linux64.zip";
> > sha1 = "26220f7e43ee3c0d714860db61c4d0ecc9bb3d89";
> > }} $TMPDIR/webdrvr/chromedriver_linux64.zip
> >
> > '';
> > });
> > }
> >
> >
> > Although we have some practical issues, I think none of them would
> impose a
> > serious problem.
> >
> > Then about npm2nix itself: Obviously, we could say that my version
> replaces
> > the upstream npm2nix and gets "blessed" into the new "official" version,
> but
> > I don't know whether everybody likes it.
> >
> > Alternatively, we could be a bit more pragmatic: I stop calling my
> > reengineering2 version npm2nix, I give it a different name and I release
> it
> > as a different package. This makes it possible for those who want it, to
> > still use the 'vanilla' npm2nix alongside my version.
> >
> > Then in Nixpkgs we can decide to:
> >
> > - to keep npm2nix the default and provide my tool as a package
> > - or to make the reengineering2 version the default, and provide npm2nix
> as
> > a package
> > - in theory: support both package sets, but this might be a bit overkill
> :)
> >
> > For those who don't know: although my repository is a fork of npm2nix,
> the
> > reengineering2 version is basically a rewrite of npm2nix and quite
> different
> > than the upstream version. It is written in JavaScript (as opposed to
> > CoffeeScript), has a different modular structure and different
> command-line
> > interface, so that's why I'm very careful in proposing to replace the
> > upstream npm2nix.
> >
> > Moreover, it also does not share any git revision history with the
> upstream
> > npm2nix. :)
> >
> > As a final note: for those who do not know about this: the reengineering2
> > tool can already be used outside Nixpkgs and this is what I have been
> doing
> > for all my projects. The expressions that it generates are based on the
> > principles I have described in this blog post:
> >
> http://sandervanderburg.blogspot.com/2014/07/managing-private-nix-packages-outside.html
> >
> > My apologies for this very long email, but I'd like to have your feedback
> > and I don't want my preferences to disrupt other people's workflows.
> >
> > What do you think?
> >
> > Best,
> >
> > Sander
> >
> > _______________________________________________
> > nix-dev mailing list
> > nix-dev at lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
> >
> > _______________________________________________
> > nix-dev mailing list
> > nix-dev at lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
>
>
>
> --
> Rok Garbas
> http://www.garbas.si
> rok at garbas.si
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
--
Tomasz Czyż
_______________________________________________
nix-dev mailing list
nix-dev at lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160705/59d316c6/attachment.html>
More information about the nix-dev
mailing list