[Nix-dev] On npm2nix and the NPM package set in Nixpkgs

Mon Jul 4 17:51:14 CEST 2016

2016-07-04 16:34 GMT+01:00 Sander van der Burg <svanderburg at gmail.com>:

> So far only one response...
>
> I'm planning to implement the most pragmatic approach very soon -- due to
> lack of a better/cooler name I'll rename my fork of npm2nix to node2nix.
>
> Moreover, I will add a second attribute set to Nixpkgs allowing people to
> deploy packages that have been generated with node2nix. Also, I will take
> the original node-packages.json as a basis, but I will remove the library
> packages that I believe that should not be in there.
>
> Because the old package set will still be there, nobody should be
> disrupted and meanwhile people can try/test the new approach.
>
> Any objections?
>
Go for it! :-)

Btw, I'm not sure I'm the only one, but I usually use per project
dependencies. In general I think keeping this autogenerated stuff in repo
is not the way to go IMHO.

Most of the time I use nix packages as a building tools but I'm not using
requirements from it. I like all the "generators" to generate dependencies
for exactly my project, in that case it won't break stuff even if nixpkgs
will move forward. For example I would never relay on django version from
nix as I want this to be specific one and I want to change it when the
project is ready not at random time after some upgrade.

So I would definitely use node2nix to generate dependencies for my
projects, thank you Sander!

Tom

>
> Regards,
>
> Sander
>
>
> On Wed, Jun 22, 2016 at 11:39 PM, Tomasz Czyż <tomasz.czyz at gmail.com>
> wrote:
>
>> Hi Sander,
>>
>> awesome stuff.
>>
>> I would say, change name to something like node2nix and let's merge the
>> thing as it looks very good.
>>
>> Pros:
>> - backward compatibility
>> - process of merging will be lot faster (IMHO) as it will not collide
>> with anything and probably this will limit non productive discussions out
>>
>> Big thanks,
>> Tom
>>
>> 2016-06-22 19:24 GMT+01:00 Sander van der Burg <svanderburg at gmail.com>:
>>
>>> Hello Nix and Node.js users,
>>>
>>> I have been absent for a while in this discussion, but as far as I know
>>> the state of the NPM packages in Nixpkgs is still quite bad and despite
>>> some discussions on the mailing list we have not really come to any
>>> consensus yet.
>>>
>>> As some of you may know, I have my own re-engineered version of npm2nix
>>> that lives in a specific branch in my own personal fork (
>>> https://github.com/svanderburg/npm2nix/tree/reengineering2). A few
>>> months ago, I did some major efforts in getting npm 3.x's behaviour
>>> supported, which I have documented in this blog post:
>>> http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html
>>>
>>> I have been using this reengineering2 branch for all my public and some
>>> of my private projects since the beginning of this year, and for me it
>>> seems to work quite well, despite the fact that some of npm 3.x's flat
>>> module installation oddities are still not accurately supported yet.
>>>
>>> I also received a couple of reports from other people claiming that
>>> their projects work and even encountered some people saying that it should
>>> replace the current npm2nix. :)
>>>
>>> Obviously, I do not want to claim that my implementation is the perfect
>>> solution as it (for example) is much slower than the vanilla npm2nix, and
>>> it composes the entire set of dependencies in one derivation as opposed to
>>> generating a Nix store path per NPM dependency. (I do this for a very good
>>> reason. For more details, please read my blog post).
>>>
>>> Furthermore, I have also spoken to people that suggested completely
>>> different kinds of approaches in getting NPM supported in a Nix environment.
>>>
>>> Something that I have not done yet is investigating whether this
>>> reengineered solution could be a potential replacement for the NPM packages
>>> set in Nixpkgs.
>>>
>>> Today, I have been working on an integration pattern, and the good news
>>> is: it seems that I was able to generate Nix expressions for almost all
>>> packages that are in pkgs/top-level/node-packages.json. The only exceptions
>>> were the node-xmpp-* and bip-* packages, but some of them seem to have
>>> broken dependencies, which is not npm2nix's fault.
>>>
>>> If we would proceed integrating, we have a number of practical
>>> implications:
>>>
>>> - I believe it is desired to have both Node.js 4.x and Node.js 5.x, 6.x
>>> supported (I actually need all of them). To support all of these, we need
>>> two different sets of generated Nix expressions. The former uses npm 2.x
>>> with the classic dependency addressing approach and the latter uses npm 3.x
>>> with flat module installations.
>>> - I think most library packages should be removed from
>>> node-packages.json: as explained in my blog post: how a package gets
>>> composed and to which version a range resolve depends on the state of the
>>> includer. When somebody wants their own NPM project to be deployed, he
>>> should use npm2nix directly on package.json, and not refer to any NPM
>>> libraries in Nixpkgs.
>>> - Some NPM packages must be overridden to provide native dependencies.
>>> The mechanisms that the reengineering2 branch use are different. It would
>>> probably take a bit of effort to get these migrated.
>>>
>>> For example, this is how I override the webdrvr package to provide
>>> phantomjs and the Selenium webdriver:
>>>
>>> {pkgs, system}:
>>>
>>> let
>>>   nodePackages = import ./composition-v4.nix {
>>>     inherit pkgs system;
>>>   };
>>> in
>>> nodePackages // {
>>>   webdrvr = nodePackages.webdrvr.override (oldAttrs: {
>>>     buildInputs = oldAttrs.buildInputs ++ [ pkgs.phantomjs ];
>>>
>>>     preRebuild = ''
>>>       mkdir $TMPDIR/webdrvr
>>>
>>>       ln -s ${pkgs.fetchurl {
>>>         url = "
>>> https://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar
>>> ";
>>>         sha1 = "ef1b5f8ae9c99332f99ba8794988a1d5b974d27b";
>>>       }} $TMPDIR/webdrvr/selenium-server-standalone-2.43.1.jar
>>>       ln -s ${pkgs.fetchurl {
>>>         url = "
>>> http://chromedriver.storage.googleapis.com/2.10/chromedriver_linux64.zip
>>> ";
>>>         sha1 = "26220f7e43ee3c0d714860db61c4d0ecc9bb3d89";
>>>       }} $TMPDIR/webdrvr/chromedriver_linux64.zip
>>>
>>>     '';
>>>   });
>>> }
>>>
>>>
>>> Although we have some practical issues, I think none of them would
>>> impose a serious problem.
>>>
>>> Then about npm2nix itself: Obviously, we could say that my version
>>> replaces the upstream npm2nix and gets "blessed" into the new "official"
>>> version, but I don't know whether everybody likes it.
>>>
>>> Alternatively, we could be a bit more pragmatic: I stop calling my
>>> reengineering2 version npm2nix, I give it a different name and I release it
>>> as a different package. This makes it possible for those who want it, to
>>> still use the 'vanilla' npm2nix alongside my version.
>>>
>>> Then in Nixpkgs we can decide to:
>>>
>>> - to keep npm2nix the default and provide my tool as a package
>>> - or to make the reengineering2 version the default, and provide npm2nix
>>> as a package
>>> - in theory: support both package sets, but this might be a bit overkill
>>> :)
>>>
>>> For those who don't know: although my repository is a fork of npm2nix,
>>> the reengineering2 version is basically a rewrite of npm2nix and quite
>>> different than the upstream version. It is written in JavaScript (as
>>> opposed to CoffeeScript), has a different modular structure and different
>>> command-line interface, so that's why I'm very careful in proposing to
>>> replace the upstream npm2nix.
>>>
>>> Moreover, it also does not share any git revision history with the
>>> upstream npm2nix. :)
>>>
>>> As a final note: for those who do not know about this: the
>>> reengineering2 tool can already be used outside Nixpkgs and this is what I
>>> have been doing for all my projects. The expressions that it generates are
>>> based on the principles I have described in this blog post:
>>> http://sandervanderburg.blogspot.com/2014/07/managing-private-nix-packages-outside.html
>>>
>>> My apologies for this very long email, but I'd like to have your
>>> feedback and I don't want my preferences to disrupt other people's
>>> workflows.
>>>
>>> What do you think?
>>>
>>> Best,
>>>
>>> Sander
>>>
>>>
>>> _______________________________________________
>>> nix-dev mailing list
>>> nix-dev at lists.science.uu.nl
>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>
>>>
>>
>>
>> --
>> Tomasz Czyż
>>
>
>

-- 
Tomasz Czyż
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160704/309d7038/attachment-0001.html>