[Nix-dev] Please provide input on my attempts to rework how NixOS builds images

[Daniel Peebles]

Hi all,

Eelco and I are having a slight disagreement on how to proceed on a PR of
mine and we figured we'd ask the community for opinions. If you've dealt
with images in Nix and/or virtual machines, please continue reading.

*tl;dr:* please give thumbs up or down on
https://github.com/NixOS/nixpkgs/pull/23026

Here's a quick summary of what's going on:

   - NixOS/nixpkgs has a remarkable amount of image building code spread
   throughout it, for building EC2/Google/Azure images, cd/dvd images,
   initramfs, and probably several others.
   - That code is generally copied and pasted a lot, but almost all of the
   image builders currently run the filesystem build inside a Linux VM; the
   usual process is:
      1. start Linux VM (because writing filesystems requires root and is
      mostly "impure")
      2. set up partitions & filesystems
      3. run nixos-install
      4. shut down VM
      5. save image to $out
      - Starting a Linux VM inside a Nix build isn't always possible or
   efficient. For one example, 95% of "the cloud" doesn't support nested
   virtualization, so running an image build on EC2 is fully emulated and
   takes a decimal order of magnitude or more longer than it does on physical
   hardware, due to the lack of hardware-assisted virtualization. There's no
   indication that any cloud providers will be be providing nested
   virtualization support anytime soon. A second example is that nested
   virtualization support in e.g., VMWare can be pretty flaky on Linux guests:
   I used to run nested virtualization fine inside NixOS but some recent Linux
   kernel changes now hang my system when I try it, and there doesn't seem to
   be any indication that there's active work on remedying such flakiness.
   - I build NixOS images *a lot* (mostly for EC2, on EC2) and don't have
   access to any physical Linux hardware.
   - I have devised a cute process
   <https://github.com/NixOS/nixpkgs/pull/21943> (using the Linux Kernel
   Library project) that can quickly and efficiently craft Linux filesystems
   using the upstream Linux kernel libraries directly, without a VM.

Here's what I'm proposing:

   1. Make the image builder code throughout NixOS (I'll also refactor and
   share more code between the image generators over the next few weeks) use
   LKL and do far less work in a VM, in order to build images efficiently on
   EC2 (image build time goes from 30+ minutes to <1 minute) and to build them
   at all on local virtualization. This means *the image builder code no
   longer calls nixos-install*.
   2. To avoid duplicating all the logic in nixos-install, factor out the
   common elements in the VM-free image builder and nixos-install into a third
   script, which I've been tentatively calling nixos-prepare-root, and have
   nixos-install and image builders both call that.

Eelco is understandably concerned that by forking the image build and
making it take a slightly different path from nixos-install (even if they
share most of the core filesystem build logic), we'll make maintenance
harder and make it easier to introduce subtle bugs. Although I agree with
the gist of that and am striving to minimize divergence from nixos-install,
I still need it to work on EC2 and VMWare.

Eelco has said he won't veto the PR merge (and my subsequent work) if the
community supports it, but doesn't particularly like it himself. That's
where you come in :) please check out my PR and let us know what you think.
I'd obviously love to merge this, but if the community really doesn't want
this change I can maintain some forked logic here for my needs.

Thanks,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.uu.nl/pipermail/nix-dev/attachments/20170413/98f64a8a/attachment.html>