[Nix-dev] Using a remote machine for builds with two NixOS machines.

[Jonathan Glines]

2015-09-30 13:31 GMT-06:00  <roconnor at theorem.ca>:
> I have a slow laptop that needs a custom kernel.  It usually takes
> overnight to recompile a kernel, so to combat this, I've recently set up
> my laptop to use my desktop to perform builds.

I did this for a short time, but I ran into a problem where my
Sandybridge desktop was compiling AVR instructions that my AMD netbook
did not understand. This was troublesome for the few packages I
compiled all the time (Asterisk, VirtualBox), but I was also worried
that AVR instructions would sneak into other packages, so I stopped
using this.

I didn't have time to look into it at the time, but if someone knows
how Hydra avoids this problem (is it even using chips with AVR
instructions?) I would be very happy to hear about it. I tried adding
a few flags to gcc-wrapper (without messing with Asterisk's build
system) to try and convince it that using AVR instructions was bad,
but I never got it working.

I am very interested in this new documentation. I might try this again.

> Since this process isn't so well documented, I though I would try to
> make a record of what I did, or rather what I would do if I were doing
> this again.  I'll replace the contents of
> https://nixos.org/wiki/Distributed_build with this, if there are no
> complaints.
>
>
> Step 1.  Create and exchange signing keys.
>
> To generate signing keys I vaguely follow the directions in
> https://github.com/NixOS/nix/blob/master/doc/signing.txt
>
> As root I run
>
> # (umask 277 && `nix-build --no-out-link "<nixpkgs>" -A libressl`/bin/openssl genrsa -out /etc/nix/signing-key.sec 4096)
> # `nix-build --no-out-link "<nixpkgs>" -A libressl`/bin/Aopenssl rsa -in /etc/nix/signing-key.sec -pubout > /etc/nix/signing-key.pub
>
> This generates /etc/nix/signing-key.sec and /etc/nix/signing-key.pub.
> This step can be done on either the client or server.  The created files
> should be copied to the other machine.  Now both machines have copies of
> both the private and public signing keys.  These keys are used to share
> closures between the two machines, via nix-copy-closure --sign, without
> needing to necessarly use root accounts.
>
>
> Step 2. Generate ssh keys
>
> On the client (i.e. my laptop):
>
> # ssh-keygen -t ed25519 -N "" -C nixBuild -f /root/.ssh/id_nixBuild -N ""
>
> This will create both /root/.ssh/id_nixBuild and
> /root/.ssh/id_nixBuild.pub files.
>
>
> Step 3. Create a nixBuild account
>
> On the server (i.e. my desktop), update /etc/nixos/configuration.nix to
> add a new user
>
>    users.extraUsers.nixBuild = {
>      name = "nixBuild";
>      useDefaultShell = true;
>      openssh.authorizedKeys.keys = [ "ssh-ed25519 AAA... nixBuild" ];
>    };
>
> replacing the string in the openssh.authorizedKeys.keys with the contents
> of the /root/.ssh/id_nixBuild.pub file generated on the client machine.
> The client will access the server via this account in order to run Nix
> commands.
>
>
> Step 4. Enable distributed builds.
>
> On the client (i.e. my laptop), update /etc/nixos/configuration.nix to
> enable distributed builds.
>
>    nix = {
>      distributedBuilds = true;
>      buildMachines = [{
>        hostName = "<insert your server machine's name here>";
>        maxJobs = <fill in maxJobs>;
>        sshKey = "/root/.ssh/id_nixBuild";
>        sshUser = "nixBuild";
>        system = "x86_64-linux";
>      }];
>    };
>
> For the maxJobs field, run
>
> $ nixos-option nix.maxJobs
>
> on the server machine and use the "Value:" listed.
>
> For the system field, run
>
> $ nix-instantiate --eval -E 'builtins.currentSystem'
>
> on both the client and server machines.  They should both print the same
> value, and you should use that value in the system field.  You cannot use
> a remote machine with a different system for remote builds.
>
>
> And that is all.  For testing purposes, you can set the nix.maxJobs value
> in your client machines /etc/nixos/hardware-configuation.nix value
> temporarily to 0.  This will force all builds to be done remotely.  You
> can restore the value after you are done testing.
>
> It goes without saying that after updating configuration.nix files, you
> will need to run nixos-rebuild to realize the changes.
>
>
> Some general comments:  This setup means that both the client and server
> trust each other's binaries.  I'd prefer that only the client trust the
> server's binaries, and have the client only upload source files to the
> server to build; however I don't know how to force this.
>
> Also it would be nice to restrict the nixBuild user on the server to only
> run nix-commands.
>
> --
> Russell O'Connor                                      <http://r6.ca/>
> ``All talk about `theft,''' the general counsel of the American Graphophone
> Company wrote, ``is the merest claptrap, for there exists no property in
> ideas musical, literary or artistic, except as defined by statute.''
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev