[Nix-dev] Hydra: infinite CPU and RAM consumption in evaluator

[Christian Theune]

Hahaha,

no, thanks. :)

Infinite is 32G at the moment and we’re on an older version. It’s one of our own changes, most likely this one:
https://github.com/flyingcircusio/nixpkgs/commit/2926129b8940e880a7745e44df511a3d6adacfa9 <https://github.com/flyingcircusio/nixpkgs/commit/2926129b8940e880a7745e44df511a3d6adacfa9>

I’m fighting against getting a very current version of Hydra running, as something blocks it from compiling. I’m guessing I need to override GCC to GCC6, but the overrideCC drops me in an infinite recursion (which doesn’t even look like an infinite recursion to me):

error: while evaluating the attribute ‘config.system.build.toplevel’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/activation/top-level.nix:246:5:
while evaluating ‘fold’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:29:19, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/activation/top-level.nix:127:12:
while evaluating ‘fold'’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:32:15, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:36:8:
while evaluating ‘showWarnings’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/activation/top-level.nix:93:18, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/activation/top-level.nix:100:16:
while evaluating ‘fold’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:29:19, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/activation/top-level.nix:93:23:
while evaluating ‘fold'’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:32:15, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:36:8:
while evaluating the attribute ‘warnings’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:199:44:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:74:45, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:199:52:
while evaluating the attribute ‘value’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:291:9:
while evaluating the option `warnings':
while evaluating the attribute ‘isDefined’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:323:5:
while evaluating ‘filterOverrides’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:395:21, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:307:18:
while evaluating ‘concatMap’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:79:18, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:401:8:
while evaluating ‘concatMap’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:79:18, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:302:17:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:302:28, called from undefined position:
while evaluating ‘dischargeProperties’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:365:25, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:303:62:
while evaluating the attribute ‘value’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:203:48:
while evaluating the attribute ‘config.warnings’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/boot/systemd.nix:637:5:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:224:10, called from undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/boot/systemd.nix:637:51, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:224:16:
while evaluating ‘optional’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:175:20, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/system/boot/systemd.nix:638:7:
while evaluating the attribute ‘serviceConfig.Type’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:199:44:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:74:45, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:199:52:
while evaluating the attribute ‘value’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:291:9:
while evaluating the option `systemd.services.hydra-evaluator.serviceConfig':
while evaluating the attribute ‘mergedValue’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:318:5:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:318:32, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:318:19:
while evaluating ‘merge’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/types.nix:140:20, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:321:8:
while evaluating ‘mapAttrs’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:198:17, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/types.nix:141:9:
while evaluating ‘filterAttrs’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:115:23, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/types.nix:141:35:
while evaluating ‘concatMap’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:79:18, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:116:18:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:116:29, called from undefined position:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/types.nix:141:51, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:116:62:
while evaluating the attribute ‘ExecStart’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:334:7:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/types.nix:141:86, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:334:15:
while evaluating the attribute ‘optionalValue’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:325:5:
while evaluating ‘filterOverrides’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:395:21, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:307:18:
while evaluating ‘concatMap’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:79:18, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:401:8:
while evaluating ‘concatMap’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:79:18, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:302:17:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:302:28, called from undefined position:
while evaluating ‘dischargeProperties’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:365:25, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:303:62:
while evaluating the attribute ‘value’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/types.nix:146:55:
while evaluating the attribute ‘ExecStart’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/nixos/modules/services/continuous-integration/hydra/default.nix:349:13:
while evaluating the attribute ‘package’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:199:44:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:74:45, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/attrsets.nix:199:52:
while evaluating the attribute ‘value’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:291:9:
while evaluating the option `services.hydra.package':
while evaluating the attribute ‘isDefined’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:323:5:
while evaluating ‘filterOverrides’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:395:21, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:307:18:
while evaluating ‘concatMap’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:79:18, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:401:8:
while evaluating ‘concatMap’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/lists.nix:79:18, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:302:17:
while evaluating anonymous function at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:302:28, called from undefined position:
while evaluating ‘dischargeProperties’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:365:25, called from /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:303:62:
while evaluating the attribute ‘value’ at /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs/lib/modules.nix:203:48:
while evaluating the attribute ‘config.package’ at /root/nixos/hosts/largo/configuration.nix:14:3:
while evaluating ‘callPackageWith’ at /nix/store/hq14617vvyd7npigrmmy6l0wlicpggix-nixos-16.09.1508.3909827/nixos/lib/customisation.nix:93:35, called from /root/nixos/hosts/largo/configuration.nix:3:15:
while evaluating ‘makeOverridable’ at /nix/store/hq14617vvyd7npigrmmy6l0wlicpggix-nixos-16.09.1508.3909827/nixos/lib/customisation.nix:54:24, called from /nix/store/hq14617vvyd7npigrmmy6l0wlicpggix-nixos-16.09.1508.3909827/nixos/lib/customisation.nix:97:8:
while evaluating anonymous function at /root/nixos/pkgs/hydra.nix:1:1, called from /nix/store/hq14617vvyd7npigrmmy6l0wlicpggix-nixos-16.09.1508.3909827/nixos/lib/customisation.nix:56:12:
while evaluating ‘nixBuild’ at /nix/store/hq14617vvyd7npigrmmy6l0wlicpggix-nixos-16.09.1508.3909827/nixos/pkgs/build-support/release/default.nix:25:14, called from /root/nixos/pkgs/hydra.nix:62:4:
while evaluating anonymous function at /nix/store/hq14617vvyd7npigrmmy6l0wlicpggix-nixos-16.09.1508.3909827/nixos/pkgs/build-support/release/nix-build.nix:8:1, called from /nix/store/hq14617vvyd7npigrmmy6l0wlicpggix-nixos-16.09.1508.3909827/nixos/pkgs/build-support/release/default.nix:25:20:
while evaluating ‘overrideCC’ at /nix/store/hq14617vvyd7npigrmmy6l0wlicpggix-nixos-16.09.1508.3909827/nixos/pkgs/stdenv/adapters.nix:11:24, called from /root/nixos/pkgs/hydra.nix:66:12:
infinite recursion encountered, at /nix/store/hq14617vvyd7npigrmmy6l0wlicpggix-nixos-16.09.1508.3909827/nixos/pkgs/stdenv/adapters.nix:11:28

Christian
> On 18 Jan 2017, at 18:03, Jean-Pierre <jean-pierre at prunetwork.fr> wrote:
> 
> Hello,
> 
> If infinite amount of memory is 1 TB (allocated memory) and process is ghc it’s not a bug … it’s expected behaviour.
> 
> haskel-packages.nix was updated recently.
> 
>> Le 18 janv. 2017 à 17:41, Christian Theune <ct at flyingcircus.io> a écrit :
>> 
>> Hi,
>> 
>> our Hydra exploded today: after a recent change to our nixpkgs branch, we encountered the evaluator consuming infinite CPU and quite a lot of memory.
>> 
>> Updating to a current version of Hydra (nixpkgs trunk) didn’t help. We’re going to go through the changes that caused this tomorrow.
>> 
>> However, I wanted to check whether anybody has experienced this before and might know what can trigger this behaviour. I also haven’t seen any changes in the last year to hydra that would seem relevant, apart from, maybe, the concurrent evaluator, which isn’t in nixpkgs master yet.
>> 
>> Christian
>> 
>> --
>> Christian Theune · ct at flyingcircus.io · +49 345 219401 0
>> Flying Circus Internet Operations GmbH · http://flyingcircus.io
>> Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
>> HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick
>> 
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> 

--
Christian Theune · ct at flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian. Theune, Christian. Zagrodnick

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20170118/140ecab1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: Message signed with OpenPGP
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20170118/140ecab1/attachment-0001.sig>