[Nix-dev] Going through hell with systemd timers
4levels
4levels at gmail.com
Thu Jan 28 10:59:43 CET 2016
Hi Zimbatm (is that your name :-)
I'm currently trying to debug by using printenv to view the differences in
both environments. I'm not yet familiar with using nix-shell :-s
But since I cannot even deploy or rebuild switch anymore (see my other
email to this list) I'm pretty stuck.
I'm currently repartitioning and reinstalling the Vultr machines (*sigh*)
to see if that brings any changes..
Thank you for your pointer to the environment vars!
Erik
On Thu, Jan 28, 2016 at 10:56 AM zimbatm <zimbatm at zimbatm.com> wrote:
> One common error with system services are missing environment variable.
> When testing with your shell you will have $HOME set for example.
>
> On Thu, 28 Jan 2016 09:43 4levels <4levels at gmail.com> wrote:
>
>> Hi Exi,
>>
>> thank you for your reply.
>>
>> This is the timers config I'm using (note that I'm starting this every 5
>> minutes to troubleshoot, is supposed to run every 2 hours or so)
>>
>> backup = {
>> description = "Backup service";
>> after = [ "network.target" "mysql.target" ];
>> path = [ pkgs.procps pkgs.gawk pkgs.nettools pkgs.mysql pkgs.php pkgs.duplicity pkgs.postfix ];
>> script =
>> ''
>> ./s3Backup.sh
>> '';
>> startAt = "*-*-* *:0/5:00";
>>
>>
>>
>> And the contents of the s3Backup.sh script:
>>
>> s3Backup = name:
>> ''
>> #!${pkgs.bash}/bin/bash
>>
>> ${builtins.readFile ./src/envrc}
>>
>> # Your GPG key
>> GPG_KEY=
>>
>> # export PATH="$PATH:/var/setuid-wrappers:/run/current-system/sw/bin:/run/current-system/sw/sbin"
>>
>> # Set up some variables for logging
>> LOGFILE="/var/lib/projects/${name}/log/duplicity-backup.log"
>> DAILYLOGFILE="/var/lib/projects/${name}/log/duplicity-backup.daily.log"
>> FULLBACKLOGFILE="/var/lib/projects/${name}/log/duplicity-backup.full.log"
>> HOST=`hostname`
>> DATE=`date +%Y-%m-%d`
>> MAILADDR="dev at domain.com"
>> TODAY=$(date +%d%m%Y)
>>
>> # The S3 destination followed by bucket name
>> DEST="s3://s3.amazonaws.com/projects-backup-eu-west/${name} <http://s3.amazonaws.com/projects-backup-eu-west/$%7Bname%7D>"
>>
>> is_running=$(ps -ef | grep duplicity | grep python | wc -l)
>>
>> if [ ! -f $FULLBACKLOGFILE ]; then
>> touch $FULLBACKLOGFILE
>> fi
>>
>> if [ $is_running -eq 0 ]; then
>> # Clear the old daily log file
>> cat /dev/null > ''${DAILYLOGFILE}
>>
>> # Trace function for logging, don't change this
>> trace () {
>> stamp=`date +%Y-%m-%d_%H:%M:%S`
>> echo "$stamp: $*" >> ''${DAILYLOGFILE}
>> }
>>
>> # Dump $PATH
>> trace "Current PATH: $PATH"
>>
>> # How long to keep backups for
>> OLDER_THAN="1M"
>>
>> # The source of your backup
>> SOURCE=/var/lib/projects/${name}
>>
>> FULL=
>> tail -1 ''${FULLBACKLOGFILE} | grep ''${TODAY} > /dev/null
>> if [ $? -ne 0 -a $(date +%d) -eq 1 ]; then
>> FULL=full
>> fi;
>>
>> trace "Backup for local filesystem started"
>>
>> trace "... removing old backups"
>>
>> duplicity remove-older-than ''${OLDER_THAN} ''${DEST} --s3-use-new-style >> ''${DAILYLOGFILE} 2>&1
>>
>> trace "... backing up filesystem"
>>
>> duplicity \
>> ''${FULL} \
>> -v9 \
>> --s3-use-new-style --s3-european-buckets --no-encryption \
>> --include=/var/lib/projects/${name}/data/backup \
>> --exclude=/** \
>> --allow-source-mismatch \
>> ''${SOURCE} ''${DEST} >> ''${DAILYLOGFILE} 2>&1
>>
>> trace "Backup for local filesystem complete"
>> trace "------------------------------------"
>>
>> # Send the daily log file by email
>> BACKUPSTATUS=`cat "$DAILYLOGFILE" | grep Errors | awk '{ print $2 }'`
>> if [ "$BACKUPSTATUS" != "0" ]; then
>> echo -e "Subject: Duplicity Backup Log for $HOST - $DATE - ${name}\n\n$(cat $DAILYLOGFILE)" | sendmail $MAILADDR
>> elif [ "$FULL" = "full" ]; then
>> echo "$(date +%d%m%Y_%T) Full Back Done" >> $FULLBACKLOGFILE
>> fi
>>
>> # Append the daily log file to the main log file
>> cat "$DAILYLOGFILE" >> $LOGFILE
>>
>> fi
>>
>> unset AWS_ACCESS_KEY_ID
>> unset AWS_SECRET_ACCESS_KEY
>> unset PASSPHRASE
>> '';
>>
>>
>>
>> On Thu, Jan 28, 2016 at 10:34 AM exi <e-nixos at wthack.de> wrote:
>>
>>> Hi Erik,
>>>
>>> does duplicity use an ssh connection? Does it depend on your ssh
>>> passphrase to be present? Do you use a ssh agent?
>>>
>>> "BackendException" from the traceback looks more like a connection issue
>>> than a nix issue.
>>> Which user is running the timer command?
>>> Could you post your timer config?
>>>
>>> Regards,
>>>
>>> exi
>>>
>>>
>>> On 28.01.2016 09:07, 4levels wrote:
>>>
>>> Hi Nix-Devs,
>>>
>>> yesterday I came to a point of really wanting to break something out of
>>> sheer frustration over failing systemd timer calls.
>>>
>>> I've setup a duplicity backup script over s3 that works flawlessly when
>>> invoked from terminal, but fails misrably when being called from a timer.
>>>
>>> I've tried everything I know, including but not limited to adding my
>>> full user $PATH to the script, adding all possible related packages to the
>>> path directive, .. nothing seems to work.
>>>
>>> The duplicity error is very vague (BackendException) and when adding
>>> maximum verbosity to the duplicity call ( -v9 ) I do get some error which
>>> seems to be related to a very old duplicity bug. Since duplicity uses
>>> python (the version I could trace seems to be 2.7) with python-boto for the
>>> s3 backend - the issue seems to be related to this, but I can't figure out
>>> what could be the reason since all required packages are installed and
>>> operational from the commandline.
>>>
>>> Has anyone experience with running python-based code in systemd timer
>>> calls (without being bitten)?
>>>
>>> On top of that, Github went down for a couple of hours last night and to
>>> make things even worse, NixOps cannot finish a deploy on any of the 5
>>> machines I'm managing with it anymore, with a vague error message:
>>>
>>> v-ams02...> updating GRUB 2 menu...
>>> v-ams02...> Died at
>>> /nix/var/nix/profiles/system/bin/switch-to-configuration line 264.
>>> v-ams02...> error: unable to activate new configuration
>>>
>>> Kind regards.
>>>
>>> Erik
>>>
>>> Duplicity error with maximum verbosity:
>>> Backend error detail: Traceback (most recent call last):
>>> File
>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
>>> line 1519, in <module>
>>> with_tempdir(main)
>>> File
>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
>>> line 1513, in with_tempdir
>>> fn()
>>> File
>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/bin/.duplicity-wrapped",
>>> line 1354, in main
>>> action = commandline.ProcessCommandLine(sys.argv[1:])
>>> File
>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/commandline.py",
>>> line 1070, in ProcessCommandLine
>>> backup, local_pathname = set_backend(args[0], args[1])
>>> File
>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/commandline.py",
>>> line 961, in set_backend
>>> globals.backend = backend.get_backend(bend)
>>> File
>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backend.py",
>>> line 223, in get_backend
>>> obj = get_backend_object(url_string)
>>> File
>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backend.py",
>>> line 209, in get_backend_object
>>> return factory(pu)
>>> File
>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backends/_boto_single.py",
>>> line 161, in __init__
>>> self.resetConnection()
>>> File
>>> "/nix/store/ap2bv0p5m8napigg7f6yciap4nm61ap8-duplicity-0.7.02/lib/python2.7/site-packages/duplicity/backends/_boto_single.py",
>>> line 187, in resetConnection
>>> raise BackendException(err.message)
>>>
>>>
>>>
>>> !DSPAM:56a9cc5f200881139745903!
>>>
>>>
>>>
>>> _______________________________________________
>>> nix-dev mailing listnix-dev at lists.science.uu.nlhttp://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>
>>>
>>> !DSPAM:56a9cc5f200881139745903!
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160128/8cd11d3b/attachment.html
More information about the nix-dev
mailing list