Table of Contents

1. Introduction
1.1. Overview of Nixpkgs
2. Quick Start to Adding a Package
3. The Standard Environment
3.1. Using stdenv
3.2. Tools provided by stdenv
3.3. Attributes
3.4. Phases
3.4.1. Controlling phases
3.4.2. The unpack phase
3.4.3. The patch phase
3.4.4. The configure phase
3.4.5. The build phase
3.4.6. The check phase
3.4.7. The install phase
3.4.8. The fixup phase
3.4.9. The installCheck phase
3.4.10. The distribution phase
3.5. Shell functions
3.6. Package setup hooks
3.7. Purity in Nixpkgs
3.8. Hardening in Nixpkgs
4. Multiple-output packages
4.1. Introduction
4.2. Installing a split package
4.3. Using a split package
4.4. Writing a split derivation
4.4.1. File type groups
4.4.2. Common caveats
5. Cross-compilation
5.1. Introduction
5.2. Packaging in a cross-friendly manner
5.2.1. Platform parameters
5.2.2. Specifying Dependencies
5.3. Cross-building packages
5.4. Cross-compilation infrastructure
6. Global configuration
6.1. Installing broken packages
6.2. Installing unfree packages
6.3. Installing insecure packages
6.4. Modify packages via packageOverrides
6.5. Declarative Package Management
6.5.1. Build an environment
6.5.2. Getting documentation
6.5.3. GNU info setup
7. Functions reference
7.1. Overriding
7.1.1. <pkg>.override
7.1.2. <pkg>.overrideAttrs
7.1.3. <pkg>.overrideDerivation
7.1.4. lib.makeOverridable
7.2. Generators
7.3. buildFHSUserEnv
7.4. pkgs.dockerTools
7.4.1. buildImage
7.4.2. pullImage
7.4.3. exportImage
7.4.4. shadowSetup
8. Meta-attributes
8.1. Standard meta-attributes
8.2. Licenses
9. Support for specific programming languages and frameworks
9.1. BEAM Languages (Erlang, Elixir & LFE)
9.1.1. Introduction
9.1.2. Structure
9.1.3. Build Tools
9.1.3.1. Rebar3
9.1.3.2. Mix & Erlang.mk
9.1.4. How to Install BEAM Packages
9.1.5. Packaging BEAM Applications
9.1.5.1. Erlang Applications
9.1.6. How to Develop
9.1.6.1. Accessing an Environment
9.1.6.2. Creating a Shell
9.1.7. Generating Packages from Hex with hex2nix
9.2. Bower
9.2.1. bower2nix usage
9.2.2. buildBowerComponents function
9.2.3. Troubleshooting
9.3. Coq
9.4. Go
9.5. User’s Guide to the Haskell Infrastructure
9.5.1. How to install Haskell packages
9.5.2. How to create a development environment
9.5.2.1. How to install a compiler
9.5.2.2. How to install a compiler with libraries
9.5.2.3. How to install a compiler with libraries, hoogle and documentation indexes
9.5.2.4. How to build a Haskell project using Stack
9.5.2.5. How to create ad hoc environments for nix-shell
9.5.3. How to create Nix builds for your own private Haskell packages
9.5.3.1. How to build a stand-alone project
9.5.3.2. How to build projects that depend on each other
9.5.4. Miscellaneous Topics
9.5.4.1. How to build with profiling enabled
9.5.4.2. How to override package versions in a compiler-specific package set
9.5.4.3. How to recover from GHC’s infamous non-deterministic library ID bug
9.5.4.4. Builds on Darwin fail with math.h not found
9.5.4.5. Builds using Stack complain about missing system libraries
9.5.4.6. Creating statically linked binaries
9.5.4.7. Building GHC with integer-simple
9.5.4.8. Quality assurance
9.5.5. Other resources
9.6. Idris packages
9.6.1. callPackage
9.6.2. builtins
9.6.3. build-idris-package
9.6.4. build-builtin-package
9.6.5. with-packages
9.7. Java
9.8. Lua
9.9. Node.js packages
9.10. Perl
9.10.1. Generation from CPAN
9.11. Python
9.11.1. User Guide
9.11.1.1. Using Python
9.11.1.2. Developing with Python
9.11.1.3. Organising your packages
9.11.1.4. Including a derivation using callPackage
9.11.2. Reference
9.11.2.1. Interpreters
9.11.2.2. Building packages and applications
9.11.2.3. Development mode
9.11.2.4. Tools
9.11.2.5. Deterministic builds
9.11.2.6. Automatic tests
9.11.3. FAQ
9.11.3.1. How to solve circular dependencies?
9.11.3.2. How to override a Python package?
9.11.3.3. python setup.py bdist_wheel cannot create .whl
9.11.3.4. install_data / data_files problems
9.11.3.5. Rationale of non-existent global site-packages
9.11.3.6. How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?
9.11.3.7. How to override a Python package from configuration.nix?
9.11.3.8. How to override a Python package using overlays?
9.11.4. Contributing
9.11.4.1. Contributing guidelines
9.12. Qt
9.12.1. Packaging Libraries for Nixpkgs
9.12.2. Packaging Applications for Nixpkgs
9.13. R packages
9.13.1. Installation
9.13.2. RStudio
9.13.3. Updating the package set
9.13.4. Testing if the Nix-expression could be evaluated
9.14. Ruby
9.15. User’s Guide to the Rust Infrastructure
9.15.1. Packaging Rust applications
9.15.2. Using the Rust nightlies overlay
9.16. TeX Live
9.16.1. User's guide
9.16.2. Known problems
9.17. User’s Guide to Vim Plugins/Addons/Bundles/Scripts in Nixpkgs
9.17.1. Custom configuration
9.17.2. Vim packages
9.17.3. VAM
9.17.3.1. dependencies by Vim plugins
9.17.3.2. Example
9.17.4. Important repositories
10. Package Notes
10.1. Linux kernel
10.2. X.org
10.3. Eclipse
10.4. Elm
10.5. Interactive shell helpers
10.6. Steam
10.6.1. Steam in Nix
10.6.2. How to play
10.6.3. Troubleshooting
10.6.4. steam-run
10.7. Emacs
10.7.1. Configuring Emacs
11. Overlays
11.1. Installing overlays
11.2. Defining overlays
12. Coding conventions
12.1. Syntax
12.2. Package naming
12.3. File naming and organisation
12.3.1. Hierarchy
12.3.2. Versioning
12.4. Fetching Sources
12.5. Patches
13. Submitting changes
13.1. Making patches
13.2. Submitting changes
13.3. Pull Request Template
13.3.1. Tested using sandboxing
13.3.2. Built on platform(s)
13.3.3. Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
13.3.4. Tested compilation of all pkgs that depend on this change using nox-review
13.3.5. Tested execution of all binary files (usually in ./result/bin/)
13.3.6. Meets nixpkgs contribution standards
13.4. Hotfixing pull requests
13.5. Commit policy
13.5.1. Master branch
13.5.2. Staging branch
13.5.3. Stable release branches
14. Reviewing contributions
14.1. Package updates
14.2. New packages
14.3. Module updates
14.4. New modules
14.5. Other submissions
14.6. Merging pull-requests
15. Contributing to this documentation

Table of Contents

1.1. Overview of Nixpkgs

The Nix Packages collection (Nixpkgs) is a set of thousands of packages for the Nix package manager, released under a permissive MIT/X11 license. Packages are available for several platforms, and can be used with the Nix package manager on most GNU/Linux distributions as well as NixOS.

This manual primarily describes how to write packages for the Nix Packages collection (Nixpkgs). Thus it’s mainly for packagers and developers who want to add packages to Nixpkgs. If you like to learn more about the Nix package manager and the Nix expression language, then you are kindly referred to the Nix manual.

1.1. Overview of Nixpkgs

Nix expressions describe how to build packages from source and are collected in the nixpkgs repository. Also included in the collection are Nix expressions for NixOS modules. With these expressions the Nix package manager can build binary packages.

Packages, including the Nix packages collection, are distributed through channels. The collection is distributed for users of Nix on non-NixOS distributions through the channel nixpkgs. Users of NixOS generally use one of the nixos-* channels, e.g. nixos-16.03, which includes all packages and modules for the stable NixOS 16.03. The purpose of stable NixOS releases are generally only given security updates. More up to date packages and modules are available via the nixos-unstable channel.

Both nixos-unstable and nixpkgs follow the master branch of the Nixpkgs repository, although both do lag the master branch by generally a couple of days. Updates to a channel are distributed as soon as all tests for that channel pass, e.g. this table shows the status of tests for the nixpkgs channel.

The tests are conducted by a cluster called Hydra, which also builds binary packages from the Nix expressions in Nixpkgs for x86_64-linux, i686-linux and x86_64-darwin. The binaries are made available via a binary cache.

The current Nix expressions of the channels are available in the nixpkgs-channels repository, which has branches corresponding to the available channels.

To add a package to Nixpkgs:

  1. Checkout the Nixpkgs source tree:

    $ git clone git://github.com/NixOS/nixpkgs.git 
    $ cd nixpkgs

  2. Find a good place in the Nixpkgs tree to add the Nix expression for your package. For instance, a library package typically goes into pkgs/development/libraries/pkgname, while a web browser goes into pkgs/applications/networking/browsers/pkgname. See Section 12.3, “File naming and organisation” for some hints on the tree organisation. Create a directory for your package, e.g.

    $ mkdir pkgs/development/libraries/libfoo

  3. In the package directory, create a Nix expression — a piece of code that describes how to build the package. In this case, it should be a function that is called with the package dependencies as arguments, and returns a build of the package in the Nix store. The expression should usually be called default.nix.

    $ emacs pkgs/development/libraries/libfoo/default.nix
    $ git add pkgs/development/libraries/libfoo/default.nix

    You can have a look at the existing Nix expressions under pkgs/ to see how it’s done. Here are some good ones:

    Some notes:

    • All meta attributes are optional, but it’s still a good idea to provide at least the description, homepage and license.

    • You can use nix-prefetch-url (or similar nix-prefetch-git, etc) url to get the SHA-256 hash of source distributions. There are similar commands as nix-prefetch-git and nix-prefetch-hg available in nix-prefetch-scripts package.

    • A list of schemes for mirror:// URLs can be found in pkgs/build-support/fetchurl/mirrors.nix.

    The exact syntax and semantics of the Nix expression language, including the built-in function, are described in the Nix manual in the chapter on writing Nix expressions.

  4. Add a call to the function defined in the previous step to pkgs/top-level/all-packages.nix with some descriptive name for the variable, e.g. libfoo.

    $ emacs pkgs/top-level/all-packages.nix

    The attributes in that file are sorted by category (like “Development / Libraries”) that more-or-less correspond to the directory structure of Nixpkgs, and then by attribute name.

  5. To test whether the package builds, run the following command from the root of the nixpkgs source tree:

    $ nix-build -A libfoo

    where libfoo should be the variable name defined in the previous step. You may want to add the flag -K to keep the temporary build directory in case something fails. If the build succeeds, a symlink ./result to the package in the Nix store is created.

  6. If you want to install the package into your profile (optional), do

    $ nix-env -f . -iA libfoo

  7. Optionally commit the new package and open a pull request, or send a patch to https://groups.google.com/forum/#!forum/nix-devel.

The standard build environment in the Nix Packages collection provides an environment for building Unix packages that does a lot of common build tasks automatically. In fact, for Unix packages that use the standard ./configure; make; make install build interface, you don’t need to write a build script at all; the standard environment does everything automatically. If stdenv doesn’t do what you need automatically, you can easily customise or override the various build phases.

3.1. Using stdenv

To build a package with the standard environment, you use the function stdenv.mkDerivation, instead of the primitive built-in function derivation, e.g.

stdenv.mkDerivation {
  name = "libfoo-1.2.3";
  src = fetchurl {
    url = http://example.org/libfoo-1.2.3.tar.bz2;
    sha256 = "0x2g1jqygyr5wiwg4ma1nd7w4ydpy82z9gkcv8vh2v8dn3y58v5m";
  };
}

(stdenv needs to be in scope, so if you write this in a separate Nix expression from pkgs/all-packages.nix, you need to pass it as a function argument.) Specifying a name and a src is the absolute minimum you need to do. Many packages have dependencies that are not provided in the standard environment. It’s usually sufficient to specify those dependencies in the buildInputs attribute:

stdenv.mkDerivation {
  name = "libfoo-1.2.3";
  ...
  buildInputs = [libbar perl ncurses];
}

This attribute ensures that the bin subdirectories of these packages appear in the PATH environment variable during the build, that their include subdirectories are searched by the C compiler, and so on. (See Section 3.6, “Package setup hooks” for details.)

Often it is necessary to override or modify some aspect of the build. To make this easier, the standard environment breaks the package build into a number of phases, all of which can be overridden or modified individually: unpacking the sources, applying patches, configuring, building, and installing. (There are some others; see Section 3.4, “Phases”.) For instance, a package that doesn’t supply a makefile but instead has to be compiled “manually” could be handled like this:

stdenv.mkDerivation {
  name = "fnord-4.5";
  ...
  buildPhase = ''
    gcc foo.c -o foo
  '';
  installPhase = ''
    mkdir -p $out/bin
    cp foo $out/bin
  '';
}

(Note the use of ''-style string literals, which are very convenient for large multi-line script fragments because they don’t need escaping of " and \, and because indentation is intelligently removed.)

There are many other attributes to customise the build. These are listed in Section 3.3, “Attributes”.

While the standard environment provides a generic builder, you can still supply your own build script:

stdenv.mkDerivation {
  name = "libfoo-1.2.3";
  ...
  builder = ./builder.sh;
}

where the builder can do anything it wants, but typically starts with

source $stdenv/setup

to let stdenv set up the environment (e.g., process the buildInputs). If you want, you can still use stdenv’s generic builder:

source $stdenv/setup

buildPhase() {
  echo "... this is my custom build phase ..."
  gcc foo.c -o foo
}

installPhase() {
  mkdir -p $out/bin
  cp foo $out/bin
}

genericBuild

3.2. Tools provided by stdenv

The standard environment provides the following packages:

  • The GNU C Compiler, configured with C and C++ support.

  • GNU coreutils (contains a few dozen standard Unix commands).

  • GNU findutils (contains find).

  • GNU diffutils (contains diff, cmp).

  • GNU sed.

  • GNU grep.

  • GNU awk.

  • GNU tar.

  • gzip, bzip2 and xz.

  • GNU Make. It has been patched to provide nested output that can be fed into the nix-log2xml command and log2html stylesheet to create a structured, readable output of the build steps performed by Make.

  • Bash. This is the shell used for all builders in the Nix Packages collection. Not using /bin/sh removes a large source of portability problems.

  • The patch command.

On Linux, stdenv also includes the patchelf utility.

3.3. Attributes

Variables affecting stdenv initialisation

NIX_DEBUG

If set, stdenv will print some debug information during the build. In particular, the gcc and ld wrapper scripts will print out the complete command line passed to the wrapped tools.

Variables specifying dependencies

nativeBuildInputs

A list of dependencies used by the new derivation at build-time. I.e. these dependencies should not make it into the package's runtime-closure, though this is currently not checked. For each dependency dir, the directory dir/bin, if it exists, is added to the PATH environment variable. Other environment variables are also set up via a pluggable mechanism. For instance, if buildInputs contains Perl, then the lib/site_perl subdirectory of each input is added to the PERL5LIB environment variable. See Section 3.6, “Package setup hooks” for details.

buildInputs

A list of dependencies used by the new derivation at run-time. Currently, the build-time environment is modified in the exact same way as with nativeBuildInputs. This is problematic in that when cross-compiling, foreign executables can clobber native ones on the PATH. Even more confusing is static-linking. A statically-linked library should be listed here because ultimately that generated machine code will be used at run-time, even though a derivation containing the object files or static archives will only be used at build-time. A less confusing solution to this would be nice.

propagatedNativeBuildInputs

Like nativeBuildInputs, but these dependencies are propagated: that is, the dependencies listed here are added to the nativeBuildInputs of any package that uses this package as a dependency. So if package Y has propagatedNativeBuildInputs = [X], and package Z has nativeBuildInputs = [Y], then package X will appear in Z’s build environment automatically.

propagatedBuildInputs

Like buildInputs, but propagated just like propagatedNativeBuildInputs. This inherits buildInputs's flaws of clobbering native executables when cross-compiling and being confusing for static linking.

Variables affecting build properties

enableParallelBuilding

If set, stdenv will pass specific flags to make and other build tools to enable parallel building with up to build-cores workers.

preferLocalBuild

If set, specifies that the package is so lightweight in terms of build operations (e.g. write a text file from a Nix string to the store) that there's no need to look for it in binary caches -- it's faster to just build it locally. It also tells Hydra and other facilities that this package doesn't need to be exported in binary caches (noone would use it, after all).

Special variables

passthru

This is an attribute set which can be filled with arbitrary values. For example:

passthru = {
  foo = "bar";
  baz = {
    value1 = 4;
    value2 = 5;
  };
}

Values inside it are not passed to the builder, so you can change them without triggering a rebuild. However, they can be accessed outside of a derivation directly, as if they were set inside a derivation itself, e.g. hello.baz.value1. We don't specify any usage or schema of passthru - it is meant for values that would be useful outside the derivation in other parts of a Nix expression (e.g. in other derivations). An example would be to convey some specific dependency of your derivation which contains a program with plugins support. Later, others who make derivations with plugins can use passed-through dependency to ensure that their plugin would be binary-compatible with built program.

3.4. Phases

The generic builder has a number of phases. Package builds are split into phases to make it easier to override specific parts of the build (e.g., unpacking the sources or installing the binaries). Furthermore, it allows a nicer presentation of build logs in the Nix build farm.

Each phase can be overridden in its entirety either by setting the environment variable namePhase to a string containing some shell commands to be executed, or by redefining the shell function namePhase. The former is convenient to override a phase from the derivation, while the latter is convenient from a build script. However, typically one only wants to add some commands to a phase, e.g. by defining postInstall or preFixup, as skipping some of the default actions may have unexpected consequences.

3.4.1. Controlling phases

There are a number of variables that control what phases are executed and in what order:

Variables affecting phase control

phases

Specifies the phases. You can change the order in which phases are executed, or add new phases, by setting this variable. If it’s not set, the default value is used, which is $prePhases unpackPhase patchPhase $preConfigurePhases configurePhase $preBuildPhases buildPhase checkPhase $preInstallPhases installPhase fixupPhase $preDistPhases distPhase $postPhases.

Usually, if you just want to add a few phases, it’s more convenient to set one of the variables below (such as preInstallPhases), as you then don’t specify all the normal phases.

prePhases

Additional phases executed before any of the default phases.

preConfigurePhases

Additional phases executed just before the configure phase.

preBuildPhases

Additional phases executed just before the build phase.

preInstallPhases

Additional phases executed just before the install phase.

preFixupPhases

Additional phases executed just before the fixup phase.

preDistPhases

Additional phases executed just before the distribution phase.

postPhases

Additional phases executed after any of the default phases.

3.4.2. The unpack phase

The unpack phase is responsible for unpacking the source code of the package. The default implementation of unpackPhase unpacks the source files listed in the src environment variable to the current directory. It supports the following files by default:

Tar files

These can optionally be compressed using gzip (.tar.gz, .tgz or .tar.Z), bzip2 (.tar.bz2 or .tbz2) or xz (.tar.xz or .tar.lzma).

Zip files

Zip files are unpacked using unzip. However, unzip is not in the standard environment, so you should add it to buildInputs yourself.

Directories in the Nix store

These are simply copied to the current directory. The hash part of the file name is stripped, e.g. /nix/store/1wydxgby13cz...-my-sources would be copied to my-sources.

Additional file types can be supported by setting the unpackCmd variable (see below).

Variables controlling the unpack phase

srcs / src

The list of source files or directories to be unpacked or copied. One of these must be set.

sourceRoot

After running unpackPhase, the generic builder changes the current directory to the directory created by unpacking the sources. If there are multiple source directories, you should set sourceRoot to the name of the intended directory.

setSourceRoot

Alternatively to setting sourceRoot, you can set setSourceRoot to a shell command to be evaluated by the unpack phase after the sources have been unpacked. This command must set sourceRoot.

preUnpack

Hook executed at the start of the unpack phase.

postUnpack

Hook executed at the end of the unpack phase.

dontMakeSourcesWritable

If set to 1, the unpacked sources are not made writable. By default, they are made writable to prevent problems with read-only sources. For example, copied store directories would be read-only without this.

unpackCmd

The unpack phase evaluates the string $unpackCmd for any unrecognised file. The path to the current source file is contained in the curSrc variable.

3.4.3. The patch phase

The patch phase applies the list of patches defined in the patches variable.

Variables controlling the patch phase

patches

The list of patches. They must be in the format accepted by the patch command, and may optionally be compressed using gzip (.gz), bzip2 (.bz2) or xz (.xz).

patchFlags

Flags to be passed to patch. If not set, the argument -p1 is used, which causes the leading directory component to be stripped from the file names in each patch.

prePatch

Hook executed at the start of the patch phase.

postPatch

Hook executed at the end of the patch phase.

3.4.4. The configure phase

The configure phase prepares the source tree for building. The default configurePhase runs ./configure (typically an Autoconf-generated script) if it exists.

Variables controlling the configure phase

configureScript

The name of the configure script. It defaults to ./configure if it exists; otherwise, the configure phase is skipped. This can actually be a command (like perl ./Configure.pl).

configureFlags

A list of strings passed as additional arguments to the configure script.

configureFlagsArray

A shell array containing additional arguments passed to the configure script. You must use this instead of configureFlags if the arguments contain spaces.

dontAddPrefix

By default, the flag --prefix=$prefix is added to the configure flags. If this is undesirable, set this variable to true.

prefix

The prefix under which the package must be installed, passed via the --prefix option to the configure script. It defaults to $out.

dontAddDisableDepTrack

By default, the flag --disable-dependency-tracking is added to the configure flags to speed up Automake-based builds. If this is undesirable, set this variable to true.

dontFixLibtool

By default, the configure phase applies some special hackery to all files called ltmain.sh before running the configure script in order to improve the purity of Libtool-based packages[1]. If this is undesirable, set this variable to true.

dontDisableStatic

By default, when the configure script has --enable-static, the option --disable-static is added to the configure flags.

If this is undesirable, set this variable to true.

configurePlatforms

By default, when cross compiling, the configure script has --build=... and --host=... passed. Packages can instead pass [ "build" "host" "target" ] or a subset to control exactly which platform flags are passed. Compilers and other tools should use this to also pass the target platform, for example. Note eventually these will be passed when in native builds too, to improve determinism: build-time guessing, as is done today, is a risk of impurity.

preConfigure

Hook executed at the start of the configure phase.

postConfigure

Hook executed at the end of the configure phase.

3.4.5. The build phase

The build phase is responsible for actually building the package (e.g. compiling it). The default buildPhase simply calls make if a file named Makefile, makefile or GNUmakefile exists in the current directory (or the makefile is explicitly set); otherwise it does nothing.

Variables controlling the build phase

dontBuild

Set to true to skip the build phase.

makefile

The file name of the Makefile.

makeFlags

A list of strings passed as additional flags to make. These flags are also used by the default install and check phase. For setting make flags specific to the build phase, use buildFlags (see below).

makeFlags = [ "PREFIX=$(out)" ];

Note: The flags are quoted in bash, but environment variables can be specified by using the make syntax.
makeFlagsArray

A shell array containing additional arguments passed to make. You must use this instead of makeFlags if the arguments contain spaces, e.g.

makeFlagsArray=(CFLAGS="-O0 -g" LDFLAGS="-lfoo -lbar")

Note that shell arrays cannot be passed through environment variables, so you cannot set makeFlagsArray in a derivation attribute (because those are passed through environment variables): you have to define them in shell code.

buildFlags / buildFlagsArray

A list of strings passed as additional flags to make. Like makeFlags and makeFlagsArray, but only used by the build phase.

preBuild

Hook executed at the start of the build phase.

postBuild

Hook executed at the end of the build phase.

You can set flags for make through the makeFlags variable.

Before and after running make, the hooks preBuild and postBuild are called, respectively.

3.4.6. The check phase

The check phase checks whether the package was built correctly by running its test suite. The default checkPhase calls make check, but only if the doCheck variable is enabled.

Variables controlling the check phase

doCheck

If set to a non-empty string, the check phase is executed, otherwise it is skipped (default). Thus you should set

doCheck = true;

in the derivation to enable checks.

makeFlags / makeFlagsArray / makefile

See the build phase for details.

checkTarget

The make target that runs the tests. Defaults to check.

checkFlags / checkFlagsArray

A list of strings passed as additional flags to make. Like makeFlags and makeFlagsArray, but only used by the check phase.

preCheck

Hook executed at the start of the check phase.

postCheck

Hook executed at the end of the check phase.

3.4.7. The install phase

The install phase is responsible for installing the package in the Nix store under out. The default installPhase creates the directory $out and calls make install.

Variables controlling the install phase

makeFlags / makeFlagsArray / makefile

See the build phase for details.

installTargets

The make targets that perform the installation. Defaults to install. Example:

installTargets = "install-bin install-doc";

installFlags / installFlagsArray

A list of strings passed as additional flags to make. Like makeFlags and makeFlagsArray, but only used by the install phase.

preInstall

Hook executed at the start of the install phase.

postInstall

Hook executed at the end of the install phase.

3.4.8. The fixup phase

The fixup phase performs some (Nix-specific) post-processing actions on the files installed under $out by the install phase. The default fixupPhase does the following:

  • It moves the man/, doc/ and info/ subdirectories of $out to share/.

  • It strips libraries and executables of debug information.

  • On Linux, it applies the patchelf command to ELF executables and libraries to remove unused directories from the RPATH in order to prevent unnecessary runtime dependencies.

  • It rewrites the interpreter paths of shell scripts to paths found in PATH. E.g., /usr/bin/perl will be rewritten to /nix/store/some-perl/bin/perl found in PATH.

Variables controlling the fixup phase

dontStrip

If set, libraries and executables are not stripped. By default, they are.

dontMoveSbin

If set, files in $out/sbin are not moved to $out/bin. By default, they are.

stripAllList

List of directories to search for libraries and executables from which all symbols should be stripped. By default, it’s empty. Stripping all symbols is risky, since it may remove not just debug symbols but also ELF information necessary for normal execution.

stripAllFlags

Flags passed to the strip command applied to the files in the directories listed in stripAllList. Defaults to -s (i.e. --strip-all).

stripDebugList

List of directories to search for libraries and executables from which only debugging-related symbols should be stripped. It defaults to lib bin sbin.

stripDebugFlags

Flags passed to the strip command applied to the files in the directories listed in stripDebugList. Defaults to -S (i.e. --strip-debug).

dontPatchELF

If set, the patchelf command is not used to remove unnecessary RPATH entries. Only applies to Linux.

dontPatchShebangs

If set, scripts starting with #! do not have their interpreter paths rewritten to paths in the Nix store.

forceShare

The list of directories that must be moved from $out to $out/share. Defaults to man doc info.

setupHook

A package can export a setup hook by setting this variable. The setup hook, if defined, is copied to $out/nix-support/setup-hook. Environment variables are then substituted in it using substituteAll.

preFixup

Hook executed at the start of the fixup phase.

postFixup

Hook executed at the end of the fixup phase.

separateDebugInfo

If set to true, the standard environment will enable debug information in C/C++ builds. After installation, the debug information will be separated from the executables and stored in the output named debug. (This output is enabled automatically; you don’t need to set the outputs attribute explicitly.) To be precise, the debug information is stored in debug/lib/debug/.build-id/XX/YYYY…, where XXYYYY… is the build ID of the binary — a SHA-1 hash of the contents of the binary. Debuggers like GDB use the build ID to look up the separated debug information.

For example, with GDB, you can add

set debug-file-directory ~/.nix-profile/lib/debug

to ~/.gdbinit. GDB will then be able to find debug information installed via nix-env -i.

3.4.9. The installCheck phase

The installCheck phase checks whether the package was installed correctly by running its test suite against the installed directories. The default installCheck calls make installcheck.

Variables controlling the installCheck phase

doInstallCheck

If set to a non-empty string, the installCheck phase is executed, otherwise it is skipped (default). Thus you should set

doInstallCheck = true;

in the derivation to enable install checks.

preInstallCheck

Hook executed at the start of the installCheck phase.

postInstallCheck

Hook executed at the end of the installCheck phase.

3.4.10. The distribution phase

The distribution phase is intended to produce a source distribution of the package. The default distPhase first calls make dist, then it copies the resulting source tarballs to $out/tarballs/. This phase is only executed if the attribute doDist is set.

Variables controlling the distribution phase

distTarget

The make target that produces the distribution. Defaults to dist.

distFlags / distFlagsArray

Additional flags passed to make.

tarballs

The names of the source distribution files to be copied to $out/tarballs/. It can contain shell wildcards. The default is *.tar.gz.

dontCopyDist

If set, no files are copied to $out/tarballs/.

preDist

Hook executed at the start of the distribution phase.

postDist

Hook executed at the end of the distribution phase.

3.5. Shell functions

The standard environment provides a number of useful functions.

makeWrapper executable wrapperfile args

Constructs a wrapper for a program with various possible arguments. For example:

# adds `FOOBAR=baz` to `$out/bin/foo`’s environment
makeWrapper $out/bin/foo $wrapperfile --set FOOBAR baz

# prefixes the binary paths of `hello` and `git`
# Be advised that paths often should be patched in directly
# (via string replacements or in `configurePhase`).
makeWrapper $out/bin/foo $wrapperfile --prefix PATH : ${lib.makeBinPath [ hello git ]}

There’s many more kinds of arguments, they are documented in nixpkgs/pkgs/build-support/setup-hooks/make-wrapper.sh.

wrapProgram is a convenience function you probably want to use most of the time.

substitute infile outfile subs

Performs string substitution on the contents of infile, writing the result to outfile. The substitutions in subs are of the following form:

--replace s1 s2

Replace every occurrence of the string s1 by s2.

--subst-var varName

Replace every occurrence of @varName@ by the contents of the environment variable varName. This is useful for generating files from templates, using @...@ in the template as placeholders.

--subst-var-by varName s

Replace every occurrence of @varName@ by the string s.

Example:

substitute ./foo.in ./foo.out \
    --replace /usr/bin/bar $bar/bin/bar \
    --replace "a string containing spaces" "some other text" \
    --subst-var someVar

substitute is implemented using the replace command. Unlike with the sed command, you don’t have to worry about escaping special characters. It supports performing substitutions on binary files (such as executables), though there you’ll probably want to make sure that the replacement string is as long as the replaced string.

substituteInPlace file subs

Like substitute, but performs the substitutions in place on the file file.

substituteAll infile outfile

Replaces every occurrence of @varName@, where varName is any environment variable, in infile, writing the result to outfile. For instance, if infile has the contents

#! @bash@/bin/sh
PATH=@coreutils@/bin
echo @foo@

and the environment contains bash=/nix/store/bmwp0q28cf21...-bash-3.2-p39 and coreutils=/nix/store/68afga4khv0w...-coreutils-6.12, but does not contain the variable foo, then the output will be

#! /nix/store/bmwp0q28cf21...-bash-3.2-p39/bin/sh
PATH=/nix/store/68afga4khv0w...-coreutils-6.12/bin
echo @foo@

That is, no substitution is performed for undefined variables.

Environment variables that start with an uppercase letter or an underscore are filtered out, to prevent global variables (like HOME) or private variables (like __ETC_PROFILE_DONE) from accidentally getting substituted. The variables also have to be valid bash “names”, as defined in the bash manpage (alphanumeric or _, must not start with a number).

substituteAllInPlace file

Like substituteAll, but performs the substitutions in place on the file file.

stripHash path

Strips the directory and hash part of a store path, outputting the name part to stdout. For example:

# prints coreutils-8.24
stripHash "/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24"

If you wish to store the result in another variable, then the following idiom may be useful:

name="/nix/store/9s9r019176g7cvn2nvcw41gsp862y6b4-coreutils-8.24"
someVar=$(stripHash $name)

wrapProgram executable makeWrapperArgs

Convenience function for makeWrapper that automatically creates a sane wrapper file It takes all the same arguments as makeWrapper, except for --argv0.

It cannot be applied multiple times, since it will overwrite the wrapper file.

3.6. Package setup hooks

The following packages provide a setup hook:

CC Wrapper

CC Wrapper wraps a C toolchain for a bunch of miscellaneous purposes. Specifically, a C compiler (GCC or Clang), Binutils (or the CCTools + binutils mashup when targetting Darwin), and a C standard library (glibc or Darwin's libSystem) are all fed in, and dependency finding, hardening (see below), and purity checks for each are handled by CC Wrapper. Packages typically depend on only CC Wrapper, instead of those 3 inputs directly.

Dependency finding is undoubtedly the main task of CC wrapper. It is currently accomplished by collecting directories of host-platform dependencies (i.e. buildInputs and nativeBuildInputs) in environment variables. CC wrapper's setup hook causes any include subdirectory of such a dependency to be added to NIX_CFLAGS_COMPILE, and any lib and lib64 subdirectories to NIX_LDFLAGS. The setup hook itself contains some lengthy comments describing the exact convoluted mechanism by which this is accomplished.

A final task of the setup hook is defining a number of standard environment variables to tell build systems which executables full-fill which purpose. They are defined to just be the base name of the tools, under the assumption that CC Wrapper's binaries will be on the path. Firstly, this helps poorly-written packages, e.g. ones that look for just gcc when CC isn't defined yet clang is to be used. Secondly, this helps packages not get confused when cross-compiling, in which case multiple CC wrappers may be simultaneous in use (targeting different platforms). BUILD_- and TARGET_-prefixed versions of the normal environment variable are defined for the additional CC Wrappers, properly disambiguating them.

A problem with this final task is that CC Wrapper is honest and defines LD as ld. Most packages, however, firstly use the C compiler for linking, secondly use LD anyways, defining it as the C compiler, and thirdly, only so define LD when it is undefined as a fallback. This triple-threat means CC Wrapper will break those packages, as LD is already defined as the actually linker which the package won't override yet doesn't want to use. The workaround is to define, just for the problematic package, LD as the C compiler. A good way to do this would be preConfigure = "LD=$CC".

Perl

Adds the lib/site_perl subdirectory of each build input to the PERL5LIB environment variable.

Python

Adds the lib/${python.libPrefix}/site-packages subdirectory of each build input to the PYTHONPATH environment variable.

pkg-config

Adds the lib/pkgconfig and share/pkgconfig subdirectories of each build input to the PKG_CONFIG_PATH environment variable.

Automake

Adds the share/aclocal subdirectory of each build input to the ACLOCAL_PATH environment variable.

Autoconf

The autoreconfHook derivation adds autoreconfPhase, which runs autoreconf, libtoolize and automake, essentially preparing the configure script in autotools-based builds.

libxml2

Adds every file named catalog.xml found under the xml/dtd and xml/xsl subdirectories of each build input to the XML_CATALOG_FILES environment variable.

teTeX / TeX Live

Adds the share/texmf-nix subdirectory of each build input to the TEXINPUTS environment variable.

Qt 4

Sets the QTDIR environment variable to Qt’s path.

gdk-pixbuf

Exports GDK_PIXBUF_MODULE_FILE environment variable the the builder. Add librsvg package to buildInputs to get svg support.

GHC

Creates a temporary package database and registers every Haskell build input in it (TODO: how?).

GStreamer

Adds the GStreamer plugins subdirectory of each build input to the GST_PLUGIN_SYSTEM_PATH_1_0 or GST_PLUGIN_SYSTEM_PATH environment variable.

paxctl

Defines the paxmark helper for setting per-executable PaX flags on Linux (where it is available by default; on all other platforms, paxmark is a no-op). For example, to disable secure memory protections on the executable foo:

      postFixup = ''
        paxmark m $out/bin/foo
      '';
    

The m flag is the most common flag and is typically required for applications that employ JIT compilation or otherwise need to execute code generated at run-time. Disabling PaX protections should be considered a last resort: if possible, problematic features should be disabled or patched to work with PaX.

3.7. Purity in Nixpkgs

[measures taken to prevent dependencies on packages outside the store, and what you can do to prevent them]

GCC doesn't search in locations such as /usr/include. In fact, attempts to add such directories through the -I flag are filtered out. Likewise, the linker (from GNU binutils) doesn't search in standard locations such as /usr/lib. Programs built on Linux are linked against a GNU C Library that likewise doesn't search in the default system locations.

3.8. Hardening in Nixpkgs

There are flags available to harden packages at compile or link-time. These can be toggled using the stdenv.mkDerivation parameters hardeningDisable and hardeningEnable.

Both parameters take a list of flags as strings. The special "all" flag can be passed to hardeningDisable to turn off all hardening. These flags can also be used as environment variables for testing or development purposes.

The following flags are enabled by default and might require disabling with hardeningDisable if the program to package is incompatible.

format

Adds the -Wformat -Wformat-security -Werror=format-security compiler options. At present, this warns about calls to printf and scanf functions where the format string is not a string literal and there are no format arguments, as in printf(foo);. This may be a security hole if the format string came from untrusted input and contains %n.

This needs to be turned off or fixed for errors similar to:

/tmp/nix-build-zynaddsubfx-2.5.2.drv-0/zynaddsubfx-2.5.2/src/UI/guimain.cpp:571:28: error: format not a string literal and no format arguments [-Werror=format-security]
         printf(help_message);
                            ^
cc1plus: some warnings being treated as errors
    
stackprotector

Adds the -fstack-protector-strong --param ssp-buffer-size=4 compiler options. This adds safety checks against stack overwrites rendering many potential code injection attacks into aborting situations. In the best case this turns code injection vulnerabilities into denial of service or into non-issues (depending on the application).

This needs to be turned off or fixed for errors similar to:

bin/blib.a(bios_console.o): In function `bios_handle_cup':
/tmp/nix-build-ipxe-20141124-5cbdc41.drv-0/ipxe-5cbdc41/src/arch/i386/firmware/pcbios/bios_console.c:86: undefined reference to `__stack_chk_fail'
    
fortify

Adds the -O2 -D_FORTIFY_SOURCE=2 compiler options. During code generation the compiler knows a great deal of information about buffer sizes (where possible), and attempts to replace insecure unlimited length buffer function calls with length-limited ones. This is especially useful for old, crufty code. Additionally, format strings in writable memory that contain '%n' are blocked. If an application depends on such a format string, it will need to be worked around.

Additionally, some warnings are enabled which might trigger build failures if compiler warnings are treated as errors in the package build. In this case, set NIX_CFLAGS_COMPILE to -Wno-error=warning-type.

This needs to be turned off or fixed for errors similar to:

malloc.c:404:15: error: return type is an incomplete type
malloc.c:410:19: error: storage size of 'ms' isn't known
    
strdup.h:22:1: error: expected identifier or '(' before '__extension__'
    
strsep.c:65:23: error: register name not specified for 'delim'
    
installwatch.c:3751:5: error: conflicting types for '__open_2'
    
fcntl2.h:50:4: error: call to '__open_missing_mode' declared with attribute error: open with O_CREAT or O_TMPFILE in second argument needs 3 arguments
    
pic

Adds the -fPIC compiler options. This options adds support for position independent code in shared libraries and thus making ASLR possible.

Most notably, the Linux kernel, kernel modules and other code not running in an operating system environment like boot loaders won't build with PIC enabled. The compiler will is most cases complain that PIC is not supported for a specific build.

This needs to be turned off or fixed for assembler errors similar to:

ccbLfRgg.s: Assembler messages:
ccbLfRgg.s:33: Error: missing or invalid displacement expression `private_key_len@GOTOFF'
    
strictoverflow

Signed integer overflow is undefined behaviour according to the C standard. If it happens, it is an error in the program as it should check for overflow before it can happen, not afterwards. GCC provides built-in functions to perform arithmetic with overflow checking, which are correct and faster than any custom implementation. As a workaround, the option -fno-strict-overflow makes gcc behave as if signed integer overflows were defined.

This flag should not trigger any build or runtime errors.

relro

Adds the -z relro linker option. During program load, several ELF memory sections need to be written to by the linker, but can be turned read-only before turning over control to the program. This prevents some GOT (and .dtors) overwrite attacks, but at least the part of the GOT used by the dynamic linker (.got.plt) is still vulnerable.

This flag can break dynamic shared object loading. For instance, the module systems of Xorg and OpenCV are incompatible with this flag. In almost all cases the bindnow flag must also be disabled and incompatible programs typically fail with similar errors at runtime.

bindnow

Adds the -z bindnow linker option. During program load, all dynamic symbols are resolved, allowing for the complete GOT to be marked read-only (due to relro). This prevents GOT overwrite attacks. For very large applications, this can incur some performance loss during initial load while symbols are resolved, but this shouldn't be an issue for daemons.

This flag can break dynamic shared object loading. For instance, the module systems of Xorg and PHP are incompatible with this flag. Programs incompatible with this flag often fail at runtime due to missing symbols, like:

intel_drv.so: undefined symbol: vgaHWFreeHWRec
    

The following flags are disabled by default and should be enabled with hardeningEnable for packages that take untrusted input like network services.

pie

Adds the -fPIE compiler and -pie linker options. Position Independent Executables are needed to take advantage of Address Space Layout Randomization, supported by modern kernel versions. While ASLR can already be enforced for data areas in the stack and heap (brk and mmap), the code areas must be compiled as position-independent. Shared libraries already do this with the pic flag, so they gain ASLR automatically, but binary .text regions need to be build with pie to gain ASLR. When this happens, ROP attacks are much harder since there are no static locations to bounce off of during a memory corruption attack.

For more in-depth information on these hardening flags and hardening in general, refer to the Debian Wiki, Ubuntu Wiki, Gentoo Wiki, and the Arch Wiki.




[1] It clears the sys_lib_*search_path variables in the Libtool script to prevent Libtool from using libraries in /usr/lib and such.

4.1. Introduction

The Nix language allows a derivation to produce multiple outputs, which is similar to what is utilized by other Linux distribution packaging systems. The outputs reside in separate nix store paths, so they can be mostly handled independently of each other, including passing to build inputs, garbage collection or binary substitution. The exception is that building from source always produces all the outputs.

The main motivation is to save disk space by reducing runtime closure sizes; consequently also sizes of substituted binaries get reduced. Splitting can be used to have more granular runtime dependencies, for example the typical reduction is to split away development-only files, as those are typically not needed during runtime. As a result, closure sizes of many packages can get reduced to a half or even much less.

Note: The reduction effects could be instead achieved by building the parts in completely separate derivations. That would often additionally reduce build-time closures, but it tends to be much harder to write such derivations, as build systems typically assume all parts are being built at once. This compromise approach of single source package producing multiple binary packages is also utilized often by rpm and deb.

4.2. Installing a split package

When installing a package via systemPackages or nix-env you have several options:

  • You can install particular outputs explicitly, as each is available in the Nix language as an attribute of the package. The outputs attribute contains a list of output names.

  • You can let it use the default outputs. These are handled by meta.outputsToInstall attribute that contains a list of output names.

    TODO: more about tweaking the attribute, etc.

  • NixOS provides configuration option environment.extraOutputsToInstall that allows adding extra outputs of environment.systemPackages atop the default ones. It's mainly meant for documentation and debug symbols, and it's also modified by specific options.

    Note: At this moment there is no similar configurability for packages installed by nix-env. You can still use approach from Section 6.4, “Modify packages via packageOverrides to override meta.outputsToInstall attributes, but that's a rather inconvenient way.

4.3. Using a split package

In the Nix language the individual outputs can be reached explicitly as attributes, e.g. coreutils.info, but the typical case is just using packages as build inputs.

When a multiple-output derivation gets into a build input of another derivation, the dev output is added if it exists, otherwise the first output is added. In addition to that, propagatedBuildOutputs of that package which by default contain $outputBin and $outputLib are also added. (See Section 4.4.1, “File type groups”.)

4.4. Writing a split derivation

Here you find how to write a derivation that produces multiple outputs.

In nixpkgs there is a framework supporting multiple-output derivations. It tries to cover most cases by default behavior. You can find the source separated in <nixpkgs/pkgs/build-support/setup-hooks/multiple-outputs.sh>; it's relatively well-readable. The whole machinery is triggered by defining the outputs attribute to contain the list of desired output names (strings).

outputs = [ "bin" "dev" "out" "doc" ];

Often such a single line is enough. For each output an equally named environment variable is passed to the builder and contains the path in nix store for that output. By convention, the first output should contain the executable programs provided by the package as that output is used by Nix in string conversions, allowing references to binaries like ${pkgs.perl}/bin/perl to always work. Typically you also want to have the main out output, as it catches any files that didn't get elsewhere.

Note: There is a special handling of the debug output, described at separateDebugInfo.

4.4.1. File type groups

The support code currently recognizes some particular kinds of outputs and either instructs the build system of the package to put files into their desired outputs or it moves the files during the fixup phase. Each group of file types has an outputFoo variable specifying the output name where they should go. If that variable isn't defined by the derivation writer, it is guessed – a default output name is defined, falling back to other possibilities if the output isn't defined.

$outputDev

is for development-only files. These include C(++) headers, pkg-config, cmake and aclocal files. They go to dev or out by default.

$outputBin

is meant for user-facing binaries, typically residing in bin/. They go to bin or out by default.

$outputLib

is meant for libraries, typically residing in lib/ and libexec/. They go to lib or out by default.

$outputDoc

is for user documentation, typically residing in share/doc/. It goes to doc or out by default.

$outputDevdoc

is for developer documentation. Currently we count gtk-doc in there. It goes to devdoc or is removed (!) by default. This is because e.g. gtk-doc tends to be rather large and completely unused by nixpkgs users.

$outputMan

is for man pages (except for section 3). They go to man or $outputBin by default.

$outputDevman

is for section 3 man pages. They go to devman or $outputMan by default.

$outputInfo

is for info pages. They go to info or $outputBin by default.

4.4.2. Common caveats

  • Some configure scripts don't like some of the parameters passed by default by the framework, e.g. --docdir=/foo/bar. You can disable this by setting setOutputFlags = false;.

  • The outputs of a single derivation can retain references to each other, but note that circular references are not allowed. (And each strongly-connected component would act as a single output anyway.)

  • Most of split packages contain their core functionality in libraries. These libraries tend to refer to various kind of data that typically gets into out, e.g. locale strings, so there is often no advantage in separating the libraries into lib, as keeping them in out is easier.

  • Some packages have hidden assumptions on install paths, which complicates splitting.

5.1. Introduction

"Cross-compilation" means compiling a program on one machine for another type of machine. For example, a typical use of cross compilation is to compile programs for embedded devices. These devices often don't have the computing power and memory to compile their own programs. One might think that cross-compilation is a fairly niche concern, but there are advantages to being rigorous about distinguishing build-time vs run-time environments even when one is developing and deploying on the same machine. Nixpkgs is increasingly adopting this opinion in that packages should be written with cross-compilation in mind, and nixpkgs should evaluate in a similar way (by minimizing cross-compilation-specific special cases) whether or not one is cross-compiling.

This chapter will be organized in three parts. First, it will describe the basics of how to package software in a way that supports cross-compilation. Second, it will describe how to use Nixpkgs when cross-compiling. Third, it will describe the internal infrastructure supporting cross-compilation.

5.2. Packaging in a cross-friendly manner

5.2.1. Platform parameters

The three GNU Autoconf platforms, build, host, and target, are historically the result of much confusion. https://gcc.gnu.org/onlinedocs/gccint/Configure-Terms.html clears this up somewhat but there is more to be said. An important advice to get out the way is, unless you are packaging a compiler or other build tool, just worry about the build and host platforms. Dealing with just two platforms usually better matches people's preconceptions, and in this case is completely correct.

In Nixpkgs, these three platforms are defined as attribute sets under the names buildPlatform, hostPlatform, and targetPlatform. All three are always defined as attributes in the standard environment, and at the top level. That means one can get at them just like a dependency in a function that is imported with callPackage:

{ stdenv, buildPlatform, hostPlatform, fooDep, barDep, .. }: ...buildPlatform...

, or just off stdenv:

{ stdenv, fooDep, barDep, .. }: ...stdenv.buildPlatform...

.

buildPlatform

The "build platform" is the platform on which a package is built. Once someone has a built package, or pre-built binary package, the build platform should not matter and be safe to ignore.

hostPlatform

The "host platform" is the platform on which a package is run. This is the simplest platform to understand, but also the one with the worst name.

targetPlatform

The "target platform" is black sheep. The other two intrinsically apply to all compiled software—or any build process with a notion of "build-time" followed by "run-time". The target platform only applies to programming tools, and even then only is a good for for some of them. Briefly, GCC, Binutils, GHC, and certain other tools are written in such a way such that a single build can only compile code for a single platform. Thus, when building them, one must think ahead about which platforms they wish to use the tool to produce machine code for, and build binaries for each.

There is no fundamental need to think about the target ahead of time like this. LLVM, for example, was designed from the beginning with cross-compilation in mind, and so a normal LLVM binary will support every architecture that LLVM supports. If the tool supports modular or pluggable backends, one might imagine specifying a set of target platforms / backends one wishes to support, rather than a single one.

The biggest reason for mess, if there is one, is that many compilers have the bad habit a build process that builds the compiler and standard library/runtime together. Then the specifying target platform is essential, because it determines the host platform of the standard library/runtime. Nixpkgs tries to avoid this where possible too, but still, because the concept of a target platform is so ingrained now in Autoconf and other tools, it is best to support it as is. Tools like LLVM that don't need up-front target platforms can safely ignore it like normal packages, and it will do no harm.

The exact schema these fields follow is a bit ill-defined due to a long and convoluted evolution, but this is slowly being cleaned up. You can see examples of ones used in practice in lib.systems.examples; note how they are not all very consistent. For now, here are few fields can count on them containing:

system

This is a two-component shorthand for the platform. Examples of this would be "x86_64-darwin" and "i686-linux"; see lib.systems.doubles for more. This format isn't very standard, but has built-in support in Nix, such as the builtins.currentSystem impure string.

config

This is a 3- or 4- component shorthand for the platform. Examples of this would be "x86_64-unknown-linux-gnu" and "aarch64-apple-darwin14". This is a standard format called the "LLVM target triple", as they are pioneered by LLVM and traditionally just used for the targetPlatform. This format is strictly more informative than the "Nix host double", as the previous format could analogously be termed. This needs a better name than config!

parsed

This is a nix representation of a parsed LLVM target triple with white-listed components. This can be specified directly, or actually parsed from the config. [Technically, only one need be specified and the others can be inferred, though the precision of inference may not be very good.] See lib.systems.parse for the exact representation.

libc

This is a string identifying the standard C library used. Valid identifiers include "glibc" for GNU libc, "libSystem" for Darwin's Libsystem, and "uclibc" for µClibc. It should probably be refactored to use the module system, like parse.

is*

These predicates are defined in lib.systems.inspect, and slapped on every platform. They are superior to the ones in stdenv as they force the user to be explicit about which platform they are inspecting. Please use these instead of those.

platform

This is, quite frankly, a dumping ground of ad-hoc settings (it's an attribute set). See lib.systems.platforms for examples—there's hopefully one in there that will work verbatim for each platform that is working. Please help us triage these flags and give them better homes!

5.2.2. Specifying Dependencies

As mentioned in the introduction to this chapter, one can think about a build time vs run time distinction whether cross-compiling or not. In the case of cross-compilation, this corresponds with whether a derivation running on the native or foreign platform is produced. An interesting thing to think about is how this corresponds with the three Autoconf platforms. In the run-time case, the depending and depended-on package simply have matching build, host, and target platforms. But in the build-time case, one can imagine "sliding" the platforms one over. The depended-on package's host and target platforms (respectively) become the depending package's build and host platforms. This is the most important guiding principle behind cross-compilation with Nixpkgs, and will be called the sliding window principle. In this manner, given the 3 platforms for one package, we can determine the three platforms for all its transitive dependencies.

Some examples will probably make this clearer. If a package is being built with a (build, host, target) platform triple of (foo, bar, bar), then its build-time dependencies would have a triple of (foo, foo, bar), and those packages' build-time dependencies would have triple of (foo, foo, foo). In other words, it should take two "rounds" of following build-time dependency edges before one reaches a fixed point where, by the sliding window principle, the platform triple no longer changes. Indeed, this happens with cross compilation, where only rounds of native dependencies starting with the second necessarily coincide with native packages.

Note: The depending package's target platform is unconstrained by the sliding window principle, which makes sense in that one can in principle build cross compilers targeting arbitrary platforms.

How does this work in practice? Nixpkgs is now structured so that build-time dependencies are taken from buildPackages, whereas run-time dependencies are taken from the top level attribute set. For example, buildPackages.gcc should be used at build time, while gcc should be used at run time. Now, for most of Nixpkgs's history, there was no buildPackages, and most packages have not been refactored to use it explicitly. Instead, one can use the four attributes used for specifying dependencies as documented in Section 3.3, “Attributes”. We "splice" together the run-time and build-time package sets with callPackage, and then mkDerivation for each of four attributes pulls the right derivation out. This splicing can be skipped when not cross compiling as the package sets are the same, but is a bit slow for cross compiling. Because of this, a best-of-both-worlds solution is in the works with no splicing or explicit access of buildPackages needed. For now, feel free to use either method.

Note: There is also a "backlink" __targetPackages, yielding a package set whose buildPackages is the current package set. This is a hack, though, to accommodate compilers with lousy build systems. Please do not use this unless you are absolutely sure you are packaging such a compiler and there is no other way.

5.3. Cross-building packages

Note: More information needs to moved from the old wiki, especially https://nixos.org/wiki/CrossCompiling, for this section.

Nixpkgs can be instantiated with localSystem alone, in which case there is no cross compiling and everything is built by and for that system, or also with crossSystem, in which case packages run on the latter, but all building happens on the former. Both parameters take the same schema as the 3 (build, host, and target) platforms defined in the previous section. As mentioned above, lib.systems.examples has some platforms which are used as arguments for these parameters in practice. You can use them programmatically, or on the command line like nix-build <nixpkgs> --arg crossSystem '(import <nixpkgs/lib>).systems.examples.fooBarBaz'.

While one is free to pass both parameters in full, there's a lot of logic to fill in missing fields. As discussed in the previous section, only one of system, config, and parsed is needed to infer the other two. Additionally, libc will be inferred from parse. Finally, localSystem.system is also impurely inferred based on the platform evaluation occurs. This means it is often not necessary to pass localSystem at all, as in the command-line example in the previous paragraph.

Note: Many sources (manual, wiki, etc) probably mention passing system, platform, along with the optional crossSystem to nixpkgs: import <nixpkgs> { system = ..; platform = ..; crossSystem = ..; }. Passing those two instead of localSystem is still supported for compatibility, but is discouraged. Indeed, much of the inference we do for these parameters is motivated by compatibility as much as convenience.

One would think that localSystem and crossSystem overlap horribly with the three *Platforms (buildPlatform, hostPlatform, and targetPlatform; see stage.nix or the manual). Actually, those identifiers are purposefully not used here to draw a subtle but important distinction: While the granularity of having 3 platforms is necessary to properly *build* packages, it is overkill for specifying the user's *intent* when making a build plan or package set. A simple "build vs deploy" dichotomy is adequate: the sliding window principle described in the previous section shows how to interpolate between the these two "end points" to get the 3 platform triple for each bootstrapping stage. That means for any package a given package set, even those not bound on the top level but only reachable via dependencies or buildPackages, the three platforms will be defined as one of localSystem or crossSystem, with the former replacing the latter as one traverses build-time dependencies. A last simple difference then is crossSystem should be null when one doesn't want to cross-compile, while the *Platforms are always non-null. localSystem is always non-null.

5.4. Cross-compilation infrastructure

To be written.

Note: If one explores nixpkgs, they will see derivations with names like gccCross. Such *Cross derivations is a holdover from before we properly distinguished between the host and target platforms —the derivation with "Cross" in the name covered the build = host != target case, while the other covered the host = target, with build platform the same or not based on whether one was using its .nativeDrv or .crossDrv. This ugliness will disappear soon.

Nix comes with certain defaults about what packages can and cannot be installed, based on a package's metadata. By default, Nix will prevent installation if any of the following criteria are true:

  • The package is thought to be broken, and has had its meta.broken set to true.

  • The package's meta.license is set to a license which is considered to be unfree.

  • The package has known security vulnerabilities but has not or can not be updated for some reason, and a list of issues has been entered in to the package's meta.knownVulnerabilities.

Note that all this is checked during evaluation already, and the check includes any package that is evaluated. In particular, all build-time dependencies are checked. nix-env -qa will (attempt to) hide any packages that would be refused.

Each of these criteria can be altered in the nixpkgs configuration.

The nixpkgs configuration for a NixOS system is set in the configuration.nix, as in the following example:

{
  nixpkgs.config = {
    allowUnfree = true;
  };
}

However, this does not allow unfree software for individual users. Their configurations are managed separately.

A user's of nixpkgs configuration is stored in a user-specific configuration file located at ~/.config/nixpkgs/config.nix. For example:

{
  allowUnfree = true;
}

Note that we are not able to test or build unfree software on Hydra due to policy. Most unfree licenses prohibit us from either executing or distributing the software.

6.1. Installing broken packages

There are two ways to try compiling a package which has been marked as broken.

  • For allowing the build of a broken package once, you can use an environment variable for a single invocation of the nix tools:

    $ export NIXPKGS_ALLOW_BROKEN=1

  • For permanently allowing broken packages to be built, you may add allowBroken = true; to your user's configuration file, like this:

    {
      allowBroken = true;
    }
    

6.2. Installing unfree packages

There are several ways to tweak how Nix handles a package which has been marked as unfree.

  • To temporarily allow all unfree packages, you can use an environment variable for a single invocation of the nix tools:

    $ export NIXPKGS_ALLOW_UNFREE=1

  • It is possible to permanently allow individual unfree packages, while still blocking unfree packages by default using the allowUnfreePredicate configuration option in the user configuration file.

    This option is a function which accepts a package as a parameter, and returns a boolean. The following example configuration accepts a package and always returns false:

    {
      allowUnfreePredicate = (pkg: false);
    }
    

    A more useful example, the following configuration allows only allows flash player and visual studio code:

    {
      allowUnfreePredicate = (pkg: elem (builtins.parseDrvName pkg.name).name [ "flashplayer" "vscode" ]);
    }
    

  • It is also possible to whitelist and blacklist licenses that are specifically acceptable or not acceptable, using whitelistedLicenses and blacklistedLicenses, respectively.

    The following example configuration whitelists the licenses amd and wtfpl:

    {
      whitelistedLicenses = with stdenv.lib.licenses; [ amd wtfpl ];
    }
    

    The following example configuration blacklists the gpl3 and agpl3 licenses:

    {
      blacklistedLicenses = with stdenv.lib.licenses; [ agpl3 gpl3 ];
    }
    

A complete list of licenses can be found in the file lib/licenses.nix of the nixpkgs tree.

6.3.  Installing insecure packages

There are several ways to tweak how Nix handles a package which has been marked as insecure.

  • To temporarily allow all insecure packages, you can use an environment variable for a single invocation of the nix tools:

    $ export NIXPKGS_ALLOW_INSECURE=1

  • It is possible to permanently allow individual insecure packages, while still blocking other insecure packages by default using the permittedInsecurePackages configuration option in the user configuration file.

    The following example configuration permits the installation of the hypothetically insecure package hello, version 1.2.3:

    {
      permittedInsecurePackages = [
        "hello-1.2.3"
      ];
    }
    

  • It is also possible to create a custom policy around which insecure packages to allow and deny, by overriding the allowInsecurePredicate configuration option.

    The allowInsecurePredicate option is a function which accepts a package and returns a boolean, much like allowUnfreePredicate.

    The following configuration example only allows insecure packages with very short names:

    {
      allowInsecurePredicate = (pkg: (builtins.stringLength (builtins.parseDrvName pkg.name).name) <= 5);
    }
    

    Note that permittedInsecurePackages is only checked if allowInsecurePredicate is not specified.

6.4. Modify packages via packageOverrides

You can define a function called packageOverrides in your local ~/.config/nixpkgs/config.nix to override nix packages. It must be a function that takes pkgs as an argument and return modified set of packages.

{
  packageOverrides = pkgs: rec {
    foo = pkgs.foo.override { ... };
  };
}

6.5. Declarative Package Management

6.5.1. Build an environment

Using packageOverrides, it is possible to manage packages declaratively. This means that we can list all of our desired packages within a declarative Nix expression. For example, to have aspell, bc, ffmpeg, coreutils, gdb, nixUnstable, emscripten, jq, nox, and silver-searcher, we could use the following in ~/.config/nixpkgs/config.nix:

{
  packageOverrides = pkgs: with pkgs; {
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      paths = [ aspell bc coreutils gdb ffmpeg nixUnstable emscripten jq nox silver-searcher ];
    };
  };
}
    

To install it into our environment, you can just run nix-env -iA nixpkgs.myPackages. If you want to load the packages to be built from a working copy of nixpkgs you just run nix-env -f. -iA myPackages. To explore what's been installed, just look through ~/.nix-profile/. You can see that a lot of stuff has been installed. Some of this stuff is useful some of it isn't. Let's tell Nixpkgs to only link the stuff that we want:

{
  packageOverrides = pkgs: with pkgs; {
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      paths = [ aspell bc coreutils gdb ffmpeg nixUnstable emscripten jq nox silver-searcher ];
      pathsToLink = [ "/share" "/bin" ];
    };
  };
}
    

pathsToLink tells Nixpkgs to only link the paths listed which gets rid of the extra stuff in the profile. /bin and /share are good defaults for a user environment, getting rid of the clutter. If you are running on Nix on MacOS, you may want to add another path as well, /Applications, that makes GUI apps available.

6.5.2. Getting documentation

After building that new environment, look through ~/.nix-profile to make sure everything is there that we wanted. Discerning readers will note that some files are missing. Look inside ~/.nix-profile/share/man/man1/ to verify this. There are no man pages for any of the Nix tools! This is because some packages like Nix have multiple outputs for things like documentation (see section 4). Let's make Nix install those as well.

{
  packageOverrides = pkgs: with pkgs; {
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      paths = [ aspell bc coreutils ffmpeg nixUnstable emscripten jq nox silver-searcher ];
      pathsToLink = [ "/share/man" "/share/doc" /bin" ];
      extraOutputsToInstall = [ "man" "doc" ];
    };
  };
}
    

This provides us with some useful documentation for using our packages. However, if we actually want those manpages to be detected by man, we need to set up our environment. This can also be managed within Nix expressions.

{
  packageOverrides = pkgs: with pkgs; rec {
    myProfile = writeText "my-profile" ''
export PATH=$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/sbin:/bin:/usr/sbin:/usr/bin
export MANPATH=$HOME/.nix-profile/share/man:/nix/var/nix/profiles/default/share/man:/usr/share/man
    '';
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      paths = [
        (runCommand "profile" {} ''
mkdir -p $out/etc/profile.d
cp ${myProfile} $out/etc/profile.d/my-profile.sh
        '')
        aspell
        bc
        coreutils
        ffmpeg
        man
        nixUnstable
        emscripten
        jq
        nox
        silver-searcher
      ];
      pathsToLink = [ "/share/man" "/share/doc" /bin" "/etc" ];
      extraOutputsToInstall = [ "man" "doc" ];
    };
  };
}
    

For this to work fully, you must also have this script sourced when you are logged in. Try adding something like this to your ~/.profile file:

#!/bin/sh
if [ -d $HOME/.nix-profile/etc/profile.d ]; then
  for i in $HOME/.nix-profile/etc/profile.d/*.sh; do
    if [ -r $i ]; then
      . $i
    fi
  done
fi
    

Now just run source $HOME/.profile and you can starting loading man pages from your environent.

6.5.3. GNU info setup

Configuring GNU info is a little bit trickier than man pages. To work correctly, info needs a database to be generated. This can be done with some small modifications to our environment scripts.

{
  packageOverrides = pkgs: with pkgs; rec {
    myProfile = writeText "my-profile" ''
export PATH=$HOME/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/sbin:/bin:/usr/sbin:/usr/bin
export MANPATH=$HOME/.nix-profile/share/man:/nix/var/nix/profiles/default/share/man:/usr/share/man
export INFOPATH=$HOME/.nix-profile/share/info:/nix/var/nix/profiles/default/share/info:/usr/share/info
    '';
    myPackages = pkgs.buildEnv {
      name = "my-packages";
      paths = [
        (runCommand "profile" {} ''
mkdir -p $out/etc/profile.d
cp ${myProfile} $out/etc/profile.d/my-profile.sh
        '')
        aspell
        bc
        coreutils
        ffmpeg
        man
        nixUnstable
        emscripten
        jq
        nox
        silver-searcher
        texinfoInteractive
      ];
      pathsToLink = [ "/share/man" "/share/doc" "/share/info" "/bin" "/etc" ];
      extraOutputsToInstall = [ "man" "doc" "info" ];
      postBuild = ''
          if [ -x $out/bin/install-info -a -w $out/share/info ]; then
            shopt -s nullglob
            for i in $out/share/info/*.info $out/share/info/*.info.gz; do
                $out/bin/install-info $i $out/share/info/dir
            done
          fi
      '';
    };
  };
}
    

postBuild tells Nixpkgs to run a command after building the environment. In this case, install-info adds the installed info pages to dir which is GNU info's default root node. Note that texinfoInteractive is added to the environment to give the install-info command.

The nixpkgs repository has several utility functions to manipulate Nix expressions.

7.1. Overriding

Sometimes one wants to override parts of nixpkgs, e.g. derivation attributes, the results of derivations or even the whole package set.

7.1.1. <pkg>.override

The function override is usually available for all the derivations in the nixpkgs expression (pkgs).

It is used to override the arguments passed to a function.

Example usages:

pkgs.foo.override { arg1 = val1; arg2 = val2; ... }

import pkgs.path { overlays = [ (self: super: {
    foo = super.foo.override { barSupport = true ; };
  })]};

mypkg = pkgs.callPackage ./mypkg.nix {
    mydep = pkgs.mydep.override { ... };
  }

In the first example, pkgs.foo is the result of a function call with some default arguments, usually a derivation. Using pkgs.foo.override will call the same function with the given new arguments.

7.1.2. <pkg>.overrideAttrs

The function overrideAttrs allows overriding the attribute set passed to a stdenv.mkDerivation call, producing a new derivation based on the original one. This function is available on all derivations produced by the stdenv.mkDerivation function, which is most packages in the nixpkgs expression pkgs.

Example usage:

helloWithDebug = pkgs.hello.overrideAttrs (oldAttrs: rec {
    separateDebugInfo = true;
  });

In the above example, the separateDebugInfo attribute is overridden to be true, thus building debug info for helloWithDebug, while all other attributes will be retained from the original hello package.

The argument oldAttrs is conventionally used to refer to the attr set originally passed to stdenv.mkDerivation.

Note: Note that separateDebugInfo is processed only by the stdenv.mkDerivation function, not the generated, raw Nix derivation. Thus, using overrideDerivation will not work in this case, as it overrides only the attributes of the final derivation. It is for this reason that overrideAttrs should be preferred in (almost) all cases to overrideDerivation, i.e. to allow using sdenv.mkDerivation to process input arguments, as well as the fact that it is easier to use (you can use the same attribute names you see in your Nix code, instead of the ones generated (e.g. buildInputs vs nativeBuildInputs, and involves less typing.

7.1.3. <pkg>.overrideDerivation

Warning: You should prefer overrideAttrs in almost all cases, see its documentation for the reasons why. overrideDerivation is not deprecated and will continue to work, but is less nice to use and does not have as many abilities as overrideAttrs.
Warning: Do not use this function in Nixpkgs as it evaluates a Derivation before modifying it, which breaks package abstraction and removes error-checking of function arguments. In addition, this evaluation-per-function application incurs a performance penalty, which can become a problem if many overrides are used. It is only intended for ad-hoc customisation, such as in ~/.config/nixpkgs/config.nix.

The function overrideDerivation creates a new derivation based on an existing one by overriding the original's attributes with the attribute set produced by the specified function. This function is available on all derivations defined using the makeOverridable function. Most standard derivation-producing functions, such as stdenv.mkDerivation, are defined using this function, which means most packages in the nixpkgs expression, pkgs, have this function.

Example usage:

mySed = pkgs.gnused.overrideDerivation (oldAttrs: {
    name = "sed-4.2.2-pre";
    src = fetchurl {
      url = ftp://alpha.gnu.org/gnu/sed/sed-4.2.2-pre.tar.bz2;
      sha256 = "11nq06d131y4wmf3drm0yk502d2xc6n5qy82cg88rb9nqd2lj41k";
    };
    patches = [];
  });

In the above example, the name, src, and patches of the derivation will be overridden, while all other attributes will be retained from the original derivation.

The argument oldAttrs is used to refer to the attribute set of the original derivation.

Note: A package's attributes are evaluated *before* being modified by the overrideDerivation function. For example, the name attribute reference in url = "mirror://gnu/hello/${name}.tar.gz"; is filled-in *before* the overrideDerivation function modifies the attribute set. This means that overriding the name attribute, in this example, *will not* change the value of the url attribute. Instead, we need to override both the name *and* url attributes.

7.1.4. lib.makeOverridable

The function lib.makeOverridable is used to make the result of a function easily customizable. This utility only makes sense for functions that accept an argument set and return an attribute set.

Example usage:

f = { a, b }: { result = a+b; }
  c = lib.makeOverridable f { a = 1; b = 2; }

The variable c is the value of the f function applied with some default arguments. Hence the value of c.result is 3, in this example.

The variable c however also has some additional functions, like c.override which can be used to override the default arguments. In this example the value of (c.override { a = 4; }).result is 6.

7.2. Generators

Generators are functions that create file formats from nix data structures, e. g. for configuration files. There are generators available for: INI, JSON and YAML

All generators follow a similar call interface: generatorName configFunctions data, where configFunctions is a set of user-defined functions that format variable parts of the content. They each have common defaults, so often they do not need to be set manually. An example is mkSectionName ? (name: libStr.escape [ "[" "]" ] name) from the INI generator. It gets the name of a section and returns a sanitized name. The default mkSectionName escapes [ and ] with a backslash.

Note: Nix store paths can be converted to strings by enclosing a derivation attribute like so: "${drv}".

Detailed documentation for each generator can be found in lib/generators.nix.

7.3. buildFHSUserEnv

buildFHSUserEnv provides a way to build and run FHS-compatible lightweight sandboxes. It creates an isolated root with bound /nix/store, so its footprint in terms of disk space needed is quite small. This allows one to run software which is hard or unfeasible to patch for NixOS -- 3rd-party source trees with FHS assumptions, games distributed as tarballs, software with integrity checking and/or external self-updated binaries. It uses Linux namespaces feature to create temporary lightweight environments which are destroyed after all child processes exit, without root user rights requirement. Accepted arguments are:

name

Environment name.

targetPkgs

Packages to be installed for the main host's architecture (i.e. x86_64 on x86_64 installations). Along with libraries binaries are also installed.

multiPkgs

Packages to be installed for all architectures supported by a host (i.e. i686 and x86_64 on x86_64 installations). Only libraries are installed by default.

extraBuildCommands

Additional commands to be executed for finalizing the directory structure.

extraBuildCommandsMulti

Like extraBuildCommands, but executed only on multilib architectures.

extraOutputsToInstall

Additional derivation outputs to be linked for both target and multi-architecture packages.

extraInstallCommands

Additional commands to be executed for finalizing the derivation with runner script.

runScript

A command that would be executed inside the sandbox and passed all the command line arguments. It defaults to bash.

One can create a simple environment using a shell.nix like that:

{ pkgs ? import <nixpkgs> {} }:

(pkgs.buildFHSUserEnv {
  name = "simple-x11-env";
  targetPkgs = pkgs: (with pkgs;
    [ udev
      alsaLib
    ]) ++ (with pkgs.xorg;
    [ libX11
      libXcursor
      libXrandr
    ]);
  multiPkgs = pkgs: (with pkgs;
    [ udev
      alsaLib
    ]);
  runScript = "bash";
}).env

Running nix-shell would then drop you into a shell with these libraries and binaries available. You can use this to run closed-source applications which expect FHS structure without hassles: simply change runScript to the application path, e.g. ./bin/start.sh -- relative paths are supported.

7.4. pkgs.dockerTools

pkgs.dockerTools is a set of functions for creating and manipulating Docker images according to the Docker Image Specification v1.2.0 . Docker itself is not used to perform any of the operations done by these functions.

Warning: The dockerTools API is unstable and may be subject to backwards-incompatible changes in the future.

7.4.1. buildImage

This function is analogous to the docker build command, in that can used to build a Docker-compatible repository tarball containing a single image with one or multiple layers. As such, the result is suitable for being loaded in Docker with docker load.

The parameters of buildImage with relative example values are described below:

Example 7.1. Docker build

  buildImage {
    name = "redis"; 1
    tag = "latest"; 2

    fromImage = someBaseImage; 3
    fromImageName = null; 4
    fromImageTag = "latest"; 5

    contents = pkgs.redis; 6
    runAsRoot = '' 7
      #!${stdenv.shell}
      mkdir -p /data
    '';

    config = { 8
      Cmd = [ "/bin/redis-server" ];
      WorkingDir = "/data";
      Volumes = {
        "/data" = {};
      };
    };
  }
  


The above example will build a Docker image redis/latest from the given base image. Loading and running this image in Docker results in redis-server being started automatically.

1

name specifies the name of the resulting image. This is the only required argument for buildImage.

2

tag specifies the tag of the resulting image. By default it's latest.

3

fromImage is the repository tarball containing the base image. It must be a valid Docker image, such as exported by docker save. By default it's null, which can be seen as equivalent to FROM scratch of a Dockerfile.

4

fromImageName can be used to further specify the base image within the repository, in case it contains multiple images. By default it's null, in which case buildImage will peek the first image available in the repository.

5

fromImageTag can be used to further specify the tag of the base image within the repository, in case an image contains multiple tags. By default it's null, in which case buildImage will peek the first tag available for the base image.

6

contents is a derivation that will be copied in the new layer of the resulting image. This can be similarly seen as ADD contents/ / in a Dockerfile. By default it's null.

7

runAsRoot is a bash script that will run as root in an environment that overlays the existing layers of the base image with the new resulting layer, including the previously copied contents derivation. This can be similarly seen as RUN ... in a Dockerfile.

Note: Using this parameter requires the kvm device to be available.

8

config is used to specify the configuration of the containers that will be started off the built image in Docker. The available options are listed in the Docker Image Specification v1.2.0 .

After the new layer has been created, its closure (to which contents, config and runAsRoot contribute) will be copied in the layer itself. Only new dependencies that are not already in the existing layers will be copied.

At the end of the process, only one new single layer will be produced and added to the resulting image.

The resulting repository will only list the single image image/tag. In the case of Example 7.1, “Docker build” it would be redis/latest.

It is possible to inspect the arguments with which an image was built using its buildArgs attribute.

Note: If you see errors similar to getProtocolByName: does not exist (no such protocol name: tcp) you may need to add pkgs.iana-etc to contents.
Note: If you see errors similar to Error_Protocol ("certificate has unknown CA",True,UnknownCa) you may need to add pkgs.cacert to contents.

7.4.2. pullImage

This function is analogous to the docker pull command, in that can be used to fetch a Docker image from a Docker registry. Currently only registry v1 is supported. By default Docker Hub is used to pull images.

Its parameters are described in the example below:

Example 7.2. Docker pull

  pullImage {
    imageName = "debian"; 1
    imageTag = "jessie"; 2
    imageId = null; 3
    sha256 = "1bhw5hkz6chrnrih0ymjbmn69hyfriza2lr550xyvpdrnbzr4gk2"; 4

    indexUrl = "https://index.docker.io"; 5
    registryVersion = "v1";
  }
  


1

imageName specifies the name of the image to be downloaded, which can also include the registry namespace (e.g. library/debian). This argument is required.

2

imageTag specifies the tag of the image to be downloaded. By default it's latest.

3

imageId, if specified this exact image will be fetched, instead of imageName/imageTag. However, the resulting repository will still be named imageName/imageTag. By default it's null.

4

sha256 is the checksum of the whole fetched image. This argument is required.

Note: The checksum is computed on the unpacked directory, not on the final tarball.

5

In the above example the default values are shown for the variables indexUrl and registryVersion. Hence by default the Docker.io registry is used to pull the images.

7.4.3. exportImage

This function is analogous to the docker export command, in that can used to flatten a Docker image that contains multiple layers. It is in fact the result of the merge of all the layers of the image. As such, the result is suitable for being imported in Docker with docker import.

Note: Using this function requires the kvm device to be available.

The parameters of exportImage are the following:

Example 7.3. Docker export

  exportImage {
    fromImage = someLayeredImage;
    fromImageName = null;
    fromImageTag = null;

    name = someLayeredImage.name;
  }
  


The parameters relative to the base image have the same synopsis as described in Section 7.4.1, “buildImage”, except that fromImage is the only required argument in this case.

The name argument is the name of the derivation output, which defaults to fromImage.name.

7.4.4. shadowSetup

This constant string is a helper for setting up the base files for managing users and groups, only if such files don't exist already. It is suitable for being used in a runAsRoot 7 script for cases like in the example below:

Example 7.4. Shadow base files

  buildImage {
    name = "shadow-basic";

    runAsRoot = ''
      #!${stdenv.shell}
      ${shadowSetup}
      groupadd -r redis
      useradd -r -g redis redis
      mkdir /data
      chown redis:redis /data
    '';
  }
  


Creating base files like /etc/passwd or /etc/login.defs are necessary for shadow-utils to manipulate users and groups.

Nix packages can declare meta-attributes that contain information about a package such as a description, its homepage, its license, and so on. For instance, the GNU Hello package has a meta declaration like this:

meta = {
  description = "A program that produces a familiar, friendly greeting";
  longDescription = ''
    GNU Hello is a program that prints "Hello, world!" when you run it.
    It is fully customizable.
  '';
  homepage = http://www.gnu.org/software/hello/manual/;
  license = stdenv.lib.licenses.gpl3Plus;
  maintainers = [ stdenv.lib.maintainers.eelco ];
  platforms = stdenv.lib.platforms.all;
};

Meta-attributes are not passed to the builder of the package. Thus, a change to a meta-attribute doesn’t trigger a recompilation of the package. The value of a meta-attribute must be a string.

The meta-attributes of a package can be queried from the command-line using nix-env:

$ nix-env -qa hello --json
{
    "hello": {
        "meta": {
            "description": "A program that produces a familiar, friendly greeting",
            "homepage": "http://www.gnu.org/software/hello/manual/",
            "license": {
                "fullName": "GNU General Public License version 3 or later",
                "shortName": "GPLv3+",
                "url": "http://www.fsf.org/licensing/licenses/gpl.html"
            },
            "longDescription": "GNU Hello is a program that prints \"Hello, world!\" when you run it.\nIt is fully customizable.\n",
            "maintainers": [
                "Ludovic Court\u00e8s <ludo@gnu.org>"
            ],
            "platforms": [
                "i686-linux",
                "x86_64-linux",
                "armv5tel-linux",
                "armv7l-linux",
                "mips64el-linux",
                "x86_64-darwin",
                "i686-cygwin",
                "i686-freebsd",
                "x86_64-freebsd",
                "i686-openbsd",
                "x86_64-openbsd"
            ],
            "position": "/home/user/dev/nixpkgs/pkgs/applications/misc/hello/default.nix:14"
        },
        "name": "hello-2.9",
        "system": "x86_64-linux"
    }
}


nix-env knows about the description field specifically:

$ nix-env -qa hello --description
hello-2.3  A program that produces a familiar, friendly greeting

8.1. Standard meta-attributes

It is expected that each meta-attribute is one of the following:

description

A short (one-line) description of the package. This is shown by nix-env -q --description and also on the Nixpkgs release pages.

Don’t include a period at the end. Don’t include newline characters. Capitalise the first character. For brevity, don’t repeat the name of package — just describe what it does.

Wrong: "libpng is a library that allows you to decode PNG images."

Right: "A library for decoding PNG images"

longDescription

An arbitrarily long description of the package.

branch

Release branch. Used to specify that a package is not going to receive updates that are not in this branch; for example, Linux kernel 3.0 is supposed to be updated to 3.0.X, not 3.1.

homepage

The package’s homepage. Example: http://www.gnu.org/software/hello/manual/

downloadPage

The page where a link to the current version can be found. Example: http://ftp.gnu.org/gnu/hello/

license

The license, or licenses, for the package. One from the attribute set defined in nixpkgs/lib/licenses.nix. At this moment using both a list of licenses and a single license is valid. If the license field is in the form of a list representation, then it means that parts of the package are licensed differently. Each license should preferably be referenced by their attribute. The non-list attribute value can also be a space delimited string representation of the contained attribute shortNames or spdxIds. The following are all valid examples:

  • Single license referenced by attribute (preferred) stdenv.lib.licenses.gpl3.

  • Single license referenced by its attribute shortName (frowned upon) "gpl3".

  • Single license referenced by its attribute spdxId (frowned upon) "GPL-3.0".

  • Multiple licenses referenced by attribute (preferred) with stdenv.lib.licenses; [ asl20 free ofl ].

  • Multiple licenses referenced as a space delimited string of attribute shortNames (frowned upon) "asl20 free ofl".

For details, see Section 8.2, “Licenses”.

maintainers

A list of names and e-mail addresses of the maintainers of this Nix expression. If you would like to be a maintainer of a package, you may want to add yourself to nixpkgs/lib/maintainers.nix and write something like [ stdenv.lib.maintainers.alice stdenv.lib.maintainers.bob ].

priority

The priority of the package, used by nix-env to resolve file name conflicts between packages. See the Nix manual page for nix-env for details. Example: "10" (a low-priority package).

platforms

The list of Nix platform types on which the package is supported. Hydra builds packages according to the platform specified. If no platform is specified, the package does not have prebuilt binaries. An example is:

meta.platforms = stdenv.lib.platforms.linux;

Attribute Set stdenv.lib.platforms in nixpkgs/lib/systems/platforms.nix defines various common lists of platforms types.

hydraPlatforms

The list of Nix platform types for which the Hydra instance at hydra.nixos.org will build the package. (Hydra is the Nix-based continuous build system.) It defaults to the value of meta.platforms. Thus, the only reason to set meta.hydraPlatforms is if you want hydra.nixos.org to build the package on a subset of meta.platforms, or not at all, e.g.

meta.platforms = stdenv.lib.platforms.linux;
meta.hydraPlatforms = [];

broken

If set to true, the package is marked as “broken”, meaning that it won’t show up in nix-env -qa, and cannot be built or installed. Such packages should be removed from Nixpkgs eventually unless they are fixed.

updateWalker

If set to true, the package is tested to be updated correctly by the update-walker.sh script without additional settings. Such packages have meta.version set and their homepage (or the page specified by meta.downloadPage) contains a direct link to the package tarball.

8.2. Licenses

The meta.license attribute should preferrably contain a value from stdenv.lib.licenses defined in nixpkgs/lib/licenses.nix, or in-place license description of the same format if the license is unlikely to be useful in another expression.

Although it's typically better to indicate the specific license, a few generic options are available:

stdenv.lib.licenses.free, "free"

Catch-all for free software licenses not listed above.

stdenv.lib.licenses.unfreeRedistributable, "unfree-redistributable"

Unfree package that can be redistributed in binary form. That is, it’s legal to redistribute the output of the derivation. This means that the package can be included in the Nixpkgs channel.

Sometimes proprietary software can only be redistributed unmodified. Make sure the builder doesn’t actually modify the original binaries; otherwise we’re breaking the license. For instance, the NVIDIA X11 drivers can be redistributed unmodified, but our builder applies patchelf to make them work. Thus, its license is "unfree" and it cannot be included in the Nixpkgs channel.

stdenv.lib.licenses.unfree, "unfree"

Unfree package that cannot be redistributed. You can build it yourself, but you cannot redistribute the output of the derivation. Thus it cannot be included in the Nixpkgs channel.

stdenv.lib.licenses.unfreeRedistributableFirmware, "unfree-redistributable-firmware"

This package supplies unfree, redistributable firmware. This is a separate value from unfree-redistributable because not everybody cares whether firmware is free.

Table of Contents

9.1. BEAM Languages (Erlang, Elixir & LFE)
9.1.1. Introduction
9.1.2. Structure
9.1.3. Build Tools
9.1.3.1. Rebar3
9.1.3.2. Mix & Erlang.mk
9.1.4. How to Install BEAM Packages
9.1.5. Packaging BEAM Applications
9.1.5.1. Erlang Applications
9.1.6. How to Develop
9.1.6.1. Accessing an Environment
9.1.6.2. Creating a Shell
9.1.7. Generating Packages from Hex with hex2nix
9.2. Bower
9.2.1. bower2nix usage
9.2.2. buildBowerComponents function
9.2.3. Troubleshooting
9.3. Coq
9.4. Go
9.5. User’s Guide to the Haskell Infrastructure
9.5.1. How to install Haskell packages
9.5.2. How to create a development environment
9.5.2.1. How to install a compiler
9.5.2.2. How to install a compiler with libraries
9.5.2.3. How to install a compiler with libraries, hoogle and documentation indexes
9.5.2.4. How to build a Haskell project using Stack
9.5.2.5. How to create ad hoc environments for nix-shell
9.5.3. How to create Nix builds for your own private Haskell packages
9.5.3.1. How to build a stand-alone project
9.5.3.2. How to build projects that depend on each other
9.5.4. Miscellaneous Topics
9.5.4.1. How to build with profiling enabled
9.5.4.2. How to override package versions in a compiler-specific package set
9.5.4.3. How to recover from GHC’s infamous non-deterministic library ID bug
9.5.4.4. Builds on Darwin fail with math.h not found
9.5.4.5. Builds using Stack complain about missing system libraries
9.5.4.6. Creating statically linked binaries
9.5.4.7. Building GHC with integer-simple
9.5.4.8. Quality assurance
9.5.5. Other resources
9.6. Idris packages
9.6.1. callPackage
9.6.2. builtins
9.6.3. build-idris-package
9.6.4. build-builtin-package
9.6.5. with-packages
9.7. Java
9.8. Lua
9.9. Node.js packages
9.10. Perl
9.10.1. Generation from CPAN
9.11. Python
9.11.1. User Guide
9.11.1.1. Using Python
9.11.1.2. Developing with Python
9.11.1.3. Organising your packages
9.11.1.4. Including a derivation using callPackage
9.11.2. Reference
9.11.2.1. Interpreters
9.11.2.2. Building packages and applications
9.11.2.3. Development mode
9.11.2.4. Tools
9.11.2.5. Deterministic builds
9.11.2.6. Automatic tests
9.11.3. FAQ
9.11.3.1. How to solve circular dependencies?
9.11.3.2. How to override a Python package?
9.11.3.3. python setup.py bdist_wheel cannot create .whl
9.11.3.4. install_data / data_files problems
9.11.3.5. Rationale of non-existent global site-packages
9.11.3.6. How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?
9.11.3.7. How to override a Python package from configuration.nix?
9.11.3.8. How to override a Python package using overlays?
9.11.4. Contributing
9.11.4.1. Contributing guidelines
9.12. Qt
9.12.1. Packaging Libraries for Nixpkgs
9.12.2. Packaging Applications for Nixpkgs
9.13. R packages
9.13.1. Installation
9.13.2. RStudio
9.13.3. Updating the package set
9.13.4. Testing if the Nix-expression could be evaluated
9.14. Ruby
9.15. User’s Guide to the Rust Infrastructure
9.15.1. Packaging Rust applications
9.15.2. Using the Rust nightlies overlay
9.16. TeX Live
9.16.1. User's guide
9.16.2. Known problems
9.17. User’s Guide to Vim Plugins/Addons/Bundles/Scripts in Nixpkgs
9.17.1. Custom configuration
9.17.2. Vim packages
9.17.3. VAM
9.17.3.1. dependencies by Vim plugins
9.17.3.2. Example
9.17.4. Important repositories

The standard build environment makes it easy to build typical Autotools-based packages with very little code. Any other kind of package can be accomodated by overriding the appropriate phases of stdenv. However, there are specialised functions in Nixpkgs to easily build packages for other programming languages, such as Perl or Haskell. These are described in this chapter.

9.1. BEAM Languages (Erlang, Elixir & LFE)

9.1.1. Introduction

In this document and related Nix expressions, we use the term, BEAM, to describe the environment. BEAM is the name of the Erlang Virtual Machine and, as far as we're concerned, from a packaging perspective, all languages that run on the BEAM are interchangeable. That which varies, like the build system, is transparent to users of any given BEAM package, so we make no distinction.

9.1.2. Structure

All BEAM-related expressions are available via the top-level beam attribute, which includes:

  • interpreters: a set of compilers running on the BEAM, including multiple Erlang/OTP versions (beam.interpreters.erlangR19, etc), Elixir (beam.interpreters.elixir) and LFE (beam.interpreters.lfe).

  • packages: a set of package sets, each compiled with a specific Erlang/OTP version, e.g. beam.packages.erlangR19.

The default Erlang compiler, defined by beam.interpreters.erlang, is aliased as erlang. The default BEAM package set is defined by beam.packages.erlang and aliased at the top level as beamPackages.

To create a package set built with a custom Erlang version, use the lambda, beam.packagesWith, which accepts an Erlang/OTP derivation and produces a package set similar to beam.packages.erlang.

Many Erlang/OTP distributions available in beam.interpreters have versions with ODBC and/or Java enabled. For example, there's beam.interpreters.erlangR19_odbc_javac, which corresponds to beam.interpreters.erlangR19.

We also provide the lambda, beam.packages.erlang.callPackage, which simplifies writing BEAM package definitions by injecting all packages from beam.packages.erlang into the top-level context.

9.1.3. Build Tools

9.1.3.1. Rebar3

By default, Rebar3 wants to manage its own dependencies. This is perfectly acceptable in the normal, non-Nix setup, but in the Nix world, it is not. To rectify this, we provide two versions of Rebar3:

  • rebar3: patched to remove the ability to download anything. When not running it via nix-shell or nix-build, it's probably not going to work as desired.

  • rebar3-open: the normal, unmodified Rebar3. It should work exactly as would any other version of Rebar3. Any Erlang package should rely on rebar3 instead. See Section 9.1.5.1.1, “Rebar3 Packages”.

9.1.3.2. Mix & Erlang.mk

Both Mix and Erlang.mk work exactly as expected. There is a bootstrap process that needs to be run for both, however, which is supported by the buildMix and buildErlangMk derivations, respectively.

9.1.4. How to Install BEAM Packages

BEAM packages are not registered at the top level, simply because they are not relevant to the vast majority of Nix users. They are installable using the beam.packages.erlang attribute set (aliased as beamPackages), which points to packages built by the default Erlang/OTP version in Nixpkgs, as defined by beam.interpreters.erlang. To list the available packages in beamPackages, use the following command:

$ nix-env -f "<nixpkgs>" -qaP -A beamPackages
beamPackages.esqlite    esqlite-0.2.1
beamPackages.goldrush   goldrush-0.1.7
beamPackages.ibrowse    ibrowse-4.2.2
beamPackages.jiffy      jiffy-0.14.5
beamPackages.lager      lager-3.0.2
beamPackages.meck       meck-0.8.3
beamPackages.rebar3-pc  pc-1.1.0
  

To install any of those packages into your profile, refer to them by their attribute path (first column):

$ nix-env -f "<nixpkgs>" -iA beamPackages.ibrowse
  

The attribute path of any BEAM package corresponds to the name of that particular package in Hex or its OTP Application/Release name.

9.1.5. Packaging BEAM Applications

9.1.5.1. Erlang Applications

9.1.5.1.1. Rebar3 Packages

The Nix function, buildRebar3, defined in beam.packages.erlang.buildRebar3 and aliased at the top level, can be used to build a derivation that understands how to build a Rebar3 project. For example, we can build hex2nix as follows:

        { stdenv, fetchFromGitHub, buildRebar3, ibrowse, jsx, erlware_commons }:

        buildRebar3 rec {
          name = "hex2nix";
          version = "0.0.1";

          src = fetchFromGitHub {
            owner = "ericbmerritt";
            repo = "hex2nix";
            rev = "${version}";
            sha256 = "1w7xjidz1l5yjmhlplfx7kphmnpvqm67w99hd2m7kdixwdxq0zqg";
          };

          beamDeps = [ ibrowse jsx erlware_commons ];
        }
      

Such derivations are callable with beam.packages.erlang.callPackage (see Section 9.1.2, “Structure”). To call this package using the normal callPackage, refer to dependency packages via beamPackages, e.g. beamPackages.ibrowse.

Notably, buildRebar3 includes beamDeps, while stdenv.mkDerivation does not. BEAM dependencies added there will be correctly handled by the system.

If a package needs to compile native code via Rebar3's port compilation mechanism, add compilePort = true; to the derivation.

9.1.5.1.2. Erlang.mk Packages

Erlang.mk functions similarly to Rebar3, except we use buildErlangMk instead of buildRebar3.

        { buildErlangMk, fetchHex, cowlib, ranch }:

        buildErlangMk {
          name = "cowboy";
          version = "1.0.4";

          src = fetchHex {
            pkg = "cowboy";
            version = "1.0.4";
            sha256 = "6a0edee96885fae3a8dd0ac1f333538a42e807db638a9453064ccfdaa6b9fdac";
          };

          beamDeps = [ cowlib ranch ];

          meta = {
            description = ''
              Small, fast, modular HTTP server written in Erlang
            '';
            license = stdenv.lib.licenses.isc;
            homepage = https://github.com/ninenines/cowboy;
          };
        }
      
9.1.5.1.3. Mix Packages

Mix functions similarly to Rebar3, except we use buildMix instead of buildRebar3.

        { buildMix, fetchHex, plug, absinthe }:

        buildMix {
          name = "absinthe_plug";
          version = "1.0.0";

          src = fetchHex {
            pkg = "absinthe_plug";
            version = "1.0.0";
            sha256 = "08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";
          };

          beamDeps = [ plug absinthe ];

          meta = {
            description = ''
              A plug for Absinthe, an experimental GraphQL toolkit
            '';
            license = stdenv.lib.licenses.bsd3;
            homepage = https://github.com/CargoSense/absinthe_plug;
          };
        }
      

Alternatively, we can use buildHex as a shortcut:

        { buildHex, buildMix, plug, absinthe }:

        buildHex {
          name = "absinthe_plug";
          version = "1.0.0";

          sha256 = "08459823fe1fd4f0325a8bf0c937a4520583a5a26d73b193040ab30a1dfc0b33";

          builder = buildMix;

          beamDeps = [ plug absinthe ];

          meta = {
            description = ''
              A plug for Absinthe, an experimental GraphQL toolkit
            '';
            license = stdenv.lib.licenses.bsd3;
            homepage = https://github.com/CargoSense/absinthe_plug;
         };
       }
      

9.1.6. How to Develop

9.1.6.1. Accessing an Environment

Often, we simply want to access a valid environment that contains a specific package and its dependencies. We can accomplish that with the env attribute of a derivation. For example, let's say we want to access an Erlang REPL with ibrowse loaded up. We could do the following:

      $ nix-shell -A beamPackages.ibrowse.env --run "erl"
      Erlang/OTP 18 [erts-7.0] [source] [64-bit] [smp:4:4] [async-threads:10] [hipe] [kernel-poll:false]

      Eshell V7.0  (abort with ^G)
      1> m(ibrowse).
      Module: ibrowse
      MD5: 3b3e0137d0cbb28070146978a3392945
      Compiled: January 10 2016, 23:34
      Object file: /nix/store/g1rlf65rdgjs4abbyj4grp37ry7ywivj-ibrowse-4.2.2/lib/erlang/lib/ibrowse-4.2.2/ebin/ibrowse.beam
      Compiler options:  [{outdir,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/ebin"},
      debug_info,debug_info,nowarn_shadow_vars,
      warn_unused_import,warn_unused_vars,warnings_as_errors,
      {i,"/tmp/nix-build-ibrowse-4.2.2.drv-0/hex-source-ibrowse-4.2.2/_build/default/lib/ibrowse/include"}]
      Exports:
      add_config/1                  send_req_direct/7
      all_trace_off/0               set_dest/3
      code_change/3                 set_max_attempts/3
      get_config_value/1            set_max_pipeline_size/3
      get_config_value/2            set_max_sessions/3
      get_metrics/0                 show_dest_status/0
      get_metrics/2                 show_dest_status/1
      handle_call/3                 show_dest_status/2
      handle_cast/2                 spawn_link_worker_process/1
      handle_info/2                 spawn_link_worker_process/2
      init/1                        spawn_worker_process/1
      module_info/0                 spawn_worker_process/2
      module_info/1                 start/0
      rescan_config/0               start_link/0
      rescan_config/1               stop/0
      send_req/3                    stop_worker_process/1
      send_req/4                    stream_close/1
      send_req/5                    stream_next/1
      send_req/6                    terminate/2
      send_req_direct/4             trace_off/0
      send_req_direct/5             trace_off/2
      send_req_direct/6             trace_on/0
      trace_on/2
      ok
      2>
    

Notice the -A beamPackages.ibrowse.env. That is the key to this functionality.

9.1.6.2. Creating a Shell

Getting access to an environment often isn't enough to do real development. Usually, we need to create a shell.nix file and do our development inside of the environment specified therein. This file looks a lot like the packaging described above, except that src points to the project root and we call the package directly.

{ pkgs ? import "<nixpkgs"> {} }:

with pkgs;

let

  f = { buildRebar3, ibrowse, jsx, erlware_commons }:
      buildRebar3 {
        name = "hex2nix";
        version = "0.1.0";
        src = ./.;
        beamDeps = [ ibrowse jsx erlware_commons ];
      };
  drv = beamPackages.callPackage f {};

in

  drv
    
9.1.6.2.1. Building in a Shell (for Mix Projects)

We can leverage the support of the derivation, irrespective of the build derivation, by calling the commands themselves.

# =============================================================================
# Variables
# =============================================================================

NIX_TEMPLATES := "$(CURDIR)/nix-templates"

TARGET := "$(PREFIX)"

PROJECT_NAME := thorndyke

NIXPKGS=../nixpkgs
NIX_PATH=nixpkgs=$(NIXPKGS)
NIX_SHELL=nix-shell -I "$(NIX_PATH)" --pure
# =============================================================================
# Rules
# =============================================================================
.PHONY= all test clean repl shell build test analyze configure install \
        test-nix-install publish plt analyze

all: build

guard-%:
        @ if [ "${${*}}" == "" ]; then \
                echo "Environment variable $* not set"; \
                exit 1; \
        fi

clean:
        rm -rf _build
        rm -rf .cache

repl:
        $(NIX_SHELL) --run "iex -pa './_build/prod/lib/*/ebin'"

shell:
        $(NIX_SHELL)

configure:
        $(NIX_SHELL) --command 'eval "$$configurePhase"'

build: configure
        $(NIX_SHELL) --command 'eval "$$buildPhase"'

install:
        $(NIX_SHELL) --command 'eval "$$installPhase"'

test:
        $(NIX_SHELL) --command 'mix test --no-start --no-deps-check'

plt:
        $(NIX_SHELL) --run "mix dialyzer.plt --no-deps-check"

analyze: build plt
        $(NIX_SHELL) --run "mix dialyzer --no-compile"

    

Using a shell.nix as described (see Section 9.1.6.2, “Creating a Shell”) should just work. Aside from test, plt, and analyze, the Make targets work just fine for all of the build derivations.

9.1.7. Generating Packages from Hex with hex2nix

Updating the Hex package set requires hex2nix. Given the path to the Erlang modules (usually pkgs/development/erlang-modules), it will dump a file called hex-packages.nix, containing all the packages that use a recognized build system in Hex. It can't be determined, however, whether every package is buildable.

To make life easier for our users, try to build every Hex package and remove those that fail. To do that, simply run the following command in the root of your nixpkgs repository:

$ nix-build -A beamPackages
    

That will attempt to build every package in beamPackages. Then manually remove those that fail. Hopefully, someone will improve hex2nix in the future to automate the process.

9.2. Bower

Bower is a package manager for web site front-end components. Bower packages (comprising of build artefacts and sometimes sources) are stored in git repositories, typically on Github. The package registry is run by the Bower team with package metadata coming from the bower.json file within each package.

The end result of running Bower is a bower_components directory which can be included in the web app's build process.

Bower can be run interactively, by installing nodePackages.bower. More interestingly, the Bower components can be declared in a Nix derivation, with the help of nodePackages.bower2nix.

9.2.1. bower2nix usage

Suppose you have a bower.json with the following contents:

Example 9.1. bower.json

{
  "name": "my-web-app",
  "dependencies": {
    "angular": "~1.5.0",
    "bootstrap": "~3.3.6"
  }
}



Running bower2nix will produce something like the following output:

{ fetchbower, buildEnv }:
buildEnv { name = "bower-env"; ignoreCollisions = true; paths = [
  (fetchbower "angular" "1.5.3" "~1.5.0" "1749xb0firxdra4rzadm4q9x90v6pzkbd7xmcyjk6qfza09ykk9y")
  (fetchbower "bootstrap" "3.3.6" "~3.3.6" "1vvqlpbfcy0k5pncfjaiskj3y6scwifxygfqnw393sjfxiviwmbv")
  (fetchbower "jquery" "2.2.2" "1.9.1 - 2" "10sp5h98sqwk90y4k6hbdviwqzvzwqf47r3r51pakch5ii2y7js1")
]; }

Using the bower2nix command line arguments, the output can be redirected to a file. A name like bower-packages.nix would be fine.

The resulting derivation is a union of all the downloaded Bower packages (and their dependencies). To use it, they still need to be linked together by Bower, which is where buildBowerComponents is useful.

9.2.2. buildBowerComponents function

The function is implemented in pkgs/development/bower-modules/generic/default.nix. Example usage:

Example 9.2. buildBowerComponents

bowerComponents = buildBowerComponents {
  name = "my-web-app";
  generated = ./bower-packages.nix; 1
  src = myWebApp; 2
};



In Example 9.2, “buildBowerComponents”, the following arguments are of special significance to the function:

1

generated specifies the file which was created by bower2nix.

2

src is your project's sources. It needs to contain a bower.json file.

buildBowerComponents will run Bower to link together the output of bower2nix, resulting in a bower_components directory which can be used.

Here is an example of a web frontend build process using gulp. You might use grunt, or anything else.

Example 9.3. Example build script (gulpfile.js)

var gulp = require('gulp');

gulp.task('default', [], function () {
  gulp.start('build');
});

gulp.task('build', [], function () {
  console.log("Just a dummy gulp build");
  gulp
    .src(["./bower_components/**/*"])
    .pipe(gulp.dest("./gulpdist/"));
});


Example 9.4. Full example — default.nix

{ myWebApp ? { outPath = ./.; name = "myWebApp"; }
, pkgs ? import <nixpkgs> {}
}:

pkgs.stdenv.mkDerivation {
  name = "my-web-app-frontend";
  src = myWebApp;

  buildInputs = [ pkgs.nodePackages.gulp ];

  bowerComponents = pkgs.buildBowerComponents { 1
    name = "my-web-app";
    generated = ./bower-packages.nix;
    src = myWebApp;
  };

  buildPhase = ''
    cp --reflink=auto --no-preserve=mode -R $bowerComponents/bower_components . 2
    export HOME=$PWD 3
    ${pkgs.nodePackages.gulp}/bin/gulp build 4
  '';

  installPhase = "mv gulpdist $out";
}


A few notes about Example 9.4, “Full example — default.nix:

1

The result of buildBowerComponents is an input to the frontend build.

2

Whether to symlink or copy the bower_components directory depends on the build tool in use. In this case a copy is used to avoid gulp silliness with permissions.

3

gulp requires HOME to refer to a writeable directory.

4

The actual build command. Other tools could be used.

9.2.3. Troubleshooting

ENOCACHE errors from buildBowerComponents

This means that Bower was looking for a package version which doesn't exist in the generated bower-packages.nix.

If bower.json has been updated, then run bower2nix again.

It could also be a bug in bower2nix or fetchbower. If possible, try reformulating the version specification in bower.json.

9.3. Coq

Coq libraries should be installed in $(out)/lib/coq/${coq.coq-version}/user-contrib/. Such directories are automatically added to the $COQPATH environment variable by the hook defined in the Coq derivation.

Some libraries require OCaml and sometimes also Camlp5. The exact versions that were used to build Coq are saved in the coq.ocaml and coq.camlp5 attributes.

Here is a simple package example. It is a pure Coq library, thus it only depends on Coq. Its makefile has been generated using coq_makefile so we only have to set the $COQLIB variable at install time.

{stdenv, fetchurl, coq}:
stdenv.mkDerivation {
  src = fetchurl {
    url = http://coq.inria.fr/pylons/contribs/files/Karatsuba/v8.4/Karatsuba.tar.gz;
    sha256 = "0ymfpv4v49k4fm63nq6gcl1hbnnxrvjjp7yzc4973n49b853c5b1";
  };

  name = "coq-karatsuba";

  buildInputs = [ coq ];

  installFlags = "COQLIB=$(out)/lib/coq/${coq.coq-version}/";
}

9.4. Go

The function buildGoPackage builds standard Go programs.

Example 9.5. buildGoPackage

deis = buildGoPackage rec {
  name = "deis-${version}";
  version = "1.13.0";

  goPackagePath = "github.com/deis/deis"; 1
  subPackages = [ "client" ]; 2

  src = fetchFromGitHub {
    owner = "deis";
    repo = "deis";
    rev = "v${version}";
    sha256 = "1qv9lxqx7m18029lj8cw3k7jngvxs4iciwrypdy0gd2nnghc68sw";
  };

  goDeps = ./deps.nix; 3

  buildFlags = "--tags release"; 4
}


Example 9.5, “buildGoPackage” is an example expression using buildGoPackage, the following arguments are of special significance to the function:

1

goPackagePath specifies the package's canonical Go import path.

2

subPackages limits the builder from building child packages that have not been listed. If subPackages is not specified, all child packages will be built.

In this example only github.com/deis/deis/client will be built.

3

goDeps is where the Go dependencies of a Go program are listed as a list of package source identified by Go import path. It could be imported as a separate deps.nix file for readability. The dependency data structure is described below.

4

buildFlags is a list of flags passed to the go build command.

The goDeps attribute can be imported from a separate nix file that defines which Go libraries are needed and should be included in GOPATH for buildPhase.

Example 9.6. deps.nix

[ 1
  {
    goPackagePath = "gopkg.in/yaml.v2"; 2
    fetch = {
      type = "git"; 3
      url = "https://gopkg.in/yaml.v2";
      rev = "a83829b6f1293c91addabc89d0571c246397bbf4";
      sha256 = "1m4dsmk90sbi17571h6pld44zxz7jc4lrnl4f27dpd1l8g5xvjhh";
    };
  }
  {
    goPackagePath = "github.com/docopt/docopt-go";
    fetch = {
      type = "git";
      url = "https://github.com/docopt/docopt-go";
      rev = "784ddc588536785e7299f7272f39101f7faccc3f";
      sha256 = "0wwz48jl9fvl1iknvn9dqr4gfy1qs03gxaikrxxp9gry6773v3sj";
    };
  }
]


1

goDeps is a list of Go dependencies.

2

goPackagePath specifies Go package import path.

3

fetch type that needs to be used to get package source. If git is used there should be url, rev and sha256 defined next to it.

To extract dependency information from a Go package in automated way use go2nix. It can produce complete derivation and goDeps file for Go programs.

buildGoPackage produces Multiple-output packages where bin includes program binaries. You can test build a Go binary as follows:

    $ nix-build -A deis.bin
  

or build all outputs with:

    $ nix-build -A deis.all
  

bin output will be installed by default with nix-env -i or systemPackages.

You may use Go packages installed into the active Nix profiles by adding the following to your ~/.bashrc:

for p in $NIX_PROFILES; do
    GOPATH="$p/share/go:$GOPATH"
done

9.5. User’s Guide to the Haskell Infrastructure

9.5.1. How to install Haskell packages

Nixpkgs distributes build instructions for all Haskell packages registered on Hackage, but strangely enough normal Nix package lookups don’t seem to discover any of them, except for the default version of ghc, cabal-install, and stack:

$ nix-env -i alex
error: selector ‘alex’ matches no derivations
$ nix-env -qa ghc
ghc-7.10.2

The Haskell package set is not registered in the top-level namespace because it is huge. If all Haskell packages were visible to these commands, then name-based search/install operations would be much slower than they are now. We avoided that by keeping all Haskell-related packages in a separate attribute set called haskellPackages, which the following command will list:

$ nix-env -f "<nixpkgs>" -qaP -A haskellPackages
haskellPackages.a50         a50-0.5
haskellPackages.abacate     haskell-abacate-0.0.0.0
haskellPackages.abcBridge   haskell-abcBridge-0.12
haskellPackages.afv         afv-0.1.1
haskellPackages.alex        alex-3.1.4
haskellPackages.Allure      Allure-0.4.101.1
haskellPackages.alms        alms-0.6.7
[... some 8000 entries omitted  ...]

To install any of those packages into your profile, refer to them by their attribute path (first column):

nix-env -f "<nixpkgs>" -iA haskellPackages.Allure ...

The attribute path of any Haskell packages corresponds to the name of that particular package on Hackage: the package cabal-install has the attribute haskellPackages.cabal-install, and so on. (Actually, this convention causes trouble with packages like 3dmodels and 4Blocks, because these names are invalid identifiers in the Nix language. The issue of how to deal with these rare corner cases is currently unresolved.)

Haskell packages who’s Nix name (second column) begins with a haskell- prefix are packages that provide a library whereas packages without that prefix provide just executables. Libraries may provide executables too, though: the package haskell-pandoc, for example, installs both a library and an application. You can install and use Haskell executables just like any other program in Nixpkgs, but using Haskell libraries for development is a bit trickier and we’ll address that subject in great detail in section How to create a development environment.

Attribute paths are deterministic inside of Nixpkgs, but the path necessary to reach Nixpkgs varies from system to system. We dodged that problem by giving nix-env an explicit -f "<nixpkgs>" parameter, but if you call nix-env without that flag, then chances are the invocation fails:

$ nix-env -iA haskellPackages.cabal-install
error: attribute ‘haskellPackages’ in selection path
       ‘haskellPackages.cabal-install’ not found

On NixOS, for example, Nixpkgs does not exist in the top-level namespace by default. To figure out the proper attribute path, it’s easiest to query for the path of a well-known Nixpkgs package, i.e.:

$ nix-env -qaP coreutils
nixos.coreutils  coreutils-8.23

If your system responds like that (most NixOS installations will), then the attribute path to haskellPackages is nixos.haskellPackages. Thus, if you want to use nix-env without giving an explicit -f flag, then that’s the way to do it:

nix-env -qaP -A nixos.haskellPackages
nix-env -iA nixos.haskellPackages.cabal-install

Our current default compiler is GHC 7.10.x and the haskellPackages set contains packages built with that particular version. Nixpkgs contains the latest major release of every GHC since 6.10.4, however, and there is a whole family of package sets available that defines Hackage packages built with each of those compilers, too:

nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc6123
nix-env -f "<nixpkgs>" -qaP -A haskell.packages.ghc763

The name haskellPackages is really just a synonym for haskell.packages.ghc7102, because we prefer that package set internally and recommend it to our users as their default choice, but ultimately you are free to compile your Haskell packages with any GHC version you please. The following command displays the complete list of available compilers:

$ nix-env -f "<nixpkgs>" -qaP -A haskell.compiler
haskell.compiler.ghc6104        ghc-6.10.4
haskell.compiler.ghc6123        ghc-6.12.3
haskell.compiler.ghc704         ghc-7.0.4
haskell.compiler.ghc722         ghc-7.2.2
haskell.compiler.ghc742         ghc-7.4.2
haskell.compiler.ghc763         ghc-7.6.3
haskell.compiler.ghc784         ghc-7.8.4
haskell.compiler.ghc7102        ghc-7.10.2
haskell.compiler.ghcHEAD        ghc-7.11.20150402
haskell.compiler.ghcNokinds     ghc-nokinds-7.11.20150704
haskell.compiler.ghcjs          ghcjs-0.1.0
haskell.compiler.jhc            jhc-0.8.2
haskell.compiler.uhc            uhc-1.1.9.0

We have no package sets for jhc or uhc yet, unfortunately, but for every version of GHC listed above, there exists a package set based on that compiler. Also, the attributes haskell.compiler.ghcXYC and haskell.packages.ghcXYC.ghc are synonymous for the sake of convenience.

9.5.2. How to create a development environment

9.5.2.1. How to install a compiler

A simple development environment consists of a Haskell compiler and one or both of the tools cabal-install and stack. We saw in section How to install Haskell packages how you can install those programs into your user profile:

nix-env -f "<nixpkgs>" -iA haskellPackages.ghc haskellPackages.cabal-install

Instead of the default package set haskellPackages, you can also use the more precise name haskell.compiler.ghc7102, which has the advantage that it refers to the same GHC version regardless of what Nixpkgs considers default at any given time.

Once you’ve made those tools available in $PATH, it’s possible to build Hackage packages the same way people without access to Nix do it all the time:

cabal get lens-4.11 && cd lens-4.11
cabal install -j --dependencies-only
cabal configure
cabal build

If you enjoy working with Cabal sandboxes, then that’s entirely possible too: just execute the command

cabal sandbox init

before installing the required dependencies.

The nix-shell utility makes it easy to switch to a different compiler version; just enter the Nix shell environment with the command

nix-shell -p haskell.compiler.ghc784

to bring GHC 7.8.4 into $PATH. Alternatively, you can use Stack instead of nix-shell directly to select compiler versions and other build tools per-project. It uses nix-shell under the hood when Nix support is turned on. See How to build a Haskell project using Stack.

If you’re using cabal-install, re-running cabal configure inside the spawned shell switches your build to use that compiler instead. If you’re working on a project that doesn’t depend on any additional system libraries outside of GHC, then it’s even sufficient to just run the cabal configure command inside of the shell:

nix-shell -p haskell.compiler.ghc784 --command "cabal configure"

Afterwards, all other commands like cabal build work just fine in any shell environment, because the configure phase recorded the absolute paths to all required tools like GHC in its build configuration inside of the dist/ directory. Please note, however, that nix-collect-garbage can break such an environment because the Nix store paths created by nix-shell aren’t alive anymore once nix-shell has terminated. If you find that your Haskell builds no longer work after garbage collection, then you’ll have to re-run cabal configure inside of a new nix-shell environment.

9.5.2.2. How to install a compiler with libraries

GHC expects to find all installed libraries inside of its own lib directory. This approach works fine on traditional Unix systems, but it doesn’t work for Nix, because GHC’s store path is immutable once it’s built. We cannot install additional libraries into that location. As a consequence, our copies of GHC don’t know any packages except their own core libraries, like base, containers, Cabal, etc.

We can register additional libraries to GHC, however, using a special build function called ghcWithPackages. That function expects one argument: a function that maps from an attribute set of Haskell packages to a list of packages, which determines the libraries known to that particular version of GHC. For example, the Nix expression ghcWithPackages (pkgs: [pkgs.mtl]) generates a copy of GHC that has the mtl library registered in addition to its normal core packages:

$ nix-shell -p "haskellPackages.ghcWithPackages (pkgs: [pkgs.mtl])"

[nix-shell:~]$ ghc-pkg list mtl
/nix/store/zy79...-ghc-7.10.2/lib/ghc-7.10.2/package.conf.d:
    mtl-2.2.1

This function allows users to define their own development environment by means of an override. After adding the following snippet to ~/.config/nixpkgs/config.nix,

{
  packageOverrides = super: let self = super.pkgs; in
  {
    myHaskellEnv = self.haskell.packages.ghc7102.ghcWithPackages
                     (haskellPackages: with haskellPackages; [
                       # libraries
                       arrows async cgi criterion
                       # tools
                       cabal-install haskintex
                     ]);
  };
}

it’s possible to install that compiler with nix-env -f "<nixpkgs>" -iA myHaskellEnv. If you’d like to switch that development environment to a different version of GHC, just replace the ghc7102 bit in the previous definition with the appropriate name. Of course, it’s also possible to define any number of these development environments! (You can’t install two of them into the same profile at the same time, though, because that would result in file conflicts.)

The generated ghc program is a wrapper script that re-directs the real GHC executable to use a new lib directory — one that we specifically constructed to contain all those packages the user requested:

$ cat $(type -p ghc)
#! /nix/store/xlxj...-bash-4.3-p33/bin/bash -e
export NIX_GHC=/nix/store/19sm...-ghc-7.10.2/bin/ghc
export NIX_GHCPKG=/nix/store/19sm...-ghc-7.10.2/bin/ghc-pkg
export NIX_GHC_DOCDIR=/nix/store/19sm...-ghc-7.10.2/share/doc/ghc/html
export NIX_GHC_LIBDIR=/nix/store/19sm...-ghc-7.10.2/lib/ghc-7.10.2
exec /nix/store/j50p...-ghc-7.10.2/bin/ghc "-B$NIX_GHC_LIBDIR" "$@"

The variables $NIX_GHC, $NIX_GHCPKG, etc. point to the new store path ghcWithPackages constructed specifically for this environment. The last line of the wrapper script then executes the real ghc, but passes the path to the new lib directory using GHC’s -B flag.

The purpose of those environment variables is to work around an impurity in the popular ghc-paths library. That library promises to give its users access to GHC’s installation paths. Only, the library can’t possible know that path when it’s compiled, because the path GHC considers its own is determined only much later, when the user configures it through ghcWithPackages. So we patched ghc-paths to return the paths found in those environment variables at run-time rather than trying to guess them at compile-time.

To make sure that mechanism works properly all the time, we recommend that you set those variables to meaningful values in your shell environment, too, i.e. by adding the following code to your ~/.bashrc:

if type >/dev/null 2>&1 -p ghc; then
  eval "$(egrep ^export "$(type -p ghc)")"
fi

If you are certain that you’ll use only one GHC environment which is located in your user profile, then you can use the following code, too, which has the advantage that it doesn’t contain any paths from the Nix store, i.e. those settings always remain valid even if a nix-env -u operation updates the GHC environment in your profile:

if [ -e ~/.nix-profile/bin/ghc ]; then
  export NIX_GHC="$HOME/.nix-profile/bin/ghc"
  export NIX_GHCPKG="$HOME/.nix-profile/bin/ghc-pkg"
  export NIX_GHC_DOCDIR="$HOME/.nix-profile/share/doc/ghc/html"
  export NIX_GHC_LIBDIR="$HOME/.nix-profile/lib/ghc-$($NIX_GHC --numeric-version)"
fi

9.5.2.3. How to install a compiler with libraries, hoogle and documentation indexes

If you plan to use your environment for interactive programming, not just compiling random Haskell code, you might want to replace ghcWithPackages in all the listings above with ghcWithHoogle.

This environment generator not only produces an environment with GHC and all the specified libraries, but also generates a hoogle and haddock indexes for all the packages, and provides a wrapper script around hoogle binary that uses all those things. A precise name for this thing would be ghcWithPackagesAndHoogleAndDocumentationIndexes, which is, regrettably, too long and scary.

For example, installing the following environment

{
  packageOverrides = super: let self = super.pkgs; in
  {
    myHaskellEnv = self.haskellPackages.ghcWithHoogle
                     (haskellPackages: with haskellPackages; [
                       # libraries
                       arrows async cgi criterion
                       # tools
                       cabal-install haskintex
                     ]);
  };
}

allows one to browse module documentation index not too dissimilar to this for all the specified packages and their dependencies by directing a browser of choice to ~/.nix-profiles/share/doc/hoogle/index.html (or /run/current-system/sw/share/doc/hoogle/index.html in case you put it in environment.systemPackages in NixOS).

After you’ve marveled enough at that try adding the following to your ~/.ghc/ghci.conf

:def hoogle \s -> return $ ":! hoogle search -cl --count=15 \"" ++ s ++ "\""
:def doc \s -> return $ ":! hoogle search -cl --info \"" ++ s ++ "\""

and test it by typing into ghci:

:hoogle a -> a
:doc a -> a

Be sure to note the links to haddock files in the output. With any modern and properly configured terminal emulator you can just click those links to navigate there.

Finally, you can run

hoogle server -p 8080

and navigate to http://localhost:8080/ for your own local Hoogle. Note, however, that Firefox and possibly other browsers disallow navigation from http: to file: URIs for security reasons, which might be quite an inconvenience. See this page for workarounds.

9.5.2.4. How to build a Haskell project using Stack

Stack is a popular build tool for Haskell projects. It has first-class support for Nix. Stack can optionally use Nix to automatically select the right version of GHC and other build tools to build, test and execute apps in an existing project downloaded from somewhere on the Internet. Pass the --nix flag to any stack command to do so, e.g.

git clone --recursive http://github.com/yesodweb/wai
cd wai
stack --nix build

If you want stack to use Nix by default, you can add a nix section to the stack.yaml file, as explained in the Stack documentation. For example:

nix:
  enable: true
  packages: [pkgconfig zeromq zlib]

The example configuration snippet above tells Stack to create an ad hoc environment for nix-shell as in the below section, in which the pkgconfig, zeromq and zlib packages from Nixpkgs are available. All stack commands will implicitly be executed inside this ad hoc environment.

Some projects have more sophisticated needs. For examples, some ad hoc environments might need to expose Nixpkgs packages compiled in a certain way, or with extra environment variables. In these cases, you’ll need a shell field instead of packages:

nix:
  enable: true
  shell-file: shell.nix

For more on how to write a shell.nix file see the below section. You’ll need to express a derivation. Note that Nixpkgs ships with a convenience wrapper function around mkDerivation called haskell.lib.buildStackProject to help you create this derivation in exactly the way Stack expects. All of the same inputs as mkDerivation can be provided. For example, to build a Stack project that including packages that link against a version of the R library compiled with special options turned on:

with (import <nixpkgs> { });

let R = pkgs.R.override { enableStrictBarrier = true; };
in
haskell.lib.buildStackProject {
  name = "HaskellR";
  buildInputs = [ R zeromq zlib ];
}

You can select a particular GHC version to compile with by setting the ghc attribute as an argument to buildStackProject. Better yet, let Stack choose what GHC version it wants based on the snapshot specified in stack.yaml (only works with Stack >= 1.1.3):

{nixpkgs ? import <nixpkgs> { }, ghc ? nixpkgs.ghc}:

with nixpkgs;

let R = pkgs.R.override { enableStrictBarrier = true; };
in
haskell.lib.buildStackProject {
  name = "HaskellR";
  buildInputs = [ R zeromq zlib ];
  inherit ghc;
}

9.5.2.5. How to create ad hoc environments for nix-shell

The easiest way to create an ad hoc development environment is to run nix-shell with the appropriate GHC environment given on the command-line:

nix-shell -p "haskellPackages.ghcWithPackages (pkgs: with pkgs; [mtl pandoc])"

For more sophisticated use-cases, however, it’s more convenient to save the desired configuration in a file called shell.nix that looks like this:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
let
  inherit (nixpkgs) pkgs;
  ghc = pkgs.haskell.packages.${compiler}.ghcWithPackages (ps: with ps; [
          monad-par mtl
        ]);
in
pkgs.stdenv.mkDerivation {
  name = "my-haskell-env-0";
  buildInputs = [ ghc ];
  shellHook = "eval $(egrep ^export ${ghc}/bin/ghc)";
}

Now run nix-shell — or even nix-shell --pure — to enter a shell environment that has the appropriate compiler in $PATH. If you use --pure, then add all other packages that your development environment needs into the buildInputs attribute. If you’d like to switch to a different compiler version, then pass an appropriate compiler argument to the expression, i.e. nix-shell --argstr compiler ghc784.

If you need such an environment because you’d like to compile a Hackage package outside of Nix — i.e. because you’re hacking on the latest version from Git —, then the package set provides suitable nix-shell environments for you already! Every Haskell package has an env attribute that provides a shell environment suitable for compiling that particular package. If you’d like to hack the lens library, for example, then you just have to check out the source code and enter the appropriate environment:

$ cabal get lens-4.11 && cd lens-4.11
Downloading lens-4.11...
Unpacking to lens-4.11/

$ nix-shell "<nixpkgs>" -A haskellPackages.lens.env
[nix-shell:/tmp/lens-4.11]$

At point, you can run cabal configure, cabal build, and all the other development commands. Note that you need cabal-install installed in your $PATH already to use it here — the nix-shell environment does not provide it.

9.5.3. How to create Nix builds for your own private Haskell packages

If your own Haskell packages have build instructions for Cabal, then you can convert those automatically into build instructions for Nix using the cabal2nix utility, which you can install into your profile by running nix-env -i cabal2nix.

9.5.3.1. How to build a stand-alone project

For example, let’s assume that you’re working on a private project called foo. To generate a Nix build expression for it, change into the project’s top-level directory and run the command:

cabal2nix . > foo.nix

Then write the following snippet into a file called default.nix:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
nixpkgs.pkgs.haskell.packages.${compiler}.callPackage ./foo.nix { }

Finally, store the following code in a file called shell.nix:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
(import ./default.nix { inherit nixpkgs compiler; }).env

At this point, you can run nix-build to have Nix compile your project and install it into a Nix store path. The local directory will contain a symlink called result after nix-build returns that points into that location. Of course, passing the flag --argstr compiler ghc763 allows switching the build to any version of GHC currently supported.

Furthermore, you can call nix-shell to enter an interactive development environment in which you can use cabal configure and cabal build to develop your code. That environment will automatically contain a proper GHC derivation with all the required libraries registered as well as all the system-level libraries your package might need.

If your package does not depend on any system-level libraries, then it’s sufficient to run

nix-shell --command "cabal configure"

once to set up your build. cabal-install determines the absolute paths to all resources required for the build and writes them into a config file in the dist/ directory. Once that’s done, you can run cabal build and any other command for that project even outside of the nix-shell environment. This feature is particularly nice for those of us who like to edit their code with an IDE, like Emacs’ haskell-mode, because it’s not necessary to start Emacs inside of nix-shell just to make it find out the necessary settings for building the project; cabal-install has already done that for us.

If you want to do some quick-and-dirty hacking and don’t want to bother setting up a default.nix and shell.nix file manually, then you can use the --shell flag offered by cabal2nix to have it generate a stand-alone nix-shell environment for you. With that feature, running

cabal2nix --shell . > shell.nix
nix-shell --command "cabal configure"

is usually enough to set up a build environment for any given Haskell package. You can even use that generated file to run nix-build, too:

nix-build shell.nix

9.5.3.2. How to build projects that depend on each other

If you have multiple private Haskell packages that depend on each other, then you’ll have to register those packages in the Nixpkgs set to make them visible for the dependency resolution performed by callPackage. First of all, change into each of your projects top-level directories and generate a default.nix file with cabal2nix:

cd ~/src/foo && cabal2nix . > default.nix
cd ~/src/bar && cabal2nix . > default.nix

Then edit your ~/.config/nixpkgs/config.nix file to register those builds in the default Haskell package set:

{
  packageOverrides = super: let self = super.pkgs; in
  {
    haskellPackages = super.haskellPackages.override {
      overrides = self: super: {
        foo = self.callPackage ../src/foo {};
        bar = self.callPackage ../src/bar {};
      };
    };
  };
}

Once that’s accomplished, nix-env -f "<nixpkgs>" -qA haskellPackages will show your packages like any other package from Hackage, and you can build them

nix-build "<nixpkgs>" -A haskellPackages.foo

or enter an interactive shell environment suitable for building them:

nix-shell "<nixpkgs>" -A haskellPackages.bar.env

9.5.4. Miscellaneous Topics

9.5.4.1. How to build with profiling enabled

Every Haskell package set takes a function called overrides that you can use to manipulate the package as much as you please. One useful application of this feature is to replace the default mkDerivation function with one that enables library profiling for all packages. To accomplish that, add configure the following snippet in your ~/.config/nixpkgs/config.nix file:

{
  packageOverrides = super: let self = super.pkgs; in
  {
    profiledHaskellPackages = self.haskellPackages.override {
      overrides = self: super: {
        mkDerivation = args: super.mkDerivation (args // {
          enableLibraryProfiling = true;
        });
      };
    };
  };
}

Then, replace instances of haskellPackages in the cabal2nix-generated default.nix or shell.nix files with profiledHaskellPackages.

9.5.4.2. How to override package versions in a compiler-specific package set

Nixpkgs provides the latest version of ghc-events, which is 0.4.4.0 at the time of this writing. This is fine for users of GHC 7.10.x, but GHC 7.8.4 cannot compile that binary. Now, one way to solve that problem is to register an older version of ghc-events in the 7.8.x-specific package set. The first step is to generate Nix build instructions with cabal2nix:

cabal2nix cabal://ghc-events-0.4.3.0 > ~/.nixpkgs/ghc-events-0.4.3.0.nix

Then add the override in ~/.config/nixpkgs/config.nix:

{
  packageOverrides = super: let self = super.pkgs; in
  {
    haskell = super.haskell // {
      packages = super.haskell.packages // {
        ghc784 = super.haskell.packages.ghc784.override {
          overrides = self: super: {
            ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {};
          };
        };
      };
    };
  };
}

This code is a little crazy, no doubt, but it’s necessary because the intuitive version

{ # ...

  haskell.packages.ghc784 = super.haskell.packages.ghc784.override {
    overrides = self: super: {
      ghc-events = self.callPackage ./ghc-events-0.4.3.0.nix {};
    };
  };
}

doesn’t do what we want it to: that code replaces the haskell package set in Nixpkgs with one that contains only one entry,packages, which contains only one entry ghc784. This override loses the haskell.compiler set, and it loses the haskell.packages.ghcXYZ sets for all compilers but GHC 7.8.4. To avoid that problem, we have to perform the convoluted little dance from above, iterating over each step in hierarchy.

Once it’s accomplished, however, we can install a variant of ghc-events that’s compiled with GHC 7.8.4:

nix-env -f "<nixpkgs>" -iA haskell.packages.ghc784.ghc-events

Unfortunately, it turns out that this build fails again while executing the test suite! Apparently, the release archive on Hackage is missing some data files that the test suite requires, so we cannot run it. We accomplish that by re-generating the Nix expression with the --no-check flag:

cabal2nix --no-check cabal://ghc-events-0.4.3.0 > ~/.nixpkgs/ghc-events-0.4.3.0.nix

Now the builds succeeds.

Of course, in the concrete example of ghc-events this whole exercise is not an ideal solution, because ghc-events can analyze the output emitted by any version of GHC later than 6.12 regardless of the compiler version that was used to build the ghc-events executable, so strictly speaking there’s no reason to prefer one built with GHC 7.8.x in the first place. However, for users who cannot use GHC 7.10.x at all for some reason, the approach of downgrading to an older version might be useful.

9.5.4.3. How to recover from GHC’s infamous non-deterministic library ID bug

GHC and distributed build farms don’t get along well:

  • https://ghc.haskell.org/trac/ghc/ticket/4012

When you see an error like this one

package foo-0.7.1.0 is broken due to missing package
text-1.2.0.4-98506efb1b9ada233bb5c2b2db516d91

then you have to download and re-install foo and all its dependents from scratch:

nix-store -q --referrers /nix/store/*-haskell-text-1.2.0.4 \
  | xargs -L 1 nix-store --repair-path

If you’re using additional Hydra servers other than hydra.nixos.org, then it might be necessary to purge the local caches that store data from those machines to disable these binary channels for the duration of the previous command, i.e. by running:

rm /nix/var/nix/binary-cache-v3.sqlite
rm /nix/var/nix/manifests/*
rm /nix/var/nix/channel-cache/*

9.5.4.4. Builds on Darwin fail with math.h not found

Users of GHC on Darwin have occasionally reported that builds fail, because the compiler complains about a missing include file:

fatal error: 'math.h' file not found

The issue has been discussed at length in ticket 6390, and so far no good solution has been proposed. As a work-around, users who run into this problem can configure the environment variables

export NIX_CFLAGS_COMPILE="-idirafter /usr/include"
export NIX_CFLAGS_LINK="-L/usr/lib"

in their ~/.bashrc file to avoid the compiler error.

9.5.4.5. Builds using Stack complain about missing system libraries

--  While building package zlib-0.5.4.2 using:
  runhaskell -package=Cabal-1.22.4.0 -clear-package-db [... lots of flags ...]
Process exited with code: ExitFailure 1
Logs have been written to: /home/foo/src/stack-ide/.stack-work/logs/zlib-0.5.4.2.log

Configuring zlib-0.5.4.2...
Setup.hs: Missing dependency on a foreign library:
* Missing (or bad) header file: zlib.h
This problem can usually be solved by installing the system package that
provides this library (you may need the "-dev" version). If the library is
already installed but in a non-standard location then you can use the flags
--extra-include-dirs= and --extra-lib-dirs= to specify where it is.
If the header file does exist, it may contain errors that are caught by the C
compiler at the preprocessing stage. In this case you can re-run configure
with the verbosity flag -v3 to see the error messages.

When you run the build inside of the nix-shell environment, the system is configured to find libz.so without any special flags – the compiler and linker just know how to find it. Consequently, Cabal won’t record any search paths for libz.so in the package description, which means that the package works fine inside of nix-shell, but once you leave the shell the shared object can no longer be found. That issue is by no means specific to Stack: you’ll have that problem with any other Haskell package that’s built inside of nix-shell but run outside of that environment.

You can remedy this issue in several ways. The easiest is to add a nix section to the stack.yaml like the following:

nix:
  enable: true
  packages: [ zlib ]

Stack’s Nix support knows to add ${zlib.out}/lib and ${zlib.dev}/include as an --extra-lib-dirs and extra-include-dirs, respectively. Alternatively, you can achieve the same effect by hand. First of all, run

$ nix-build --no-out-link "<nixpkgs>" -A zlib
/nix/store/alsvwzkiw4b7ip38l4nlfjijdvg3fvzn-zlib-1.2.8

to find out the store path of the system’s zlib library. Now, you can

  1. add that path (plus a /lib suffix) to your $LD_LIBRARY_PATH environment variable to make sure your system linker finds libz.so automatically. It’s no pretty solution, but it will work.

  2. As a variant of (1), you can also install any number of system libraries into your user’s profile (or some other profile) and point $LD_LIBRARY_PATH to that profile instead, so that you don’t have to list dozens of those store paths all over the place.

  3. The solution I prefer is to call stack with an appropriate –extra-lib-dirs flag like so:

    stack --extra-lib-dirs=/nix/store/alsvwzkiw4b7ip38l4nlfjijdvg3fvzn-zlib-1.2.8/lib build
    

    Typically, you’ll need --extra-include-dirs as well. It’s possible to add those flag to the project’s stack.yaml or your user’s global ~/.stack/global/stack.yaml file so that you don’t have to specify them manually every time. But again, you’re likely better off using Stack’s Nix support instead.

    The same thing applies to cabal configure, of course, if you’re building with cabal-install instead of Stack.

9.5.4.6. Creating statically linked binaries

There are two levels of static linking. The first option is to configure the build with the Cabal flag --disable-executable-dynamic. In Nix expressions, this can be achieved by setting the attribute:

enableSharedExecutables = false;

That gives you a binary with statically linked Haskell libraries and dynamically linked system libraries.

To link both Haskell libraries and system libraries statically, the additional flags --ghc-option=-optl=-static --ghc-option=-optl=-pthread need to be used. In Nix, this is accomplished with:

configureFlags = [ "--ghc-option=-optl=-static" "--ghc-option=-optl=-pthread" ];

It’s important to realize, however, that most system libraries in Nix are built as shared libraries only, i.e. there is just no static library available that Cabal could link!

9.5.4.7. Building GHC with integer-simple

By default GHC implements the Integer type using the GNU Multiple Precision Arithmetic (GMP) library. The implementation can be found in the integer-gmp package.

A potential problem with this is that GMP is licensed under the GNU Lesser General Public License (LGPL), a kind of copyleft license. According to the terms of the LGPL, paragraph 5, you may distribute a program that is designed to be compiled and dynamically linked with the library under the terms of your choice (i.e., commercially) but if your program incorporates portions of the library, if it is linked statically, then your program is a derivative–a work based on the library–and according to paragraph 2, section c, you must cause the whole of the work to be licensed under the terms of the LGPL (including for free).

The LGPL licensing for GMP is a problem for the overall licensing of binary programs compiled with GHC because most distributions (and builds) of GHC use static libraries. (Dynamic libraries are currently distributed only for macOS.) The LGPL licensing situation may be worse: even though The Glasgow Haskell Compiler License is essentially a free software license (BSD3), according to paragraph 2 of the LGPL, GHC must be distributed under the terms of the LGPL!

To work around these problems GHC can be build with a slower but LGPL-free alternative implemention for Integer called integer-simple.

To get a GHC compiler build with integer-simple instead of integer-gmp use the attribute: haskell.compiler.integer-simple."${ghcVersion}". For example:

$ nix-build -E '(import <nixpkgs> {}).haskell.compiler.integer-simple.ghc802'
...
$ result/bin/ghc-pkg list | grep integer
    integer-simple-0.1.1.1

The following command displays the complete list of GHC compilers build with integer-simple:

$ nix-env -f "<nixpkgs>" -qaP -A haskell.compiler.integer-simple
haskell.compiler.integer-simple.ghc7102  ghc-7.10.2
haskell.compiler.integer-simple.ghc7103  ghc-7.10.3
haskell.compiler.integer-simple.ghc722   ghc-7.2.2
haskell.compiler.integer-simple.ghc742   ghc-7.4.2
haskell.compiler.integer-simple.ghc783   ghc-7.8.3
haskell.compiler.integer-simple.ghc784   ghc-7.8.4
haskell.compiler.integer-simple.ghc801   ghc-8.0.1
haskell.compiler.integer-simple.ghc802   ghc-8.0.2
haskell.compiler.integer-simple.ghcHEAD  ghc-8.1.20170106

To get a package set supporting integer-simple use the attribute: haskell.packages.integer-simple."${ghcVersion}". For example use the following to get the scientific package build with integer-simple:

nix-build -A haskell.packages.integer-simple.ghc802.scientific

9.5.4.8. Quality assurance

The haskell.lib library includes a number of functions for checking for various imperfections in Haskell packages. It’s useful to apply these functions to your own Haskell packages and integrate that in a Continuous Integration server like hydra to assure your packages maintain a minimum level of quality. This section discusses some of these functions.

9.5.4.8.1. buildStrictly

Applying haskell.lib.buildStrictly to a Haskell package enables the -Wall and -Werror GHC options to turn all warnings into build failures. Additionally the source of your package is gotten from first invoking cabal sdist to ensure all needed files are listed in the Cabal file.

9.5.4.8.2. checkUnusedPackages

Applying haskell.lib.checkUnusedPackages to a Haskell package invokes the packunused tool on the package. packunused complains when it finds packages listed as build-depends in the Cabal file which are redundant. For example:

$ nix-build -E 'let pkgs = import <nixpkgs> {}; in pkgs.haskell.lib.checkUnusedPackages {} pkgs.haskellPackages.scientific'
these derivations will be built:
  /nix/store/3lc51cxj2j57y3zfpq5i69qbzjpvyci1-scientific-0.3.5.1.drv
...
detected package components
~~~~~~~~~~~~~~~~~~~~~~~~~~~

 - library
 - testsuite(s): test-scientific
 - benchmark(s): bench-scientific*

(component names suffixed with '*' are not configured to be built)

library
~~~~~~~

The following package dependencies seem redundant:

 - ghc-prim-0.5.0.0

testsuite(test-scientific)
~~~~~~~~~~~~~~~~~~~~~~~~~~

no redundant packages dependencies found

builder for ‘/nix/store/3lc51cxj2j57y3zfpq5i69qbzjpvyci1-scientific-0.3.5.1.drv’ failed with exit code 1
error: build of ‘/nix/store/3lc51cxj2j57y3zfpq5i69qbzjpvyci1-scientific-0.3.5.1.drv’ failed

As you can see, packunused finds out that although the testsuite component has no redundant dependencies the library component of scientific-0.3.5.1 depends on ghc-prim which is unused in the library.

9.5.5. Other resources

  • The Youtube video Nix Loves Haskell provides an introduction into Haskell NG aimed at beginners. The slides are available at http://cryp.to/nixos-meetup-3-slides.pdf and also – in a form ready for cut & paste – at https://github.com/NixOS/cabal2nix/blob/master/doc/nixos-meetup-3-slides.md.

  • Another Youtube video is Escaping Cabal Hell with Nix, which discusses the subject of Haskell development with Nix but also provides a basic introduction to Nix as well, i.e. it’s suitable for viewers with almost no prior Nix experience.

  • Oliver Charles wrote a very nice Tutorial how to develop Haskell packages with Nix.

  • The Journey into the Haskell NG infrastructure series of postings describe the new Haskell infrastructure in great detail:

    • Part 1 explains the differences between the old and the new code and gives instructions how to migrate to the new setup.

    • Part 2 looks in-depth at how to tweak and configure your setup by means of overrides.

    • Part 3 describes the infrastructure that keeps the Haskell package set in Nixpkgs up-to-date.

9.6. Idris packages

This directory contains build rules for idris packages. In addition, it contains several functions to build and compose those packages. Everything is exposed to the user via the idrisPackages attribute.

9.6.1. callPackage

This is like the normal nixpkgs callPackage function, specialized to idris packages.

9.6.2. builtins

This is a list of all of the libraries that come packaged with Idris itself.

9.6.3. build-idris-package

A function to build an idris package. Its sole argument is a set like you might pass to stdenv.mkDerivation, except build-idris-package sets several attributes for you. See build-idris-package.nix for details.

9.6.4. build-builtin-package

A version of build-idris-package specialized to builtin libraries. Mostly for internal use.

9.6.5. with-packages

Bundle idris together with a list of packages. Because idris currently only supports a single directory in its library path, you must include all desired libraries here, including prelude and base.

9.7. Java

Ant-based Java packages are typically built from source as follows:

stdenv.mkDerivation {
  name = "...";
  src = fetchurl { ... };

  buildInputs = [ jdk ant ];

  buildPhase = "ant";
}

Note that jdk is an alias for the OpenJDK.

JAR files that are intended to be used by other packages should be installed in $out/share/java. The OpenJDK has a stdenv setup hook that adds any JARs in the share/java directories of the build inputs to the CLASSPATH environment variable. For instance, if the package libfoo installs a JAR named foo.jar in its share/java directory, and another package declares the attribute

buildInputs = [ jdk libfoo ];

then CLASSPATH will be set to /nix/store/...-libfoo/share/java/foo.jar.

Private JARs should be installed in a location like $out/share/package-name.

If your Java package provides a program, you need to generate a wrapper script to run it using the OpenJRE. You can use makeWrapper for this:

buildInputs = [ makeWrapper ];

installPhase =
  ''
    mkdir -p $out/bin
    makeWrapper ${jre}/bin/java $out/bin/foo \
      --add-flags "-cp $out/share/java/foo.jar org.foo.Main"
  '';

Note the use of jre, which is the part of the OpenJDK package that contains the Java Runtime Environment. By using ${jre}/bin/java instead of ${jdk}/bin/java, you prevent your package from depending on the JDK at runtime.

It is possible to use a different Java compiler than javac from the OpenJDK. For instance, to use the Eclipse Java Compiler:

buildInputs = [ jre ant ecj ];

(Note that here you don’t need the full JDK as an input, but just the JRE.) The ECJ has a stdenv setup hook that sets some environment variables to cause Ant to use ECJ, but this doesn’t work with all Ant files. Similarly, you can use the GNU Java Compiler:

buildInputs = [ gcj ant ];

Here, Ant will automatically use gij (the GNU Java Runtime) instead of the OpenJRE.

9.8. Lua

Lua packages are built by the buildLuaPackage function. This function is implemented in pkgs/development/lua-modules/generic/default.nix and works similarly to buildPerlPackage. (See Section 9.10, “Perl” for details.)

Lua packages are defined in pkgs/top-level/lua-packages.nix. Most of them are simple. For example:

fileSystem = buildLuaPackage {
  name = "filesystem-1.6.2";
  src = fetchurl {
    url = "https://github.com/keplerproject/luafilesystem/archive/v1_6_2.tar.gz";
    sha256 = "1n8qdwa20ypbrny99vhkmx8q04zd2jjycdb5196xdhgvqzk10abz";
  };
  meta = {
    homepage = "https://github.com/keplerproject/luafilesystem";
    hydraPlatforms = stdenv.lib.platforms.linux;
    maintainers = with maintainers; [ flosse ];
  };
};
  

Though, more complicated package should be placed in a seperate file in pkgs/development/lua-modules.

Lua packages accept additional parameter disabled, which defines the condition of disabling package from luaPackages. For example, if package has disabled assigned to lua.luaversion != "5.1", it will not be included in any luaPackages except lua51Packages, making it only be built for lua 5.1.

9.9. Node.js packages

The pkgs/development/node-packages folder contains a generated collection of NPM packages that can be installed with the Nix package manager.

As a rule of thumb, the package set should only provide end user software packages, such as command-line utilities. Libraries should only be added to the package set if there is a non-NPM package that requires it.

When it is desired to use NPM libraries in a development project, use the node2nix generator directly on the package.json configuration file of the project.

The package set also provides support for multiple Node.js versions. The policy is that a new package should be added to the collection for the latest stable LTS release (which is currently 6.x), unless there is an explicit reason to support a different release.

To add a package from NPM to nixpkgs:

  1. Modify pkgs/development/node-packages/node-packages-v6.json to add, update or remove package entries. (Or pkgs/development/node-packages/node-packages-v4.json for packages depending on Node.js 4.x)

  2. Run the script: (cd pkgs/development/node-packages && ./generate.sh).

  3. Build your new package to test your changes: cd /path/to/nixpkgs && nix-build -A nodePackages.<new-or-updated-package>. To build against a specific Node.js version (e.g. 4.x): nix-build -A nodePackages_4_x.<new-or-updated-package>

  4. Add and commit all modified and generated files.

For more information about the generation process, consult the README.md file of the node2nix tool.

9.10. Perl

Nixpkgs provides a function buildPerlPackage, a generic package builder function for any Perl package that has a standard Makefile.PL. It’s implemented in pkgs/development/perl-modules/generic.

Perl packages from CPAN are defined in pkgs/top-level/perl-packages.nix, rather than pkgs/all-packages.nix. Most Perl packages are so straight-forward to build that they are defined here directly, rather than having a separate function for each package called from perl-packages.nix. However, more complicated packages should be put in a separate file, typically in pkgs/development/perl-modules. Here is an example of the former:

ClassC3 = buildPerlPackage rec {
  name = "Class-C3-0.21";
  src = fetchurl {
    url = "mirror://cpan/authors/id/F/FL/FLORA/${name}.tar.gz";
    sha256 = "1bl8z095y4js66pwxnm7s853pi9czala4sqc743fdlnk27kq94gz";
  };
};

Note the use of mirror://cpan/, and the ${name} in the URL definition to ensure that the name attribute is consistent with the source that we’re actually downloading. Perl packages are made available in all-packages.nix through the variable perlPackages. For instance, if you have a package that needs ClassC3, you would typically write

foo = import ../path/to/foo.nix {
  inherit stdenv fetchurl ...;
  inherit (perlPackages) ClassC3;
};

in all-packages.nix. You can test building a Perl package as follows:

$ nix-build -A perlPackages.ClassC3

buildPerlPackage adds perl- to the start of the name attribute, so the package above is actually called perl-Class-C3-0.21. So to install it, you can say:

$ nix-env -i perl-Class-C3

(Of course you can also install using the attribute name: nix-env -i -A perlPackages.ClassC3.)

So what does buildPerlPackage do? It does the following:

  1. In the configure phase, it calls perl Makefile.PL to generate a Makefile. You can set the variable makeMakerFlags to pass flags to Makefile.PL

  2. It adds the contents of the PERL5LIB environment variable to #! .../bin/perl line of Perl scripts as -Idir flags. This ensures that a script can find its dependencies.

  3. In the fixup phase, it writes the propagated build inputs (propagatedBuildInputs) to the file $out/nix-support/propagated-user-env-packages. nix-env recursively installs all packages listed in this file when you install a package that has it. This ensures that a Perl package can find its dependencies.

buildPerlPackage is built on top of stdenv, so everything can be customised in the usual way. For instance, the BerkeleyDB module has a preConfigure hook to generate a configuration file used by Makefile.PL:

{ buildPerlPackage, fetchurl, db }:

buildPerlPackage rec {
  name = "BerkeleyDB-0.36";

  src = fetchurl {
    url = "mirror://cpan/authors/id/P/PM/PMQS/${name}.tar.gz";
    sha256 = "07xf50riarb60l1h6m2dqmql8q5dij619712fsgw7ach04d8g3z1";
  };

  preConfigure = ''
    echo "LIB = ${db}/lib" > config.in
    echo "INCLUDE = ${db}/include" >> config.in
  '';
}

Dependencies on other Perl packages can be specified in the buildInputs and propagatedBuildInputs attributes. If something is exclusively a build-time dependency, use buildInputs; if it’s (also) a runtime dependency, use propagatedBuildInputs. For instance, this builds a Perl module that has runtime dependencies on a bunch of other modules:

ClassC3Componentised = buildPerlPackage rec {
  name = "Class-C3-Componentised-1.0004";
  src = fetchurl {
    url = "mirror://cpan/authors/id/A/AS/ASH/${name}.tar.gz";
    sha256 = "0xql73jkcdbq4q9m0b0rnca6nrlvf5hyzy8is0crdk65bynvs8q1";
  };
  propagatedBuildInputs = [
    ClassC3 ClassInspector TestException MROCompat
  ];
};

9.10.1. Generation from CPAN

Nix expressions for Perl packages can be generated (almost) automatically from CPAN. This is done by the program nix-generate-from-cpan, which can be installed as follows:

$ nix-env -i nix-generate-from-cpan

This program takes a Perl module name, looks it up on CPAN, fetches and unpacks the corresponding package, and prints a Nix expression on standard output. For example:

$ nix-generate-from-cpan XML::Simple
  XMLSimple = buildPerlPackage rec {
    name = "XML-Simple-2.22";
    src = fetchurl {
      url = "mirror://cpan/authors/id/G/GR/GRANTM/${name}.tar.gz";
      sha256 = "b9450ef22ea9644ae5d6ada086dc4300fa105be050a2030ebd4efd28c198eb49";
    };
    propagatedBuildInputs = [ XMLNamespaceSupport XMLSAX XMLSAXExpat ];
    meta = {
      description = "An API for simple XML files";
      license = with stdenv.lib.licenses; [ artistic1 gpl1Plus ];
    };
  };

The output can be pasted into pkgs/top-level/perl-packages.nix or wherever else you need it.

9.11. Python

9.11.1. User Guide

9.11.1.1. Using Python

9.11.1.1.1. Overview

Several versions of the Python interpreter are available on Nix, as well as a high amount of packages. The attribute python refers to the default interpreter, which is currently CPython 2.7. It is also possible to refer to specific versions, e.g. python35 refers to CPython 3.5, and pypy refers to the default PyPy interpreter.

Python is used a lot, and in different ways. This affects also how it is packaged. In the case of Python on Nix, an important distinction is made between whether the package is considered primarily an application, or whether it should be used as a library, i.e., of primary interest are the modules in site-packages that should be importable.

In the Nixpkgs tree Python applications can be found throughout, depending on what they do, and are called from the main package set. Python libraries, however, are in separate sets, with one set per interpreter version.

The interpreters have several common attributes. One of these attributes is pkgs, which is a package set of Python libraries for this specific interpreter. E.g., the toolz package corresponding to the default interpreter is python.pkgs.toolz, and the CPython 3.5 version is python35.pkgs.toolz. The main package set contains aliases to these package sets, e.g. pythonPackages refers to python.pkgs and python35Packages to python35.pkgs.

9.11.1.1.2. Installing Python and packages

The Nix and NixOS manuals explain how packages are generally installed. In the case of Python and Nix, it is important to make a distinction between whether the package is considered an application or a library.

Applications on Nix are typically installed into your user profile imperatively using nix-env -i, and on NixOS declaratively by adding the package name to environment.systemPackages in /etc/nixos/configuration.nix. Dependencies such as libraries are automatically installed and should not be installed explicitly.

The same goes for Python applications and libraries. Python applications can be installed in your profile. But Python libraries you would like to use for development cannot be installed, at least not individually, because they won’t be able to find each other resulting in import errors. Instead, it is possible to create an environment with python.buildEnv or python.withPackages where the interpreter and other executables are able to find each other and all of the modules.

In the following examples we create an environment with Python 3.5, numpy and toolz. As you may imagine, there is one limitation here, and that’s that you can install only one environment at a time. You will notice the complaints about collisions when you try to install a second environment.

9.11.1.1.2.1. Environment defined in separate .nix file

Create a file, e.g. build.nix, with the following expression

with import <nixpkgs> {};

python35.withPackages (ps: with ps; [ numpy toolz ])

and install it in your profile with

nix-env -if build.nix

Now you can use the Python interpreter, as well as the extra packages (numpy, toolz) that you added to the environment.

9.11.1.1.2.2. Environment defined in ~/.config/nixpkgs/config.nix

If you prefer to, you could also add the environment as a package override to the Nixpkgs set, e.g. using config.nix,

{ # ...

  packageOverrides = pkgs: with pkgs; {
    myEnv = python35.withPackages (ps: with ps; [ numpy toolz ]);
  };
}

and install it in your profile with

nix-env -iA nixpkgs.myEnv

The environment is is installed by referring to the attribute, and considering the nixpkgs channel was used.

9.11.1.1.2.3. Environment defined in /etc/nixos/configuration.nix

For the sake of completeness, here’s another example how to install the environment system-wide.

{ # ...

  environment.systemPackages = with pkgs; [
    (python35.withPackages(ps: with ps; [ numpy toolz ]))
  ];
}
9.11.1.1.3. Temporary Python environment with nix-shell

The examples in the previous section showed how to install a Python environment into a profile. For development you may need to use multiple environments. nix-shell gives the possibility to temporarily load another environment, akin to virtualenv.

There are two methods for loading a shell with Python packages. The first and recommended method is to create an environment with python.buildEnv or python.withPackages and load that. E.g.

$ nix-shell -p 'python35.withPackages(ps: with ps; [ numpy toolz ])'

opens a shell from which you can launch the interpreter

[nix-shell:~] python3

The other method, which is not recommended, does not create an environment and requires you to list the packages directly,

$ nix-shell -p python35.pkgs.numpy python35.pkgs.toolz

Again, it is possible to launch the interpreter from the shell. The Python interpreter has the attribute pkgs which contains all Python libraries for that specific interpreter.

9.11.1.1.3.1. Load environment from .nix expression

As explained in the Nix manual, nix-shell can also load an expression from a .nix file. Say we want to have Python 3.5, numpy and toolz, like before, in an environment. Consider a shell.nix file with

with import <nixpkgs> {};

python35.withPackages (ps: [ps.numpy ps.toolz])

Executing nix-shell gives you again a Nix shell from which you can run Python.

What’s happening here?

  1. We begin with importing the Nix Packages collections. import <nixpkgs> imports the <nixpkgs> function, {} calls it and the with statement brings all attributes of nixpkgs in the local scope. These attributes form the main package set.

  2. Then we create a Python 3.5 environment with the withPackages function.

  3. The withPackages function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages numpy and toolz from the package set.

9.11.1.1.3.2. Execute command with --run

A convenient option with nix-shell is the --run option, with which you can execute a command in the nix-shell. We can e.g. directly open a Python shell

$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3"

or run a script

$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py"
9.11.1.1.3.3. nix-shell as shebang

In fact, for the second use case, there is a more convenient method. You can add a shebang to your script specifying which dependencies nix-shell needs. With the following shebang, you can just execute ./myscript.py, and it will make available all dependencies and run the script in the python3 shell.

#! /usr/bin/env nix-shell
#! nix-shell -i 'python3.withPackages(ps: [ps.numpy])'

import numpy

print(numpy.__version__)

9.11.1.2. Developing with Python

Now that you know how to get a working Python environment with Nix, it is time to go forward and start actually developing with Python. We will first have a look at how Python packages are packaged on Nix. Then, we will look at how you can use development mode with your code.

9.11.1.2.1. Packaging a library

With Nix all packages are built by functions. The main function in Nix for building Python libraries is buildPythonPackage. Let’s see how we can build the toolz package.

{ # ...

  toolz = buildPythonPackage rec {
    pname = "toolz";
    version = "0.7.4";
    name = "${pname}-${version}";

    src = fetchPypi {
      inherit pname version;
      sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
    };

    doCheck = false;

    meta = {
      homepage = "http://github.com/pytoolz/toolz/";
      description = "List processing tools and functional utilities";
      license = licenses.bsd3;
      maintainers = with maintainers; [ fridh ];
    };
  };
}

What happens here? The function buildPythonPackage is called and as argument it accepts a set. In this case the set is a recursive set, rec. One of the arguments is the name of the package, which consists of a basename (generally following the name on PyPi) and a version. Another argument, src specifies the source, which in this case is fetched from PyPI using the helper function fetchPypi. The argument doCheck is used to set whether tests should be run when building the package. Furthermore, we specify some (optional) meta information. The output of the function is a derivation.

An expression for toolz can be found in the Nixpkgs repository. As explained in the introduction of this Python section, a derivation of toolz is available for each interpreter version, e.g. python35.pkgs.toolz refers to the toolz derivation corresponding to the CPython 3.5 interpreter. The above example works when you’re directly working on pkgs/top-level/python-packages.nix in the Nixpkgs repository. Often though, you will want to test a Nix expression outside of the Nixpkgs tree.

The following expression creates a derivation for the toolz package, and adds it along with a numpy package to a Python environment.

with import <nixpkgs> {};

( let
    my_toolz = python35.pkgs.buildPythonPackage rec {
      pname = "toolz";
      version = "0.7.4";
      name = "${pname}-${version}";

      src = python35.pkgs.fetchPypi {
        inherit pname version;
        sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
      };

      doCheck = false;

      meta = {
        homepage = "http://github.com/pytoolz/toolz/";
        description = "List processing tools and functional utilities";
      };
    };

  in python35.withPackages (ps: [ps.numpy my_toolz])
).env

Executing nix-shell will result in an environment in which you can use Python 3.5 and the toolz package. As you can see we had to explicitly mention for which Python version we want to build a package.

So, what did we do here? Well, we took the Nix expression that we used earlier to build a Python environment, and said that we wanted to include our own version of toolz, named my_toolz. To introduce our own package in the scope of withPackages we used a let expression. You can see that we used ps.numpy to select numpy from the nixpkgs package set (ps). We did not take toolz from the Nixpkgs package set this time, but instead took our own version that we introduced with the let expression.

9.11.1.2.2. Handling dependencies

Our example, toolz, does not have any dependencies on other Python packages or system libraries. According to the manual, buildPythonPackage uses the arguments buildInputs and propagatedBuildInputs to specify dependencies. If something is exclusively a build-time dependency, then the dependency should be included as a buildInput, but if it is (also) a runtime dependency, then it should be added to propagatedBuildInputs. Test dependencies are considered build-time dependencies.

The following example shows which arguments are given to buildPythonPackage in order to build datashape.

{ # ...

  datashape = buildPythonPackage rec {
    name = "datashape-${version}";
    version = "0.4.7";

    src = pkgs.fetchurl {
      url = "mirror://pypi/D/DataShape/${name}.tar.gz";
      sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
    };

    buildInputs = with self; [ pytest ];
    propagatedBuildInputs = with self; [ numpy multipledispatch dateutil ];

    meta = {
      homepage = https://github.com/ContinuumIO/datashape;
      description = "A data description language";
      license = licenses.bsd2;
      maintainers = with maintainers; [ fridh ];
    };
  };
}

We can see several runtime dependencies, numpy, multipledispatch, and dateutil. Furthermore, we have one buildInput, i.e. pytest. pytest is a test runner and is only used during the checkPhase and is therefore not added to propagatedBuildInputs.

In the previous case we had only dependencies on other Python packages to consider. Occasionally you have also system libraries to consider. E.g., lxml provides Python bindings to libxml2 and libxslt. These libraries are only required when building the bindings and are therefore added as buildInputs.

{ # ...

  lxml = buildPythonPackage rec {
    name = "lxml-3.4.4";

    src = pkgs.fetchurl {
      url = "mirror://pypi/l/lxml/${name}.tar.gz";
      sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
    };

    buildInputs = with self; [ pkgs.libxml2 pkgs.libxslt ];

    meta = {
      description = "Pythonic binding for the libxml2 and libxslt libraries";
      homepage = http://lxml.de;
      license = licenses.bsd3;
      maintainers = with maintainers; [ sjourdois ];
    };
  };
}

In this example lxml and Nix are able to work out exactly where the relevant files of the dependencies are. This is not always the case.

The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as FFTW. On Nix we have separate packages of FFTW for the different types of floats ("single", "double", "long-double"). The bindings need all three types, and therefore we add all three as buildInputs. The bindings don’t expect to find each of them in a different folder, and therefore we have to set LDFLAGS and CFLAGS.

{ # ...

  pyfftw = buildPythonPackage rec {
    name = "pyfftw-${version}";
    version = "0.9.2";

    src = pkgs.fetchurl {
      url = "mirror://pypi/p/pyFFTW/pyFFTW-${version}.tar.gz";
      sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
    };

    buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];

    propagatedBuildInputs = with self; [ numpy scipy ];

    # Tests cannot import pyfftw. pyfftw works fine though.
    doCheck = false;

    preConfigure = ''
      export LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
      export CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
    '';

    meta = {
      description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
      homepage = http://hgomersall.github.com/pyFFTW/;
      license = with licenses; [ bsd2 bsd3 ];
      maintainer = with maintainers; [ fridh ];
    };
  };
}

Note also the line doCheck = false;, we explicitly disabled running the test-suite.

9.11.1.2.3. Develop local package

As a Python developer you’re likely aware of development mode (python setup.py develop); instead of installing the package this command creates a special link to the project code. That way, you can run updated code without having to reinstall after each and every change you make. Development mode is also available. Let’s see how you can use it.

In the previous Nix expression the source was fetched from an url. We can also refer to a local source instead using src = ./path/to/source/tree;

If we create a shell.nix file which calls buildPythonPackage, and if src is a local source, and if the local source has a setup.py, then development mode is activated.

In the following example we create a simple environment that has a Python 3.5 version of our package in it, as well as its dependencies and other packages we like to have in the environment, all specified with propagatedBuildInputs. Indeed, we can just add any package we like to have in our environment to propagatedBuildInputs.

with import <nixpkgs> {};
with pkgs.python35Packages;

buildPythonPackage rec {
  name = "mypackage";
  src = ./path/to/package/source;
  propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
}

It is important to note that due to how development mode is implemented on Nix it is not possible to have multiple packages simultaneously in development mode.

9.11.1.3. Organising your packages

So far we discussed how you can use Python on Nix, and how you can develop with it. We’ve looked at how you write expressions to package Python packages, and we looked at how you can create environments in which specified packages are available.

At some point you’ll likely have multiple packages which you would like to be able to use in different projects. In order to minimise unnecessary duplication we now look at how you can maintain yourself a repository with your own packages. The important functions here are import and callPackage.

9.11.1.4. Including a derivation using callPackage

Earlier we created a Python environment using withPackages, and included the toolz package via a let expression. Let’s split the package definition from the environment definition.

We first create a function that builds toolz in ~/path/to/toolz/release.nix

{ pkgs, buildPythonPackage }:

buildPythonPackage rec {
  name = "toolz-${version}";
  version = "0.7.4";

  src = pkgs.fetchurl {
    url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
    sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
  };

  meta = {
    homepage = "http://github.com/pytoolz/toolz/";
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
}

It takes two arguments, pkgs and buildPythonPackage. We now call this function using callPackage in the definition of our environment

with import <nixpkgs> {};

( let
    toolz = pkgs.callPackage /path/to/toolz/release.nix {
      pkgs = pkgs;
      buildPythonPackage = pkgs.python35Packages.buildPythonPackage;
    };
  in pkgs.python35.withPackages (ps: [ ps.numpy toolz ])
).env

Important to remember is that the Python version for which the package is made depends on the python derivation that is passed to buildPythonPackage. Nix tries to automatically pass arguments when possible, which is why generally you don’t explicitly define which python derivation should be used. In the above example we use buildPythonPackage that is part of the set python35Packages, and in this case the python35 interpreter is automatically used.

9.11.2. Reference

9.11.2.1. Interpreters

Versions 2.7, 3.3, 3.4, 3.5 and 3.6 of the CPython interpreter are available as respectively python27, python34, python35 and python36. The PyPy interpreter is available as pypy. The aliases python2 and python3 correspond to respectively python27 and python35. The default interpreter, python, maps to python2. The Nix expressions for the interpreters can be found in pkgs/development/interpreters/python.

All packages depending on any Python interpreter get appended out/{python.sitePackages} to $PYTHONPATH if such directory exists.

9.11.2.1.1. Missing tkinter module standard library

To reduce closure size the Tkinter/tkinter is available as a separate package, pythonPackages.tkinter.

9.11.2.1.2. Attributes on interpreters packages

Each interpreter has the following attributes:

  • libPrefix. Name of the folder in ${python}/lib/ for corresponding interpreter.

  • interpreter. Alias for ${python}/bin/${executable}.

  • buildEnv. Function to build python interpreter environments with extra packages bundled together. See section python.buildEnv function for usage and documentation.

  • withPackages. Simpler interface to buildEnv. See section python.withPackages function for usage and documentation.

  • sitePackages. Alias for lib/${libPrefix}/site-packages.

  • executable. Name of the interpreter executable, e.g. python3.4.

  • pkgs. Set of Python packages for that specific interpreter. The package set can be modified by overriding the interpreter and passing packageOverrides.

9.11.2.2. Building packages and applications

Python libraries and applications that use setuptools or distutils are typically build with respectively the buildPythonPackage and buildPythonApplication functions. These two functions also support installing a wheel.

All Python packages reside in pkgs/top-level/python-packages.nix and all applications elsewhere. In case a package is used as both a library and an application, then the package should be in pkgs/top-level/python-packages.nix since only those packages are made available for all interpreter versions. The preferred location for library expressions is in pkgs/development/python-modules. It is important that these packages are called from pkgs/top-level/python-packages.nix and not elsewhere, to guarantee the right version of the package is built.

Based on the packages defined in pkgs/top-level/python-packages.nix an attribute set is created for each available Python interpreter. The available sets are

  • pkgs.python26Packages

  • pkgs.python27Packages

  • pkgs.python34Packages

  • pkgs.python35Packages

  • pkgs.python36Packages

  • pkgs.pypyPackages

and the aliases

  • pkgs.python2Packages pointing to pkgs.python27Packages

  • pkgs.python3Packages pointing to pkgs.python35Packages

  • pkgs.pythonPackages pointing to pkgs.python2Packages

9.11.2.2.1. buildPythonPackage function

The buildPythonPackage function is implemented in pkgs/development/interpreters/python/build-python-package.nix

The following is an example:

{ # ...

  twisted = buildPythonPackage {
    name = "twisted-8.1.0";

    src = pkgs.fetchurl {
      url = http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/Twisted-8.1.0.tar.bz2;
      sha256 = "0q25zbr4xzknaghha72mq57kh53qw1bf8csgp63pm9sfi72qhirl";
    };

    propagatedBuildInputs = [ self.ZopeInterface ];

    meta = {
      homepage = http://twistedmatrix.com/;
      description = "Twisted, an event-driven networking engine written in Python";
      license = stdenv.lib.licenses.mit;
    };
  };
}

The buildPythonPackage mainly does four things:

  • In the buildPhase, it calls ${python.interpreter} setup.py bdist_wheel to build a wheel binary zipfile.

  • In the installPhase, it installs the wheel file using pip install *.whl.

  • In the postFixup phase, the wrapPythonPrograms bash function is called to wrap all programs in the $out/bin/* directory to include $PATH environment variable and add dependent libraries to script’s sys.path.

  • In the installCheck phase, ${python.interpreter} setup.py test is ran.

As in Perl, dependencies on other Python packages can be specified in the buildInputs and propagatedBuildInputs attributes. If something is exclusively a build-time dependency, use buildInputs; if it’s (also) a runtime dependency, use propagatedBuildInputs.

By default tests are run because doCheck = true. Test dependencies, like e.g. the test runner, should be added to buildInputs.

By default meta.platforms is set to the same value as the interpreter unless overridden otherwise.

9.11.2.2.1.1. buildPythonPackage parameters

All parameters from mkDerivation function are still supported.

  • namePrefix: Prepended text to ${name} parameter. Defaults to "python3.3-" for Python 3.3, etc. Set it to "" if you’re packaging an application or a command line tool.

  • disabled: If true, package is not build for particular python interpreter version. Grep around pkgs/top-level/python-packages.nix for examples.

  • setupPyBuildFlags: List of flags passed to setup.py build_ext command.

  • pythonPath: List of packages to be added into $PYTHONPATH. Packages in pythonPath are not propagated (contrary to propagatedBuildInputs).

  • preShellHook: Hook to execute commands before shellHook.

  • postShellHook: Hook to execute commands after shellHook.

  • makeWrapperArgs: A list of strings. Arguments to be passed to makeWrapper, which wraps generated binaries. By default, the arguments to makeWrapper set PATH and PYTHONPATH environment variables before calling the binary. Additional arguments here can allow a developer to set environment variables which will be available when the binary is run. For example, makeWrapperArgs = ["--set FOO BAR" "--set BAZ QUX"].

  • installFlags: A list of strings. Arguments to be passed to pip install. To pass options to python setup.py install, use --install-option. E.g., `installFlags=[–install-option=–cpp_implementation].

  • format: Format of the source. Valid options are setuptools (default), flit, wheel, and other. setuptools is for when the source has a setup.py and setuptools is used to build a wheel, flit, in case flit should be used to build a wheel, and wheel in case a wheel is provided. In case you need to provide your own buildPhase and installPhase you can use other.

  • catchConflicts If true, abort package build if a package name appears more than once in dependency tree. Default is true.

  • checkInputs Dependencies needed for running the checkPhase. These are added to buildInputs when doCheck = true.

9.11.2.2.1.2. Overriding Python packages

The buildPythonPackage function has a overridePythonAttrs method that can be used to override the package. In the following example we create an environment where we have the blaze package using an older version of pandas. We override first the Python interpreter and pass packageOverrides which contains the overrides for packages in the package set.

with import <nixpkgs> {};

(let
  python = let
    packageOverrides = self: super: {
      pandas = super.pandas.overridePythonAttrs(old: rec {
        version = "0.19.1";
        name = "pandas-${version}";
        src =  super.fetchPypi {
          pname = "pandas";
          inherit version;
          sha256 = "08blshqj9zj1wyjhhw3kl2vas75vhhicvv72flvf1z3jvapgw295";
        };
      });
    };
  in pkgs.python3.override {inherit packageOverrides;};

in python.withPackages(ps: [ps.blaze])).env
9.11.2.2.2. buildPythonApplication function

The buildPythonApplication function is practically the same as buildPythonPackage. The difference is that buildPythonPackage by default prefixes the names of the packages with the version of the interpreter. Because with an application we’re not interested in multiple version the prefix is dropped.

9.11.2.2.3. python.buildEnv function

Python environments can be created using the low-level pkgs.buildEnv function. This example shows how to create an environment that has the Pyramid Web Framework. Saving the following as default.nix

with import <nixpkgs> {};

python.buildEnv.override {
  extraLibs = [ pkgs.pythonPackages.pyramid ];
  ignoreCollisions = true;
}

and running nix-build will create

/nix/store/cf1xhjwzmdki7fasgr4kz6di72ykicl5-python-2.7.8-env

with wrapped binaries in bin/.

You can also use the env attribute to create local environments with needed packages installed. This is somewhat comparable to virtualenv. For example, running nix-shell with the following shell.nix

with import <nixpkgs> {};

(python3.buildEnv.override {
  extraLibs = with python3Packages; [ numpy requests ];
}).env

will drop you into a shell where Python will have the specified packages in its path.

9.11.2.2.3.1. python.buildEnv arguments
  • extraLibs: List of packages installed inside the environment.

  • postBuild: Shell command executed after the build of environment.

  • ignoreCollisions: Ignore file collisions inside the environment (default is false).

9.11.2.2.4. python.withPackages function

The python.withPackages function provides a simpler interface to the python.buildEnv functionality. It takes a function as an argument that is passed the set of python packages and returns the list of the packages to be included in the environment. Using the withPackages function, the previous example for the Pyramid Web Framework environment can be written like this:

with import <nixpkgs> {};

python.withPackages (ps: [ps.pyramid])

withPackages passes the correct package set for the specific interpreter version as an argument to the function. In the above example, ps equals pythonPackages. But you can also easily switch to using python3:

with import <nixpkgs> {};

python3.withPackages (ps: [ps.pyramid])

Now, ps is set to python3Packages, matching the version of the interpreter.

As python.withPackages simply uses python.buildEnv under the hood, it also supports the env attribute. The shell.nix file from the previous section can thus be also written like this:

with import <nixpkgs> {};

(python36.withPackages (ps: [ps.numpy ps.requests])).env

In contrast to python.buildEnv, python.withPackages does not support the more advanced options such as ignoreCollisions = true or postBuild. If you need them, you have to use python.buildEnv.

Python 2 namespace packages may provide __init__.py that collide. In that case python.buildEnv should be used with ignoreCollisions = true.

9.11.2.3. Development mode

Development or editable mode is supported. To develop Python packages buildPythonPackage has additional logic inside shellPhase to run pip install -e . --prefix $TMPDIR/for the package.

Warning: shellPhase is executed only if setup.py exists.

Given a default.nix:

with import <nixpkgs> {};

buildPythonPackage { name = "myproject";

buildInputs = with pkgs.pythonPackages; [ pyramid ];

src = ./.; }

Running nix-shell with no arguments should give you the environment in which the package would be built with nix-build.

Shortcut to setup environments with C headers/libraries and python packages:

nix-shell -p pythonPackages.pyramid zlib libjpeg git

Note: There is a boolean value lib.inNixShell set to true if nix-shell is invoked.

9.11.2.4. Tools

Packages inside nixpkgs are written by hand. However many tools exist in community to help save time. No tool is preferred at the moment.

9.11.2.5. Deterministic builds

Python 2.7, 3.5 and 3.6 are now built deterministically and 3.4 mostly. Minor modifications had to be made to the interpreters in order to generate deterministic bytecode. This has security implications and is relevant for those using Python in a nix-shell.

When the environment variable DETERMINISTIC_BUILD is set, all bytecode will have timestamp 1. The buildPythonPackage function sets DETERMINISTIC_BUILD=1 and PYTHONHASHSEED=0. Both are also exported in nix-shell.

9.11.2.6. Automatic tests

It is recommended to test packages as part of the build process. Source distributions (sdist) often include test files, but not always.

By default the command python setup.py test is run as part of the checkPhase, but often it is necessary to pass a custom checkPhase. An example of such a situation is when py.test is used.

9.11.2.6.1. Common issues
  • Non-working tests can often be deselected. By default buildPythonPackage runs python setup.py test. Most python modules follows the standard test protocol where the pytest runner can be used instead. py.test supports a -k parameter to ignore test methods or classes:

nix buildPythonPackage { # ... # assumes the tests are located in tests checkInputs = [ pytest ]; checkPhase = '' py.test -k 'not function_name and not other_function' tests ''; } - Unicode issues can typically be fixed by including glibcLocales in buildInputs and exporting LC_ALL=en_US.utf-8. - Tests that attempt to access $HOME can be fixed by using the following work-around before running tests (e.g. preCheck): export HOME=$(mktemp -d)

9.11.3. FAQ

9.11.3.1. How to solve circular dependencies?

Consider the packages A and B that depend on each other. When packaging B, a solution is to override package A not to depend on B as an input. The same should also be done when packaging A.

9.11.3.2. How to override a Python package?

We can override the interpreter and pass packageOverrides. In the following example we rename the pandas package and build it.

with import <nixpkgs> {};

(let
  python = let
    packageOverrides = self: super: {
      pandas = super.pandas.overridePythonAttrs(old: {name="foo";});
    };
  in pkgs.python35.override {inherit packageOverrides;};

in python.withPackages(ps: [ps.pandas])).env

Using nix-build on this expression will build an environment that contains the package pandas but with the new name foo.

All packages in the package set will use the renamed package. A typical use case is to switch to another version of a certain package. For example, in the Nixpkgs repository we have multiple versions of django and scipy. In the following example we use a different version of scipy and create an environment that uses it. All packages in the Python package set will now use the updated scipy version.

with import <nixpkgs> {};

( let
    packageOverrides = self: super: {
      scipy = super.scipy_0_17;
    };
  in (pkgs.python35.override {inherit packageOverrides;}).withPackages (ps: [ps.blaze])
).env

The requested package blaze depends on pandas which itself depends on scipy.

If you want the whole of Nixpkgs to use your modifications, then you can use overlays as explained in this manual. In the following example we build a inkscape using a different version of numpy.

let
  pkgs = import <nixpkgs> {};
  newpkgs = import pkgs.path { overlays = [ (pkgsself: pkgssuper: {
    python27 = let
      packageOverrides = self: super: {
        numpy = super.numpy_1_10;
      };
    in pkgssuper.python27.override {inherit packageOverrides;};
  } ) ]; };
in newpkgs.inkscape

9.11.3.3. python setup.py bdist_wheel cannot create .whl

Executing python setup.py bdist_wheel in a nix-shellfails with

ValueError: ZIP does not support timestamps before 1980

This is because files are included that depend on items in the Nix store which have a timestamp of, that is, it corresponds to January the 1st, 1970 at 00:00:00. And as the error informs you, ZIP does not support that. The command bdist_wheel takes into account SOURCE_DATE_EPOCH, and nix-shell sets this to 1. By setting it to a value corresponding to 1980 or later, or by unsetting it, it is possible to build wheels.

Use 1980 as timestamp:

nix-shell --run "SOURCE_DATE_EPOCH=315532800 python3 setup.py bdist_wheel"

or the current time:

nix-shell --run "SOURCE_DATE_EPOCH=$(date +%s) python3 setup.py bdist_wheel"

or unset:

nix-shell --run "unset SOURCE_DATE_EPOCH; python3 setup.py bdist_wheel"

9.11.3.4. install_data / data_files problems

If you get the following error:

could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
Permission denied

This is a known bug in setuptools. Setuptools install_data does not respect --prefix. An example of such package using the feature is pkgs/tools/X11/xpra/default.nix. As workaround install it as an extra preInstall step:

${python.interpreter} setup.py install_data --install-dir=$out --root=$out
sed -i '/ = data\_files/d' setup.py

9.11.3.5. Rationale of non-existent global site-packages

On most operating systems a global site-packages is maintained. This however becomes problematic if you want to run multiple Python versions or have multiple versions of certain libraries for your projects. Generally, you would solve such issues by creating virtual environments using virtualenv.

On Nix each package has an isolated dependency tree which, in the case of Python, guarantees the right versions of the interpreter and libraries or packages are available. There is therefore no need to maintain a global site-packages.

If you want to create a Python environment for development, then the recommended method is to use nix-shell, either with or without the python.buildEnv function.

9.11.3.6. How to consume python modules using pip in a virtualenv like I am used to on other Operating Systems ?

This is an example of a default.nix for a nix-shell, which allows to consume a virtualenv environment, and install python modules through pip the traditional way.

Create this default.nix file, together with a requirements.txt and simply execute nix-shell.

with import <nixpkgs> {};
with pkgs.python27Packages;

stdenv.mkDerivation {
  name = "impurePythonEnv";
  buildInputs = [
    # these packages are required for virtualenv and pip to work:
    #
    python27Full
    python27Packages.virtualenv
    python27Packages.pip
    # the following packages are related to the dependencies of your python
    # project.
    # In this particular example the python modules listed in the
    # requirements.tx require the following packages to be installed locally
    # in order to compile any binary extensions they may require.
    #
    taglib
    openssl
    git
    libxml2
    libxslt
    libzip
    stdenv
    zlib ];
  src = null;
  shellHook = ''
  # set SOURCE_DATE_EPOCH so that we can use python wheels
  SOURCE_DATE_EPOCH=$(date +%s)
  virtualenv --no-setuptools venv
  export PATH=$PWD/venv/bin:$PATH
  pip install -r requirements.txt
  '';
}

Note that the pip install is an imperative action. So every time nix-shell is executed it will attempt to download the python modules listed in requirements.txt. However these will be cached locally within the virtualenv folder and not downloaded again.

9.11.3.7. How to override a Python package from configuration.nix?

If you need to change a package’s attribute(s) from configuration.nix you could do:

  nixpkgs.config.packageOverrides = superP: {
    pythonPackages = superP.pythonPackages.override {
      overrides = self: super: {
        bepasty-server = super.bepasty-server.overrideAttrs ( oldAttrs: {
          src = pkgs.fetchgit {
            url = "https://github.com/bepasty/bepasty-server";
            sha256 = "9ziqshmsf0rjvdhhca55sm0x8jz76fsf2q4rwh4m6lpcf8wr0nps";
            rev = "e2516e8cf4f2afb5185337073607eb9e84a61d2d";
          };
        });
      };
    };
  };

If you are using the bepasty-server package somewhere, for example in systemPackages or indirectly from services.bepasty, then a nixos-rebuild switch will rebuild the system but with the bepasty-server package using a different src attribute. This way one can modify python based software/libraries easily. Using self and super one can also alter dependencies (buildInputs) between the old state (self) and new state (super).

9.11.3.8. How to override a Python package using overlays?

To alter a python package using overlays, you would use the following approach:

self: super:
rec {
  python = super.python.override {
    packageOverrides = python-self: python-super: {
      bepasty-server = python-super.bepasty-server.overrideAttrs ( oldAttrs: {
        src = self.pkgs.fetchgit {
          url = "https://github.com/bepasty/bepasty-server";
          sha256 = "9ziqshmsf0rjvdhhca55sm0x8jz76fsf2q4rwh4m6lpcf8wr0nps";
          rev = "e2516e8cf4f2afb5185337073607eb9e84a61d2d";
        };
      });
    };
  };
  pythonPackages = python.pkgs;
}

9.11.4. Contributing

9.11.4.1. Contributing guidelines

Following rules are desired to be respected:

  • Python libraries are called from python-packages.nix and packaged with buildPythonPackage. The expression of a library should be in pkgs/development/python-modules/<name>/default.nix. Libraries in pkgs/top-level/python-packages.nix are sorted quasi-alphabetically to avoid merge conflicts.

  • Python applications live outside of python-packages.nix and are packaged with buildPythonApplication.

  • Make sure libraries build for all Python interpreters.

  • By default we enable tests. Make sure the tests are found and, in the case of libraries, are passing for all interpreters. If certain tests fail they can be disabled individually. Try to avoid disabling the tests altogether. In any case, when you disable tests, leave a comment explaining why.

  • Commit names of Python libraries should reflect that they are Python libraries, so write for example pythonPackages.numpy: 1.11 -> 1.12.

9.12. Qt

Qt is a comprehensive desktop and mobile application development toolkit for C++. Legacy support is available for Qt 3 and Qt 4, but all current development uses Qt 5. The Qt 5 packages in Nixpkgs are updated frequently to take advantage of new features, but older versions are typically retained until their support window ends. The most important consideration in packaging Qt-based software is ensuring that each package and all its dependencies use the same version of Qt 5; this consideration motivates most of the tools described below.

9.12.1. Packaging Libraries for Nixpkgs

Whenever possible, libraries that use Qt 5 should be built with each available version. Packages providing libraries should be added to the top-level function mkLibsForQt5, which is used to build a set of libraries for every Qt 5 version. A special callPackage function is used in this scope to ensure that the entire dependency tree uses the same Qt 5 version. Import dependencies unqualified, i.e., qtbase not qt5.qtbase. Do not import a package set such as qt5 or libsForQt5.

If a library does not support a particular version of Qt 5, it is best to mark it as broken by setting its meta.broken attribute. A package may be marked broken for certain versions by testing the qtbase.version attribute, which will always give the current Qt 5 version.

9.12.2. Packaging Applications for Nixpkgs

Call your application expression using libsForQt5.callPackage instead of callPackage. Import dependencies unqualified, i.e., qtbase not qt5.qtbase. Do not import a package set such as qt5 or libsForQt5.

Qt 5 maintains strict backward compatibility, so it is generally best to build an application package against the latest version using the libsForQt5 library set. In case a package does not build with the latest Qt version, it is possible to pick a set pinned to a particular version, e.g. libsForQt55 for Qt 5.5, if that is the latest version the package supports. If a package must be pinned to an older Qt version, be sure to file a bug upstream; because Qt is strictly backwards-compatible, any incompatibility is by definition a bug in the application.

When testing applications in Nixpkgs, it is a common practice to build the package with nix-build and run it using the created symbolic link. This will not work with Qt applications, however, because they have many hard runtime requirements that can only be guaranteed if the package is actually installed. To test a Qt application, install it with nix-env or run it inside nix-shell.

9.13. R packages

9.13.1. Installation

Define an environment for R that contains all the libraries that you’d like to use by adding the following snippet to your $HOME/.config/nixpkgs/config.nix file:

{
    packageOverrides = super: let self = super.pkgs; in
    {

        rEnv = super.rWrapper.override {
            packages = with self.rPackages; [
                devtools
                ggplot2
                reshape2
                yaml
                optparse
                ];
        };
    };
}

Then you can use nix-env -f "<nixpkgs>" -iA rEnv to install it into your user profile. The set of available libraries can be discovered by running the command nix-env -f "<nixpkgs>" -qaP -A rPackages. The first column from that output is the name that has to be passed to rWrapper in the code snipped above.

However, if you’d like to add a file to your project source to make the environment available for other contributors, you can create a default.nix file like so:

let
  pkgs = import <nixpkgs> {};
  stdenv = pkgs.stdenv;
in with pkgs; {
  myProject = stdenv.mkDerivation {
    name = "myProject";
    version = "1";
    src = if pkgs.lib.inNixShell then null else nix;

    buildInputs = with rPackages; [
      R
      ggplot2
      knitr
    ];
  };
}

and then run nix-shell . to be dropped into a shell with those packages available.

9.13.2. RStudio

RStudio uses a standard set of packages and ignores any custom R environments or installed packages you may have. To create a custom environment, see rstudioWrapper, which functions similarly to rWrapper:

{
    packageOverrides = super: let self = super.pkgs; in
    {

        rstudioEnv = super.rstudioWrapper.override {
            packages = with self.rPackages; [
                dplyr
                ggplot2
                reshape2
                ];
        };
    };
}

Then like above, nix-env -f "<nixpkgs>" -iA rstudioEnv will install this into your user profile.

9.13.3. Updating the package set

nix-shell generate-shell.nix

Rscript generate-r-packages.R cran  > cran-packages.nix.new
mv cran-packages.nix.new cran-packages.nix

Rscript generate-r-packages.R bioc  > bioc-packages.nix.new
mv bioc-packages.nix.new bioc-packages.nix

generate-r-packages.R <repo> reads <repo>-packages.nix, therefor the renaming.

9.13.4. Testing if the Nix-expression could be evaluated

nix-build test-evaluation.nix --dry-run

If this exits fine, the expression is ok. If not, you have to edit default.nix

9.14. Ruby

There currently is support to bundle applications that are packaged as Ruby gems. The utility "bundix" allows you to write a Gemfile, let bundler create a Gemfile.lock, and then convert this into a nix expression that contains all Gem dependencies automatically.

For example, to package sensu, we did:

$ cd pkgs/servers/monitoring
$ mkdir sensu
$ cd sensu
$ cat > Gemfile
source 'https://rubygems.org'
gem 'sensu'
$ $(nix-build '<nixpkgs>' -A bundix --no-out-link)/bin/bundix --magic
$ cat > default.nix
{ lib, bundlerEnv, ruby }:

bundlerEnv rec {
  name = "sensu-${version}";

  version = (import gemset).sensu.version;
  inherit ruby;
  # expects Gemfile, Gemfile.lock and gemset.nix in the same directory
  gemdir = ./.;

  meta = with lib; {
    description = "A monitoring framework that aims to be simple, malleable, and scalable";
    homepage    = http://sensuapp.org/;
    license     = with licenses; mit;
    maintainers = with maintainers; [ theuni ];
    platforms   = platforms.unix;
  };
}

Please check in the Gemfile, Gemfile.lock and the gemset.nix so future updates can be run easily.

For tools written in Ruby - i.e. where the desire is to install a package and then execute e.g. rake at the command line, there is an alternative builder called bundlerApp. Set up the gemset.nix the same way, and then, for example:

{ lib, bundlerApp }:

bundlerApp {
  pname = "corundum";
  gemdir = ./.;
  exes = [ "corundum-skel" ];

  meta = with lib; {
    description = "Tool and libraries for maintaining Ruby gems.";
    homepage    = https://github.com/nyarly/corundum;
    license     = licenses.mit;
    maintainers = [ maintainers.nyarly ];
    platforms   = platforms.unix;
  };
}

The chief advantage of bundlerApp over bundlerEnv is the executables introduced in the environment are precisely those selected in the exes list, as opposed to bundlerEnv which adds all the executables made available by gems in the gemset, which can mean e.g. rspec or rake in unpredictable versions available from various packages.

Resulting derivations for both builders also have two helpful attributes, env and wrappedRuby. The first one allows one to quickly drop into nix-shell with the specified environment present. E.g. nix-shell -A sensu.env would give you an environment with Ruby preset so it has all the libraries necessary for sensu in its paths. The second one can be used to make derivations from custom Ruby scripts which have Gemfiles with their dependencies specified. It is a derivation with ruby wrapped so it can find all the needed dependencies. For example, to make a derivation my-script for a my-script.rb (which should be placed in bin) you should run bundix as specified above and then use bundlerEnv like this:

let env = bundlerEnv {
  name = "my-script-env";

  inherit ruby;
  gemfile = ./Gemfile;
  lockfile = ./Gemfile.lock;
  gemset = ./gemset.nix;
};

in stdenv.mkDerivation {
  name = "my-script";
  buildInputs = [ env.wrappedRuby ];
  script = ./my-script.rb;
  buildCommand = ''
    install -D -m755 $script $out/bin/my-script
    patchShebangs $out/bin/my-script
  '';
}

9.15. User’s Guide to the Rust Infrastructure

To install the rust compiler and cargo put

rustStable.rustc
rustStable.cargo

into the environment.systemPackages or bring them into scope with nix-shell -p rustStable.rustc -p rustStable.cargo.

There are also rustBeta and rustNightly package sets available. These are not updated very regularly. For daily builds use either rustup from nixpkgs or use the Rust nightlies overlay.

9.15.1. Packaging Rust applications

Rust applications are packaged by using the buildRustPackage helper from rustPlatform:

with rustPlatform;

buildRustPackage rec {
  name = "ripgrep-${version}";
  version = "0.4.0";

  src = fetchFromGitHub {
    owner = "BurntSushi";
    repo = "ripgrep";
    rev = "${version}";
    sha256 = "0y5d1n6hkw85jb3rblcxqas2fp82h3nghssa4xqrhqnz25l799pj";
  };

  depsSha256 = "0q68qyl2h6i0qsz82z840myxlnjay8p1w5z7hfyr8fqp7wgwa9cx";

  meta = with stdenv.lib; {
    description = "A utility that combines the usability of The Silver Searcher with the raw speed of grep";
    homepage = https://github.com/BurntSushi/ripgrep;
    license = with licenses; [ unlicense ];
    maintainers = [ maintainers.tailhook ];
    platforms = platforms.all;
  };
}

buildRustPackage requires a depsSha256 attribute which is computed over all crate sources of this package. Currently it is obtained by inserting a fake checksum into the expression and building the package once. The correct checksum can be then take from the failed build.

To install crates with nix there is also an experimental project called nixcrates.

9.15.2. Using the Rust nightlies overlay

Mozilla provides an overlay for nixpkgs to bring a nightly version of Rust into scope. This overlay can also be used to install recent unstable or stable versions of Rust, if desired.

To use this overlay, clone nixpkgs-mozilla, and create a symbolic link to the file rust-overlay.nix in the ~/.config/nixpkgs/overlays directory.

$ git clone https://github.com/mozilla/nixpkgs-mozilla.git
$ mkdir -p ~/.config/nixpkgs/overlays
$ ln -s $(pwd)/nixpkgs-mozilla/rust-overlay.nix ~/.config/nixpkgs/overlays/rust-overlay.nix

The latest version can be installed with the following command:

$ nix-env -Ai nixos.rustChannels.stable.rust

Or using the attribute with nix-shell:

$ nix-shell -p nixos.rustChannels.stable.rust

To install the beta or nightly channel, stable should be substituted by nightly or beta, or use the function provided by this overlay to pull a version based on a build date.

The overlay automatically updates itself as it uses the same source as rustup.

9.16. TeX Live

Since release 15.09 there is a new TeX Live packaging that lives entirely under attribute texlive.

9.16.1. User's guide

  • For basic usage just pull texlive.combined.scheme-basic for an environment with basic LaTeX support.

  • It typically won't work to use separately installed packages together. Instead, you can build a custom set of packages like this:

    texlive.combine {
      inherit (texlive) scheme-small collection-langkorean algorithms cm-super;
    }
          

    There are all the schemes, collections and a few thousand packages, as defined upstream (perhaps with tiny differences).

  • By default you only get executables and files needed during runtime, and a little documentation for the core packages. To change that, you need to add pkgFilter function to combine.

    texlive.combine {
      # inherit (texlive) whatever-you-want;
      pkgFilter = pkg:
        pkg.tlType == "run" || pkg.tlType == "bin" || pkg.pname == "cm-super";
      # elem tlType [ "run" "bin" "doc" "source" ]
      # there are also other attributes: version, name
    }
          

  • You can list packages e.g. by nix-repl.

    $ nix-repl
    nix-repl> :l <nixpkgs>
    nix-repl> texlive.collection-<TAB>
          

9.16.2. Known problems

  • Some tools are still missing, e.g. luajittex;

  • some apps aren't packaged/tested yet (asymptote, biber, etc.);

  • feature/bug: when a package is rejected by pkgFilter, its dependencies are still propagated;

  • in case of any bugs or feature requests, file a github issue or better a pull request and /cc @vcunat.

9.17. User’s Guide to Vim Plugins/Addons/Bundles/Scripts in Nixpkgs

You’ll get a vim(-your-suffix) in PATH also loading the plugins you want. Loading can be deferred; see examples.

Vim packages, VAM (=vim-addon-manager) and Pathogen are supported to load packages.

9.17.1. Custom configuration

Adding custom .vimrc lines can be done using the following code:

vim_configurable.customize {
  name = "vim-with-plugins";

  vimrcConfig.customRC = ''
    set hidden
  '';
}

9.17.2. Vim packages

To store you plugins in Vim packages the following example can be used:

vim_configurable.customize {
  vimrcConfig.packages.myVimPackage = with pkgs.vimPlugins; {
    # loaded on launch
    start = [ youcompleteme fugitive ];
    # manually loadable by calling `:packadd $plugin-name`
    opt = [ phpCompletion elm-vim ];
    # To automatically load a plugin when opening a filetype, add vimrc lines like:
    # autocmd FileType php :packadd phpCompletion
  }
};

9.17.3. VAM

9.17.3.1. dependencies by Vim plugins

VAM introduced .json files supporting dependencies without versioning assuming that using latest version is ok most of the time.

9.17.3.2. Example

First create a vim-scripts file having one plugin name per line. Example:

"tlib"
{'name': 'vim-addon-sql'}
{'filetype_regex': '\%(vim)$', 'names': ['reload', 'vim-dev-plugin']}

Such vim-scripts file can be read by VAM as well like this:

call vam#Scripts(expand('~/.vim-scripts'), {})

Create a default.nix file:

{ nixpkgs ? import <nixpkgs> {}, compiler ? "ghc7102" }:
nixpkgs.vim_configurable.customize { name = "vim"; vimrcConfig.vam.pluginDictionaries = [ "vim-addon-vim2nix" ]; }

Create a generate.vim file:

ActivateAddons vim-addon-vim2nix
let vim_scripts = "vim-scripts"
call nix#ExportPluginsForNix({
\  'path_to_nixpkgs': eval('{"'.substitute(substitute(substitute($NIX_PATH, ':', ',', 'g'), '=',':', 'g'), '\([:,]\)', '"\1"',"g").'"}')["nixpkgs"],
\  'cache_file': '/tmp/vim2nix-cache',
\  'try_catch': 0,
\  'plugin_dictionaries': ["vim-addon-manager"]+map(readfile(vim_scripts), 'eval(v:val)')
\ })

Then run

nix-shell -p vimUtils.vim_with_vim2nix --command "vim -c 'source generate.vim'"

You should get a Vim buffer with the nix derivations (output1) and vam.pluginDictionaries (output2). You can add your vim to your system’s configuration file like this and start it by vim-my:

my-vim =
 let plugins = let inherit (vimUtils) buildVimPluginFrom2Nix; in {
      copy paste output1 here
 }; in vim_configurable.customize {
   name = "vim-my";

   vimrcConfig.vam.knownPlugins = plugins; # optional
   vimrcConfig.vam.pluginDictionaries = [
      copy paste output2 here
   ];

   # Pathogen would be
   # vimrcConfig.pathogen.knownPlugins = plugins; # plugins
   # vimrcConfig.pathogen.pluginNames = ["tlib"];
 };

Sample output1:

"reload" = buildVimPluginFrom2Nix { # created by nix#NixDerivation
  name = "reload";
  src = fetchgit {
    url = "git://github.com/xolox/vim-reload";
    rev = "0a601a668727f5b675cb1ddc19f6861f3f7ab9e1";
    sha256 = "0vb832l9yxj919f5hfg6qj6bn9ni57gnjd3bj7zpq7d4iv2s4wdh";
  };
  dependencies = ["nim-misc"];

};
[...]

Sample output2:

[
  ''vim-addon-manager''
  ''tlib''
  { "name" = ''vim-addon-sql''; }
  { "filetype_regex" = ''\%(vim)$$''; "names" = [ ''reload'' ''vim-dev-plugin'' ]; }
]

9.17.4. Important repositories

  • vim-pi is a plugin repository from VAM plugin manager meant to be used by others as well used by

  • vim2nix which generates the .nix code

This chapter contains information about how to use and maintain the Nix expressions for a number of specific packages, such as the Linux kernel or X.org.

10.1. Linux kernel

The Nix expressions to build the Linux kernel are in pkgs/os-specific/linux/kernel.

The function that builds the kernel has an argument kernelPatches which should be a list of {name, patch, extraConfig} attribute sets, where name is the name of the patch (which is included in the kernel’s meta.description attribute), patch is the patch itself (possibly compressed), and extraConfig (optional) is a string specifying extra options to be concatenated to the kernel configuration file (.config).

The kernel derivation exports an attribute features specifying whether optional functionality is or isn’t enabled. This is used in NixOS to implement kernel-specific behaviour. For instance, if the kernel has the iwlwifi feature (i.e. has built-in support for Intel wireless chipsets), then NixOS doesn’t have to build the external iwlwifi package:

modulesTree = [kernel]
  ++ pkgs.lib.optional (!kernel.features ? iwlwifi) kernelPackages.iwlwifi
  ++ ...;

How to add a new (major) version of the Linux kernel to Nixpkgs:

  1. Copy the old Nix expression (e.g. linux-2.6.21.nix) to the new one (e.g. linux-2.6.22.nix) and update it.

  2. Add the new kernel to all-packages.nix (e.g., create an attribute kernel_2_6_22).

  3. Now we’re going to update the kernel configuration. First unpack the kernel. Then for each supported platform (i686, x86_64, uml) do the following:

    1. Make an copy from the old config (e.g. config-2.6.21-i686-smp) to the new one (e.g. config-2.6.22-i686-smp).

    2. Copy the config file for this platform (e.g. config-2.6.22-i686-smp) to .config in the kernel source tree.

    3. Run make oldconfig ARCH={i386,x86_64,um} and answer all questions. (For the uml configuration, also add SHELL=bash.) Make sure to keep the configuration consistent between platforms (i.e. don’t enable some feature on i686 and disable it on x86_64).

    4. If needed you can also run make menuconfig:

      $ nix-env -i ncurses
      $ export NIX_CFLAGS_LINK=-lncurses
      $ make menuconfig ARCH=arch

    5. Copy .config over the new config file (e.g. config-2.6.22-i686-smp).

  4. Test building the kernel: nix-build -A kernel_2_6_22. If it compiles, ship it! For extra credit, try booting NixOS with it.

  5. It may be that the new kernel requires updating the external kernel modules and kernel-dependent packages listed in the linuxPackagesFor function in all-packages.nix (such as the NVIDIA drivers, AUFS, etc.). If the updated packages aren’t backwards compatible with older kernels, you may need to keep the older versions around.

10.2. X.org

The Nix expressions for the X.org packages reside in pkgs/servers/x11/xorg/default.nix. This file is automatically generated from lists of tarballs in an X.org release. As such it should not be modified directly; rather, you should modify the lists, the generator script or the file pkgs/servers/x11/xorg/overrides.nix, in which you can override or add to the derivations produced by the generator.

The generator is invoked as follows:

$ cd pkgs/servers/x11/xorg
$ cat tarballs-7.5.list extra.list old.list \
  | perl ./generate-expr-from-tarballs.pl

For each of the tarballs in the .list files, the script downloads it, unpacks it, and searches its configure.ac and *.pc.in files for dependencies. This information is used to generate default.nix. The generator caches downloaded tarballs between runs. Pay close attention to the NOT FOUND: name messages at the end of the run, since they may indicate missing dependencies. (Some might be optional dependencies, however.)

A file like tarballs-7.5.list contains all tarballs in a X.org release. It can be generated like this:

$ export i="mirror://xorg/X11R7.4/src/everything/"
$ cat $(PRINT_PATH=1 nix-prefetch-url $i | tail -n 1) \
  | perl -e 'while (<>) { if (/(href|HREF)="([^"]*.bz2)"/) { print "$ENV{'i'}$2\n"; }; }' \
  | sort > tarballs-7.4.list

extra.list contains libraries that aren’t part of X.org proper, but are closely related to it, such as libxcb. old.list contains some packages that were removed from X.org, but are still needed by some people or by other packages (such as imake).

If the expression for a package requires derivation attributes that the generator cannot figure out automatically (say, patches or a postInstall hook), you should modify pkgs/servers/x11/xorg/overrides.nix.

10.3. Eclipse

The Nix expressions related to the Eclipse platform and IDE are in pkgs/applications/editors/eclipse.

Nixpkgs provides a number of packages that will install Eclipse in its various forms, these range from the bare-bones Eclipse Platform to the more fully featured Eclipse SDK or Scala-IDE packages and multiple version are often available. It is possible to list available Eclipse packages by issuing the command:

$ nix-env -f '<nixpkgs>' -qaP -A eclipses --description

Once an Eclipse variant is installed it can be run using the eclipse command, as expected. From within Eclipse it is then possible to install plugins in the usual manner by either manually specifying an Eclipse update site or by installing the Marketplace Client plugin and using it to discover and install other plugins. This installation method provides an Eclipse installation that closely resemble a manually installed Eclipse.

If you prefer to install plugins in a more declarative manner then Nixpkgs also offer a number of Eclipse plugins that can be installed in an Eclipse environment. This type of environment is created using the function eclipseWithPlugins found inside the nixpkgs.eclipses attribute set. This function takes as argument { eclipse, plugins ? [], jvmArgs ? [] } where eclipse is a one of the Eclipse packages described above, plugins is a list of plugin derivations, and jvmArgs is a list of arguments given to the JVM running the Eclipse. For example, say you wish to install the latest Eclipse Platform with the popular Eclipse Color Theme plugin and also allow Eclipse to use more RAM. You could then add

packageOverrides = pkgs: {
  myEclipse = with pkgs.eclipses; eclipseWithPlugins {
    eclipse = eclipse-platform;
    jvmArgs = [ "-Xmx2048m" ];
    plugins = [ plugins.color-theme ];
  };
}

to your Nixpkgs configuration (~/.config/nixpkgs/config.nix) and install it by running nix-env -f '<nixpkgs>' -iA myEclipse and afterward run Eclipse as usual. It is possible to find out which plugins are available for installation using eclipseWithPlugins by running

$ nix-env -f '<nixpkgs>' -qaP -A eclipses.plugins --description

If there is a need to install plugins that are not available in Nixpkgs then it may be possible to define these plugins outside Nixpkgs using the buildEclipseUpdateSite and buildEclipsePlugin functions found in the nixpkgs.eclipses.plugins attribute set. Use the buildEclipseUpdateSite function to install a plugin distributed as an Eclipse update site. This function takes { name, src } as argument where src indicates the Eclipse update site archive. All Eclipse features and plugins within the downloaded update site will be installed. When an update site archive is not available then the buildEclipsePlugin function can be used to install a plugin that consists of a pair of feature and plugin JARs. This function takes an argument { name, srcFeature, srcPlugin } where srcFeature and srcPlugin are the feature and plugin JARs, respectively.

Expanding the previous example with two plugins using the above functions we have

packageOverrides = pkgs: {
  myEclipse = with pkgs.eclipses; eclipseWithPlugins {
    eclipse = eclipse-platform;
    jvmArgs = [ "-Xmx2048m" ];
    plugins = [
      plugins.color-theme
      (plugins.buildEclipsePlugin {
        name = "myplugin1-1.0";
        srcFeature = fetchurl {
          url = "http://…/features/myplugin1.jar";
          sha256 = "123…";
        };
        srcPlugin = fetchurl {
          url = "http://…/plugins/myplugin1.jar";
          sha256 = "123…";
        };
      });
      (plugins.buildEclipseUpdateSite {
        name = "myplugin2-1.0";
        src = fetchurl {
          stripRoot = false;
          url = "http://…/myplugin2.zip";
          sha256 = "123…";
        };
      });
    ];
  };
}

10.4. Elm

The Nix expressions for Elm reside in pkgs/development/compilers/elm. They are generated automatically by update-elm.rb script. One should specify versions of Elm packages inside the script, clear the packages directory and run the script from inside it. elm-reactor is special because it also has Elm package dependencies. The process is not automated very much for now -- you should get the elm-reactor source tree (e.g. with nix-shell) and run elm2nix.rb inside it. Place the resulting package.nix file into packages/elm-reactor-elm.nix.

10.5. Interactive shell helpers

Some packages provide the shell integration to be more useful. But unlike other systems, nix doesn't have a standard share directory location. This is why a bunch PACKAGE-share scripts are shipped that print the location of the corresponding shared folder. Current list of such packages is as following:

  • autojump: autojump-share

  • fzf: fzf-share

E.g. autojump can then used in the .bashrc like this:

  source "$(autojump-share)/autojump.bash"

10.6. Steam

10.6.1. Steam in Nix

Steam is distributed as a .deb file, for now only as an i686 package (the amd64 package only has documentation). When unpacked, it has a script called steam that in ubuntu (their target distro) would go to /usr/bin . When run for the first time, this script copies some files to the user's home, which include another script that is the ultimate responsible for launching the steam binary, which is also in $HOME.

Nix problems and constraints:

  • We don't have /bin/bash and many scripts point there. Similarly for /usr/bin/python .

  • We don't have the dynamic loader in /lib .

  • The steam.sh script in $HOME can not be patched, as it is checked and rewritten by steam.

  • The steam binary cannot be patched, it's also checked.

The current approach to deploy Steam in NixOS is composing a FHS-compatible chroot environment, as documented here. This allows us to have binaries in the expected paths without disrupting the system, and to avoid patching them to work in a non FHS environment.

10.6.2. How to play

For 64-bit systems it's important to have

hardware.opengl.driSupport32Bit = true;

in your /etc/nixos/configuration.nix. You'll also need

hardware.pulseaudio.support32Bit = true;

if you are using PulseAudio - this will enable 32bit ALSA apps integration. To use the Steam controller, you need to add

services.udev.extraRules = ''
    SUBSYSTEM=="usb", ATTRS{idVendor}=="28de", MODE="0666"
    KERNEL=="uinput", MODE="0660", GROUP="users", OPTIONS+="static_node=uinput"
  '';

to your configuration.

10.6.3. Troubleshooting

Steam fails to start. What do I do?

Try to run

strace steam

to see what is causing steam to fail.

Using the FOSS Radeon or nouveau (nvidia) drivers
  • The newStdcpp parameter was removed since NixOS 17.09 and should not be needed anymore.

  • Steam ships statically linked with a version of libcrypto that conflics with the one dynamically loaded by radeonsi_dri.so. If you get the error

    steam.sh: line 713: 7842 Segmentation fault (core dumped)

    have a look at this pull request.

Java
  1. There is no java in steam chrootenv by default. If you get a message like

    /home/foo/.local/share/Steam/SteamApps/common/towns/towns.sh: line 1: java: command not found

    You need to add

     steam.override { withJava = true; };

    to your configuration.

10.6.4. steam-run

The FHS-compatible chroot used for steam can also be used to run other linux games that expect a FHS environment. To do it, add

pkgs.(steam.override {
          nativeOnly = true;
          newStdcpp = true;
        }).run

to your configuration, rebuild, and run the game with

steam-run ./foo

10.7. Emacs

10.7.1. Configuring Emacs

The Emacs package comes with some extra helpers to make it easier to configure. emacsWithPackages allows you to manage packages from ELPA. This means that you will not have to install that packages from within Emacs. For instance, if you wanted to use company, counsel, flycheck, ivy, magit, projectile, and use-package you could use this as a ~/.config/nixpkgs/config.nix override:

{
  packageOverrides = pkgs: with pkgs; {
    myEmacs = emacsWithPackages (epkgs: (with epkgs.melpaStablePackages; [
      company
      counsel
      flycheck
      ivy
      magit
      projectile
      use-package
    ]));
  }
}

You can install it like any other packages via nix-env -iA myEmacs. However, this will only install those packages. It will not configure them for us. To do this, we need to provide a configuration file. Luckily, it is possible to do this from within Nix! By modifying the above example, we can make Emacs load a custom config file. The key is to create a package that provide a default.el file in /share/emacs/site-start/. Emacs knows to load this file automatically when it starts.

{
  packageOverrides = pkgs: with pkgs; rec {
    myEmacsConfig = writeText "default.el" ''
;; initialize package

(require 'package)
(package-initialize 'noactivate)
(eval-when-compile
  (require 'use-package))

;; load some packages

(use-package company
  :bind ("<C-tab>" . company-complete)
  :diminish company-mode
  :commands (company-mode global-company-mode)
  :defer 1
  :config
  (global-company-mode))

(use-package counsel
  :commands (counsel-descbinds)
  :bind (([remap execute-extended-command] . counsel-M-x)
         ("C-x C-f" . counsel-find-file)
         ("C-c g" . counsel-git)
         ("C-c j" . counsel-git-grep)
         ("C-c k" . counsel-ag)
         ("C-x l" . counsel-locate)
         ("M-y" . counsel-yank-pop)))

(use-package flycheck
  :defer 2
  :config (global-flycheck-mode))

(use-package ivy
  :defer 1
  :bind (("C-c C-r" . ivy-resume)
         ("C-x C-b" . ivy-switch-buffer)
         :map ivy-minibuffer-map
         ("C-j" . ivy-call))
  :diminish ivy-mode
  :commands ivy-mode
  :config
  (ivy-mode 1))

(use-package magit
  :defer
  :if (executable-find "git")
  :bind (("C-x g" . magit-status)
         ("C-x G" . magit-dispatch-popup))
  :init
  (setq magit-completing-read-function 'ivy-completing-read))

(use-package projectile
  :commands projectile-mode
  :bind-keymap ("C-c p" . projectile-command-map)
  :defer 5
  :config
  (projectile-global-mode))
    '';
    myEmacs = emacsWithPackages (epkgs: (with epkgs.melpaStablePackages; [
      (runCommand "default.el" {} ''
mkdir -p $out/share/emacs/site-lisp
cp ${myEmacsConfig} $out/share/emacs/site-lisp/default.el
'')
      company
      counsel
      flycheck
      ivy
      magit
      projectile
      use-package
    ]));
  };
}

This provides a fairly full Emacs start file. It will load in addition to the user's presonal config. You can always disable it by passing -q to the Emacs command.

This chapter describes how to extend and change Nixpkgs packages using overlays. Overlays are used to add layers in the fix-point used by Nixpkgs to compose the set of all packages.

Nixpkgs can be configured with a list of overlays, which are applied in order. This means that the order of the overlays can be significant if multiple layers override the same package.

11.1. Installing overlays

The list of overlays is determined as follows.

If the overlays argument is not provided explicitly, we look for overlays in a path. The path is determined as follows:

  1. First, if an overlays argument to the nixpkgs function itself is given, then that is used.

    This can be passed explicitly when importing nipxkgs, for example import <nixpkgs> { overlays = [ overlay1 overlay2 ]; }.

  2. Otherwise, if the Nix path entry <nixpkgs-overlays> exists, we look for overlays at that path, as described below.

    See the section on NIX_PATH in the Nix manual for more details on how to set a value for <nixpkgs-overlays>.

  3. If one of ~/.config/nixpkgs/overlays.nix and ~/.config/nixpkgs/overlays/ exists, then we look for overlays at that path, as described below. It is an error if both exist.

If we are looking for overlays at a path, then there are two cases:

  • If the path is a file, then the file is imported as a Nix expression and used as the list of overlays.

  • If the path is a directory, then we take the content of the directory, order it lexicographically, and attempt to interpret each as an overlay by:

    • Importing the file, if it is a .nix file.

    • Importing a top-level default.nix file, if it is a directory.

On a NixOS system the value of the nixpkgs.overlays option, if present, is passed to the system Nixpkgs directly as an argument. Note that this does not affect the overlays for non-NixOS operations (e.g. nix-env), which are looked up independently.

The overlays.nix option therefore provides a convenient way to use the same overlays for a NixOS system configuration and user configuration: the same file can be used as overlays.nix and imported as the value of nixpkgs.overlays.

11.2. Defining overlays

Overlays are Nix functions which accept two arguments, conventionally called self and super, and return a set of packages. For example, the following is a valid overlay.

self: super:

{
  boost = super.boost.override {
    python = self.python3;
  };
  rr = super.callPackage ./pkgs/rr {
    stdenv = self.stdenv_32bit;
  };
}

The first argument (self) corresponds to the final package set. You should use this set for the dependencies of all packages specified in your overlay. For example, all the dependencies of rr in the example above come from self, as well as the overridden dependencies used in the boost override.

The second argument (super) corresponds to the result of the evaluation of the previous stages of Nixpkgs. It does not contain any of the packages added by the current overlay, nor any of the following overlays. This set should be used either to refer to packages you wish to override, or to access functions defined in Nixpkgs. For example, the original recipe of boost in the above example, comes from super, as well as the callPackage function.

The value returned by this function should be a set similar to pkgs/top-level/all-packages.nix, containing overridden and/or new packages.

Overlays are similar to other methods for customizing Nixpkgs, in particular the packageOverrides attribute described in Section 6.4, “Modify packages via packageOverrides. Indeed, packageOverrides acts as an overlay with only the super argument. It is therefore appropriate for basic use, but overlays are more powerful and easier to distribute.

12.1. Syntax

  • Use 2 spaces of indentation per indentation level in Nix expressions, 4 spaces in shell scripts.

  • Do not use tab characters, i.e. configure your editor to use soft tabs. For instance, use (setq-default indent-tabs-mode nil) in Emacs. Everybody has different tab settings so it’s asking for trouble.

  • Use lowerCamelCase for variable names, not UpperCamelCase. TODO: naming of attributes in all-packages.nix?

  • Function calls with attribute set arguments are written as

    foo {
      arg = ...;
    }
    

    not

    foo
    {
      arg = ...;
    }
    

    Also fine is

    foo { arg = ...; }
    

    if it's a short call.

  • In attribute sets or lists that span multiple lines, the attribute names or list elements should be aligned:

    # A long list.
    list =
      [ elem1
        elem2
        elem3
      ];
    
    # A long attribute set.
    attrs =
      { attr1 = short_expr;
        attr2 =
          if true then big_expr else big_expr;
      };
    
    # Alternatively:
    attrs = {
      attr1 = short_expr;
      attr2 =
        if true then big_expr else big_expr;
    };
    

  • Short lists or attribute sets can be written on one line:

    # A short list.
    list = [ elem1 elem2 elem3 ];
    
    # A short set.
    attrs = { x = 1280; y = 1024; };
    

  • Breaking in the middle of a function argument can give hard-to-read code, like

    someFunction { x = 1280;
      y = 1024; } otherArg
      yetAnotherArg
    

    (especially if the argument is very large, spanning multiple lines).

    Better:

    someFunction
      { x = 1280; y = 1024; }
      otherArg
      yetAnotherArg
    

    or

    let res = { x = 1280; y = 1024; };
    in someFunction res otherArg yetAnotherArg
    

  • The bodies of functions, asserts, and withs are not indented to prevent a lot of superfluous indentation levels, i.e.

    { arg1, arg2 }:
    assert system == "i686-linux";
    stdenv.mkDerivation { ...
    

    not

    { arg1, arg2 }:
      assert system == "i686-linux";
        stdenv.mkDerivation { ...
    

  • Function formal arguments are written as:

    { arg1, arg2, arg3 }:
    

    but if they don't fit on one line they're written as:

    { arg1, arg2, arg3
    , arg4, ...
    , # Some comment...
      argN
    }:
    

  • Functions should list their expected arguments as precisely as possible. That is, write

    { stdenv, fetchurl, perl }: ...
    

    instead of

    args: with args; ...
    

    or

    { stdenv, fetchurl, perl, ... }: ...
    

    For functions that are truly generic in the number of arguments (such as wrappers around mkDerivation) that have some required arguments, you should write them using an @-pattern:

    { stdenv, doCoverageAnalysis ? false, ... } @ args:
    
    stdenv.mkDerivation (args // {
      ... if doCoverageAnalysis then "bla" else "" ...
    })
    

    instead of

    args:
    
    args.stdenv.mkDerivation (args // {
      ... if args ? doCoverageAnalysis && args.doCoverageAnalysis then "bla" else "" ...
    })
    

12.2. Package naming

In Nixpkgs, there are generally three different names associated with a package:

  • The name attribute of the derivation (excluding the version part). This is what most users see, in particular when using nix-env.

  • The variable name used for the instantiated package in all-packages.nix, and when passing it as a dependency to other functions. This is what Nix expression authors see. It can also be used when installing using nix-env -iA.

  • The filename for (the directory containing) the Nix expression.

Most of the time, these are the same. For instance, the package e2fsprogs has a name attribute "e2fsprogs-version", is bound to the variable name e2fsprogs in all-packages.nix, and the Nix expression is in pkgs/os-specific/linux/e2fsprogs/default.nix.

There are a few naming guidelines:

  • Generally, try to stick to the upstream package name.

  • Don’t use uppercase letters in the name attribute — e.g., "mplayer-1.0rc2" instead of "MPlayer-1.0rc2".

  • The version part of the name attribute must start with a digit (following a dash) — e.g., "hello-0.3.1rc2".

  • If a package is not a release but a commit from a repository, then the version part of the name must be the date of that (fetched) commit. The date must be in "YYYY-MM-DD" format. Also append "unstable" to the name - e.g., "pkgname-unstable-2014-09-23".

  • Dashes in the package name should be preserved in new variable names, rather than converted to underscores (which was convention up to around 2013 and most names still have underscores instead of dashes) — e.g., http-parser instead of http_parser.

  • If there are multiple versions of a package, this should be reflected in the variable names in all-packages.nix, e.g. json-c-0-9 and json-c-0-11. If there is an obvious “default” version, make an attribute like json-c = json-c-0-9;. See also Section 12.3.2, “Versioning”

12.3. File naming and organisation

Names of files and directories should be in lowercase, with dashes between words — not in camel case. For instance, it should be all-packages.nix, not allPackages.nix or AllPackages.nix.

12.3.1. Hierarchy

Each package should be stored in its own directory somewhere in the pkgs/ tree, i.e. in pkgs/category/subcategory/.../pkgname. Below are some rules for picking the right category for a package. Many packages fall under several categories; what matters is the primary purpose of a package. For example, the libxml2 package builds both a library and some tools; but it’s a library foremost, so it goes under pkgs/development/libraries.

When in doubt, consider refactoring the pkgs/ tree, e.g. creating new categories or splitting up an existing category.

If it’s used to support software development:
If it’s a library used by other packages:

development/libraries (e.g. libxml2)

If it’s a compiler:

development/compilers (e.g. gcc)

If it’s an interpreter:

development/interpreters (e.g. guile)

If it’s a (set of) development tool(s):
If it’s a parser generator (including lexers):

development/tools/parsing (e.g. bison, flex)

If it’s a build manager:

development/tools/build-managers (e.g. gnumake)

Else:

development/tools/misc (e.g. binutils)

Else:

development/misc

If it’s a (set of) tool(s):

(A tool is a relatively small program, especially one intended to be used non-interactively.)

If it’s for networking:

tools/networking (e.g. wget)

If it’s for text processing:

tools/text (e.g. diffutils)

If it’s a system utility, i.e., something related or essential to the operation of a system:

tools/system (e.g. cron)

If it’s an archiver (which may include a compression function):

tools/archivers (e.g. zip, tar)

If it’s a compression program:

tools/compression (e.g. gzip, bzip2)

If it’s a security-related program:

tools/security (e.g. nmap, gnupg)

Else:

tools/misc

If it’s a shell:

shells (e.g. bash)

If it’s a server:
If it’s a web server:

servers/http (e.g. apache-httpd)

If it’s an implementation of the X Windowing System:

servers/x11 (e.g. xorg — this includes the client libraries and programs)

Else:

servers/misc

If it’s a desktop environment:

desktops (e.g. kde, gnome, enlightenment)

If it’s a window manager:

applications/window-managers (e.g. awesome, stumpwm)

If it’s an application:

A (typically large) program with a distinct user interface, primarily used interactively.

If it’s a version management system:

applications/version-management (e.g. subversion)

If it’s for video playback / editing:

applications/video (e.g. vlc)

If it’s for graphics viewing / editing:

applications/graphics (e.g. gimp)

If it’s for networking:
If it’s a mailreader:

applications/networking/mailreaders (e.g. thunderbird)

If it’s a newsreader:

applications/networking/newsreaders (e.g. pan)

If it’s a web browser:

applications/networking/browsers (e.g. firefox)

Else:

applications/networking/misc

Else:

applications/misc

If it’s data (i.e., does not have a straight-forward executable semantics):
If it’s a font:

data/fonts

If it’s related to SGML/XML processing:
If it’s an XML DTD:

data/sgml+xml/schemas/xml-dtd (e.g. docbook)

If it’s an XSLT stylesheet:

(Okay, these are executable...)

data/sgml+xml/stylesheets/xslt (e.g. docbook-xsl)

If it’s a game:

games

Else:

misc

12.3.2. Versioning

Because every version of a package in Nixpkgs creates a potential maintenance burden, old versions of a package should not be kept unless there is a good reason to do so. For instance, Nixpkgs contains several versions of GCC because other packages don’t build with the latest version of GCC. Other examples are having both the latest stable and latest pre-release version of a package, or to keep several major releases of an application that differ significantly in functionality.

If there is only one version of a package, its Nix expression should be named e2fsprogs/default.nix. If there are multiple versions, this should be reflected in the filename, e.g. e2fsprogs/1.41.8.nix and e2fsprogs/1.41.9.nix. The version in the filename should leave out unnecessary detail. For instance, if we keep the latest Firefox 2.0.x and 3.5.x versions in Nixpkgs, they should be named firefox/2.0.nix and firefox/3.5.nix, respectively (which, at a given point, might contain versions 2.0.0.20 and 3.5.4). If a version requires many auxiliary files, you can use a subdirectory for each version, e.g. firefox/2.0/default.nix and firefox/3.5/default.nix.

All versions of a package must be included in all-packages.nix to make sure that they evaluate correctly.

12.4. Fetching Sources

There are multiple ways to fetch a package source in nixpkgs. The general guideline is that you should package sources with a high degree of availability. Right now there is only one fetcher which has mirroring support and that is fetchurl. Note that you should also prefer protocols which have a corresponding proxy environment variable.

You can find many source fetch helpers in pkgs/build-support/fetch*.

In the file pkgs/top-level/all-packages.nix you can find fetch helpers, these have names on the form fetchFrom*. The intention of these are to provide snapshot fetches but using the same api as some of the version controlled fetchers from pkgs/build-support/. As an example going from bad to good:

  • Bad: Uses git:// which won't be proxied.

    src = fetchgit {
      url = "git://github.com/NixOS/nix.git";
      rev = "1f795f9f44607cc5bec70d1300150bfefcef2aae";
      sha256 = "1cw5fszffl5pkpa6s6wjnkiv6lm5k618s32sp60kvmvpy7a2v9kg";
    }
    

  • Better: This is ok, but an archive fetch will still be faster.

    src = fetchgit {
      url = "https://github.com/NixOS/nix.git";
      rev = "1f795f9f44607cc5bec70d1300150bfefcef2aae";
      sha256 = "1cw5fszffl5pkpa6s6wjnkiv6lm5k618s32sp60kvmvpy7a2v9kg";
    }
    

  • Best: Fetches a snapshot archive and you get the rev you want.

    src = fetchFromGitHub {
      owner = "NixOS";
      repo = "nix";
      rev = "1f795f9f44607cc5bec70d1300150bfefcef2aae";
      sha256 = "04yri911rj9j19qqqn6m82266fl05pz98inasni0vxr1cf1gdgv9";
    }
    

12.5. Patches

Only patches that are unique to nixpkgs should be included in nixpkgs source.

Patches available online should be retrieved using fetchpatch.

patches = [
  (fetchpatch {
    name = "fix-check-for-using-shared-freetype-lib.patch";
    url = "http://git.ghostscript.com/?p=ghostpdl.git;a=patch;h=8f5d285";
    sha256 = "1f0k043rng7f0rfl9hhb89qzvvksqmkrikmm38p61yfx51l325xr";
  })
];

13.1. Making patches

  • Read Manual (How to write packages for Nix).

  • Fork the repository on GitHub.

  • Create a branch for your future fix.

    • You can make branch from a commit of your local nixos-version. That will help you to avoid additional local compilations. Because you will receive packages from binary cache.

      • For example: nixos-version returns 15.05.git.0998212 (Dingo). So you can do:

      $ git checkout 0998212
      $ git checkout -b 'fix/pkg-name-update'
      

    • Please avoid working directly on the master branch.

  • Make commits of logical units.

    • If you removed pkgs, made some major NixOS changes etc., write about them in nixos/doc/manual/release-notes/rl-unstable.xml.

  • Check for unnecessary whitespace with git diff --check before committing.

  • Format the commit in a following way:

    (pkg-name | nixos/<module>): (from -> to | init at version | refactor | etc)
    Additional information.
    
    • Examples:

      • nginx: init at 2.0.1

      • firefox: 54.0.1 -> 55.0

      • nixos/hydra: add bazBaz option

      • nixos/nginx: refactor config generation

  • Test your changes. If you work with

    • nixpkgs:

      • update pkg ->

        • nix-env -i pkg-name -f <path to your local nixpkgs folder>

      • add pkg ->

        • Make sure it's in pkgs/top-level/all-packages.nix

        • nix-env -i pkg-name -f <path to your local nixpkgs folder>

      • If you don't want to install pkg in you profile.

        • nix-build -A pkg-attribute-name <path to your local nixpkgs folder>/default.nix and check results in the folder result. It will appear in the same directory where you did nix-build.

      • If you did nix-env -i pkg-name you can do nix-env -e pkg-name to uninstall it from your system.

    • NixOS and its modules:

      • You can add new module to your NixOS configuration file (usually it's /etc/nixos/configuration.nix). And do sudo nixos-rebuild test -I nixpkgs=<path to your local nixpkgs folder> --fast.

  • If you have commits pkg-name: oh, forgot to insert whitespace: squash commits in this case. Use git rebase -i.

  • Rebase you branch against current master.

13.2. Submitting changes

  • Push your changes to your fork of nixpkgs.

  • Create pull request:

    • Write the title in format (pkg-name | nixos/<module>): improvement.

      • If you update the pkg, write versions from -> to.

    • Write in comment if you have tested your patch. Do not rely much on TravisCI.

    • If you make an improvement, write about your motivation.

    • Notify maintainers of the package. For example add to the message: cc @jagajaga @domenkozar.

13.3. Pull Request Template

The pull request template helps determine what steps have been made for a contribution so far, and will help guide maintainers on the status of a change. The motivation section of the PR should include any extra details the title does not address and link any existing issues related to the pull request.

When a PR is created, it will be pre-populated with some checkboxes detailed below:

13.3.1. Tested using sandboxing

When sandbox builds are enabled, Nix will setup an isolated environment for each build process. It is used to remove further hidden dependencies set by the build environment to improve reproducibility. This includes access to the network during the build outside of fetch* functions and files outside the Nix store. Depending on the operating system access to other resources are blocked as well (ex. inter process communication is isolated on Linux); see build-use-sandbox in Nix manual for details.

Sandboxing is not enabled by default in Nix due to a small performance hit on each build. In pull requests for nixpkgs people are asked to test builds with sandboxing enabled (see Tested using sandboxing in the pull request template) because inhttps://nixos.org/hydra/ sandboxing is also used.

Depending if you use NixOS or other platforms you can use one of the following methods to enable sandboxing before building the package:

  • Globally enable sandboxing on NixOS: add the following to configuration.nix

    nix.useSandbox = true;

  • Globally enable sandboxing on non-NixOS platforms: add the following to: /etc/nix/nix.conf

    build-use-sandbox = true

13.3.2. Built on platform(s)

Many Nix packages are designed to run on multiple platforms. As such, it's important to let the maintainer know which platforms your changes have been tested on. It's not always practical to test a change on all platforms, and is not required for a pull request to be merged. Only check the systems you tested the build on in this section.

13.3.3. Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)

Packages with automated tests are much more likely to be merged in a timely fashion because it doesn't require as much manual testing by the maintainer to verify the functionality of the package. If there are existing tests for the package, they should be run to verify your changes do not break the tests. Tests only apply to packages with NixOS modules defined and can only be run on Linux. For more details on writing and running tests, see the section in the NixOS manual.

13.3.4. Tested compilation of all pkgs that depend on this change using nox-review

If you are updating a package's version, you can use nox to make sure all packages that depend on the updated package still compile correctly. This can be done using the nox utility. The nox-review utility can look for and build all dependencies either based on uncommited changes with the wip option or specifying a github pull request number.

review uncommitted changes:

nix-shell -p nox --run nox-review wip

review changes from pull request number 12345:

nix-shell -p nox --run nox-review pr 12345

13.3.5. Tested execution of all binary files (usually in ./result/bin/)

It's important to test any executables generated by a build when you change or create a package in nixpkgs. This can be done by looking in ./result/bin and running any files in there, or at a minimum, the main executable for the package. For example, if you make a change to texlive, you probably would only check the binaries associated with the change you made rather than testing all of them.

13.3.6. Meets nixpkgs contribution standards

The last checkbox is fits CONTRIBUTING.md. The contributing document has detailed information on standards the Nix community has for commit messages, reviews, licensing of contributions you make to the project, etc... Everyone should read and understand the standards the community has for contributing before submitting a pull request.

13.4. Hotfixing pull requests

  • Make the appropriate changes in you branch.

  • Don't create additional commits, do

    • git rebase -i

    • git push --force to your branch.

13.5. Commit policy

  • Commits must be sufficiently tested before being merged, both for the master and staging branches.

  • Hydra builds for master and staging should not be used as testing platform, it's a build farm for changes that have been already tested.

  • When changing the bootloader installation process, extra care must be taken. Grub installations cannot be rolled back, hence changes may break people's installations forever. For any non-trivial change to the bootloader please file a PR asking for review, especially from @edolstra.

13.5.1. Master branch

  • It should only see non-breaking commits that do not cause mass rebuilds.

13.5.2. Staging branch

  • It's only for non-breaking mass-rebuild commits. That means it's not to be used for testing, and changes must have been well tested already. Read policy here.

  • If the branch is already in a broken state, please refrain from adding extra new breakages. Stabilize it for a few days, merge into master, then resume development on staging. Keep an eye on the staging evaluations here. If any fixes for staging happen to be already in master, then master can be merged into staging.

13.5.3. Stable release branches

  • If you're cherry-picking a commit to a stable release branch, always use git cherry-pick -xe and ensure the message contains a clear description about why this needs to be included in the stable branch.

    An example of a cherry-picked commit would look like this:

    nixos: Refactor the world.
    
    The original commit message describing the reason why the world was torn apart.
    
    (cherry picked from commit abcdef)
    Reason: I just had a gut feeling that this would also be wanted by people from
    the stone age.
          
Warning: The following section is a draft and reviewing policy is still being discussed.

The nixpkgs projects receives a fairly high number of contributions via GitHub pull-requests. Reviewing and approving these is an important task and a way to contribute to the project.

The high change rate of nixpkgs make any pull request that is open for long enough subject to conflicts that will require extra work from the submitter or the merger. Reviewing pull requests in a timely manner and being responsive to the comments is the key to avoid these. GitHub provides sort filters that can be used to see the most recently and the least recently updated pull-requests.

When reviewing a pull request, please always be nice and polite. Controversial changes can lead to controversial opinions, but it is important to respect every community members and their work.

GitHub provides reactions, they are a simple and quick way to provide feedback to pull-requests or any comments. The thumb-down reaction should be used with care and if possible accompanied with some explanations so the submitter has directions to improve his contribution.

Pull-requests reviews should include a list of what has been reviewed in a comment, so other reviewers and mergers can know the state of the review.

All the review template samples provided in this section are generic and meant as examples. Their usage is optional and the reviewer is free to adapt them to his liking.

14.1. Package updates

A package update is the most trivial and common type of pull-request. These pull-requests mainly consist in updating the version part of the package name and the source hash.

It can happen that non trivial updates include patches or more complex changes.

Reviewing process:

  • Add labels to the pull-request. (Requires commit rights)

    • 8.has: package (update) and any topic label that fit the updated package.

  • Ensure that the package versioning is fitting the guidelines.

  • Ensure that the commit text is fitting the guidelines.

  • Ensure that the package maintainers are notified.

    • mention-bot usually notify GitHub users based on the submitted changes, but it can happen that it misses some of the package maintainers.

  • Ensure that the meta field contains correct information.

    • License can change with version updates, so it should be checked to be fitting upstream license.

    • If the package has no maintainer, a maintainer must be set. This can be the update submitter or a community member that accepts to take maintainership of the package.

  • Ensure that the code contains no typos.

  • Building the package locally.

    • Pull-requests are often targeted to the master or staging branch so building the pull-request locally as it is submitted can trigger a large amount of source builds.

      It is possible to rebase the changes on nixos-unstable or nixpkgs-unstable for easier review by running the following commands from a nixpkgs clone.

      $ git remote add channels https://github.com/NixOS/nixpkgs-channels.git 1
      $ git fetch channels nixos-unstable 2
      $ git fetch origin pull/PRNUMBER/head 3
      $ git rebase --onto nixos-unstable BASEBRANCH FETCH_HEAD 4
      

      1

      This should be done only once to be able to fetch channel branches from the nixpkgs-channels repository.

      2

      Fetching the nixos-unstable branch.

      3

      Fetching the pull-request changes, PRNUMBER is the number at the end of the pull-request title and BASEBRANCH the base branch of the pull-request.

      3

      Rebasing the pull-request changes to the nixos-unstable branch.

    • The nox tool can be used to review a pull-request content in a single command. It doesn't rebase on a channel branch so it might trigger multiple source builds. PRNUMBER should be replaced by the number at the end of the pull-request title.

      $ nix-shell -p nox --run "nox-review -k pr PRNUMBER"
      
  • Running every binary.

Example 14.1. Sample template for a package update review

##### Reviewed points

- [ ] package name fits guidelines
- [ ] package version fits guidelines
- [ ] package build on ARCHITECTURE
- [ ] executables tested on ARCHITECTURE
- [ ] all depending packages build

##### Possible improvements

##### Comments



14.2. New packages

New packages are a common type of pull-requests. These pull requests consists in adding a new nix-expression for a package.

Reviewing process:

  • Add labels to the pull-request. (Requires commit rights)

    • 8.has: package (new) and any topic label that fit the new package.

  • Ensure that the package versioning is fitting the guidelines.

  • Ensure that the commit name is fitting the guidelines.

  • Ensure that the meta field contains correct information.

    • License must be checked to be fitting upstream license.

    • Platforms should be set or the package will not get binary substitutes.

    • A maintainer must be set, this can be the package submitter or a community member that accepts to take maintainership of the package.

  • Ensure that the code contains no typos.

  • Ensure the package source.

    • Mirrors urls should be used when available.

    • The most appropriate function should be used (e.g. packages from GitHub should use fetchFromGitHub).

  • Building the package locally.

  • Running every binary.

Example 14.2. Sample template for a new package review

##### Reviewed points

- [ ] package path fits guidelines
- [ ] package name fits guidelines
- [ ] package version fits guidelines
- [ ] package build on ARCHITECTURE
- [ ] executables tested on ARCHITECTURE
- [ ] `meta.description` is set and fits guidelines
- [ ] `meta.license` fits upstream license
- [ ] `meta.platforms` is set
- [ ] `meta.maintainers` is set
- [ ] build time only dependencies are declared in `nativeBuildInputs`
- [ ] source is fetched using the appropriate function
- [ ] phases are respected
- [ ] patches that are remotely available are fetched with `fetchpatch`

##### Possible improvements

##### Comments



14.3. Module updates

Module updates are submissions changing modules in some ways. These often contains changes to the options or introduce new options.

Reviewing process

  • Add labels to the pull-request. (Requires commit rights)

    • 8.has: module (update) and any topic label that fit the module.

  • Ensure that the module maintainers are notified.

    • Mention-bot notify GitHub users based on the submitted changes, but it can happen that it miss some of the package maintainers.

  • Ensure that the module tests, if any, are succeeding.

  • Ensure that the introduced options are correct.

    • Type should be appropriate (string related types differs in their merging capabilities, optionSet and string types are deprecated).

    • Description, default and example should be provided.

  • Ensure that option changes are backward compatible.

    • mkRenamedOptionModule and mkAliasOptionModule functions provide way to make option changes backward compatible.

  • Ensure that removed options are declared with mkRemovedOptionModule

  • Ensure that changes that are not backward compatible are mentioned in release notes.

  • Ensure that documentations affected by the change is updated.

Example 14.3. Sample template for a module update review

##### Reviewed points

- [ ] changes are backward compatible
- [ ] removed options are declared with `mkRemovedOptionModule`
- [ ] changes that are not backward compatible are documented in release notes
- [ ] module tests succeed on ARCHITECTURE
- [ ] options types are appropriate
- [ ] options description is set
- [ ] options example is provided
- [ ] documentation affected by the changes is updated

##### Possible improvements

##### Comments



14.4. New modules

New modules submissions introduce a new module to NixOS.

  • Add labels to the pull-request. (Requires commit rights)

    • 8.has: module (new) and any topic label that fit the module.

  • Ensure that the module tests, if any, are succeeding.

  • Ensure that the introduced options are correct.

    • Type should be appropriate (string related types differs in their merging capabilities, optionSet and string types are deprecated).

    • Description, default and example should be provided.

  • Ensure that module meta field is present

    • Maintainers should be declared in meta.maintainers.

    • Module documentation should be declared with meta.doc.

  • Ensure that the module respect other modules functionality.

    • For example, enabling a module should not open firewall ports by default.

Example 14.4. Sample template for a new module review

##### Reviewed points

- [ ] module path fits the guidelines
- [ ] module tests succeed on ARCHITECTURE
- [ ] options have appropriate types
- [ ] options have default
- [ ] options have example
- [ ] options have descriptions
- [ ] No unneeded package is added to system.environmentPackages
- [ ] meta.maintainers is set
- [ ] module documentation is declared in meta.doc

##### Possible improvements

##### Comments



14.5. Other submissions

Other type of submissions requires different reviewing steps.

If you consider having enough knowledge and experience in a topic and would like to be a long-term reviewer for related submissions, please contact the current reviewers for that topic. They will give you information about the reviewing process. The main reviewers for a topic can be hard to find as there is no list, but checking past pull-requests to see who reviewed or git-blaming the code to see who committed to that topic can give some hints.

Container system, boot system and library changes are some examples of the pull requests fitting this category.

14.6. Merging pull-requests

It is possible for community members that have enough knowledge and experience on a special topic to contribute by merging pull requests.

TODO: add the procedure to request merging rights.

In a case a contributor leaves definitively the Nix community, he should create an issue or notify the mailing list with references of packages and modules he maintains so the maintainership can be taken over by other contributors.

The DocBook sources of the Nixpkgs manual are in the doc subdirectory of the Nixpkgs repository. If you make modifications to the manual, it's important to build it before committing. You can do that as follows:

$ cd /path/to/nixpkgs
$ nix-build doc

If the build succeeds, the manual will be in ./result/share/doc/nixpkgs/manual.html.