Nix papers

[Show all abstracts]

[Hide all abstracts]

2011

• Joachim Schiele. Multi-Platform Software Package Management. March 2011. Diplomarbeit at the University of Tübingen. [pdf] [abstract]
  Abstract: Today's package management is a complex field. This diploma thesis analyses different package management systems in several regards: how the packaging is done; how the user interacts with the package manager; Apt (Debian Linux), Portage (Gentoo Linux) and Nix (Nix OS) are compared in great detail. To get practical results, the author used the evopedia application, an open source offline Wikipedia reader, to experiment with several different package managers. This diploma thesis also tries to answer why there are no complex package managers for Microsoft Windows. Most package managers use their own terms which can not be generalized to other package managers. This problem was solved by defining meaningful words which describe certain properties of a package manager.

2010

• Sander van der Burg and Eelco Dolstra. Automating System Tests Using Declarative Virtual Machines. In 21st IEEE International Symposium on Software Reliability Engineering (ISSRE 2010), San Jose, California, USA. November 2010. [pdf] [abstract]

  Abstract: Automated regression test suites are an essential software engineering practice: they provide developers with rapid feedback on the impact of changes to a system’s source code. The inclusion of a test case in an automated test suite requires that the system’s build process can automatically provide all the environmental dependencies of the test. These are external elements necessary for a test to succeed, such as shared libraries, running programs, and so on. For some tests (e.g., a compiler’s), these requirements are simple to meet.

  However, many kinds of tests, especially at the integration or system level, have complex dependencies that are hard to provide automatically, such as running database servers, administrative privileges, services on external machines or specific network topologies. As such dependencies make tests difficult to script, they are often only performed manually, if at all. This particularly affects testing of distributed systems and system-level software.

  This paper shows how we can automatically instantiate the complex environments necessary for tests by creating (networks of) virtual machines on the fly from declarative specifications. Building on NixOS, a Linux distribution with a declarative configuration model, these specifications concisely model the required environmental dependencies. We also describe techniques that allow efficient instantiation of VMs. As a result, complex system tests become as easy to specify and execute as unit tests. We evaluate our approach using a number of representative problems, including automated regression testing of a Linux distribution.

• Eelco Dolstra and Andres Löh and Nicolas Pierron. NixOS: A Purely Functional Linux Distribution. Cambridge University Press, 2010. [pdf] [doi] [abstract]
  Abstract: Existing package and system configuration management tools suffer from an imperative model, where system administration actions such as upgrading packages or changes to system configuration files are stateful: they destructively update the state of the system. This leads to many problems, such as the inability to roll back changes easily, to deploy multiple versions of a package side-by-side, to reproduce a configuration deterministically on another machine, or to reliably upgrade a system. In this article we show that we can overcome these problems by moving to a purely functional system configuration model. This means that all static parts of a system (such as software packages, configuration files and system startup scripts) are built by pure functions and are immutable, stored in a way analogously to a heap in a purely functional language. We have implemented this model in NixOS, a non-trivial Linux distribution that uses the Nix package manager to build the entire system configuration from a modular, purely functional specification.

2009

• Sander van der Burg and Eelco Dolstra and Merijn de Jonge and Eelco Visser. Software Deployment in a Dynamic Cloud — From Device to Service Orientation in a Hospital Environment. In ICSE 2009 Workshop on Software Engineering Challenges in Cloud Computing, pages 61–66, Vancouver, British Columbia, Canada. May 2009. [pdf] [acm] [abstract]
  Abstract: Hospital environments are currently primarily device-oriented: software services are installed, often manually, on specific devices. For instance, an application to view MRI scans may only be available on a limited number of workstations. The medical world is changing to a service-oriented environment, which means that every software service should be available on every device. However, these devices have widely varying capabilities, ranging from powerful workstations to PDAs, and high-bandwidth local machines to low-bandwidth remote machines. To support running applications in such an environment, we need to treat the hospital machines as a cloud, where components of the application are automatically deployed to machines in the cloud with the required capabilities and connectivity. In this paper, we suggest an architecture for applications in such a cloud, in which components are reliably and automatically deployed on the basis of a declarative model of the application using the Nix package manager.

2008

• Sander van der Burg and Eelco Dolstra and Merijn de Jonge. Atomic Upgrading of Distributed Systems. In First ACM Workshop on Hot Topics in Software Upgrades (HotSWUp), Nashville, Tennessee, USA. October 2008. [pdf] [acm] [abstract]
  Abstract: Upgrading distributed systems is a complex process. It requires installing the right services on the right machines, configuring them correctly, and so on, which is error-prone and tedious. Moreover, since services in a distributed system depend on each other and are updated separately, upgrades typically are not atomic: there is a time window during which some but not all services are updated, and a new version of one service might temporarily talk to an old version of another service. Previously we implemented the Nix package management system, which allows atomic upgrades and rollbacks on single systems. In this paper we show an extension to Nix that enables the deployment of distributed systems on the basis of a declarative deployment model, and supports atomic upgrades of such systems.
• Eelco Dolstra and Andres Löh. NixOS: A Purely Functional Linux Distribution. In ICFP 2008: 13th ACM SIGPLAN International Conference on Functional Programming, pages 367–378, Victoria, British Columbia, Canada. September 2008. [pdf] [acm] [abstract]
  Abstract: Existing package and system configuration management tools suffer from an imperative model, where system administration actions such as upgrading packages or changes to system configuration files are stateful: they destructively update the state of the system. This leads to many problems, such as the inability to roll back changes easily, to run multiple versions of a package side-by-side, to reproduce a configuration deterministically on another machine, or to reliably upgrade a system. In this paper we show that we can overcome these problems by moving to a purely functional system configuration model. This means that all static parts of a system (such as software packages, configuration files and system startup scripts) are built by pure functions and are immutable, stored in a way analogously to a heap in a purely function language. We have implemented this model in NixOS, a non-trivial Linux distribution that uses the Nix package manager to build the entire system configuration from a purely functional specification.
• Eelco Dolstra and Eelco Visser. The Nix Build Farm: A Declarative Approach to Continuous Integration. In International Workshop on Advanced Software Development Tools and Techniques (WASDeTT 2008), Paphos, Cyprus. July 2008. To appear. [pdf] [abstract]
  Abstract: There are many tools to support continuous integration (the process of automatically and continuously building a project from a version management repository). However, they do not have good support for variability in the build environment: dependencies such as compilers, libraries or testing tools must typically be installed manually on all machines on which automated builds are performed. The Nix package manager solves this problem: it has a purely functional language for describing package build actions and their dependencies, allowing the build environment for projects to be produced automatically and deterministically. We have used Nix to build a continuous integration tool, the Nix build farm, that is in use to continuously build and release a large set of projects.
• Eelco Dolstra. Maximal Laziness — An Efficient Interpretation Technique for Purely Functional DSLs. In Eighth Workshop on Language Descriptions, Tools and Applications (LDTA 2008), volume 238, number 5 of Electronic Notes in Theoretical Computer Science, pages 81–99, Budapest, Hungary. Elsevier Science Publishers, April 2008. To appear. [pdf] [doi] [abstract]
  Abstract: In lazy functional languages, any variable is evaluated at most once. This paper proposes the notion of maximal laziness, in which syntactically equal terms are evaluated at most once: if two terms e₁ and e₂ arising during the evaluation of a program have the same abstract syntax representation, then only one will be evaluated, while the other will reuse the former’s evaluation result. Maximal laziness can be implemented easily in interpreters for purely functional languages based on term rewriting systems that have the property of maximal sharing — if two terms are equal, they have the same address. It makes it easier to write interpreters, as techniques such as closure updating, which would otherwise be required for efficiency, are not needed. Instead, a straight-forward translation of call-by-name semantic rules yields a call-by-need interpreter, reducing the gap between the language specification and its implementation. Moreover, maximal laziness obviates the need for optimisations such as memoisation and let-floating.

2007

• Eelco Dolstra and Armijn Hemel. Purely Functional System Configuration Management. In 11th Workshop on Hot Topics in Operating Systems (HotOS XI), San Diego, California, USA. USENIX, May 2007. [pdf] [abstract]
  Abstract: System configuration management is difficult because systems evolve in an undisciplined way: packages are upgraded, configuration files are edited, and so on. The management of existing operating systems is strongly imperative in nature, since software packages and configuration data (e.g., /bin and /etc in Unix) can be seen as imperative data structures: they are updated in-place by system administration actions. In this paper we present an alternative approach to system configuration management: a purely functional method, analogous to languages like Haskell. In this approach, the static parts of a configuration — software packages, configuration files, control scripts — are built from pure functions, i.e., the results depend solely on the specified inputs of the function and are immutable. As a result, realising a system configuration becomes deterministic and reproducible. Upgrading to a new configuration is mostly atomic and doesn’t overwrite anything of the old configuration, thus enabling rollbacks. We have implemented the purely functional model in a small but realistic Linux-based operating system distribution called NixOS.
• Eelco Dolstra and Eelco Visser. Automated Software Testing and Release with Nix Build Farms. In Verification and Validation of Software Systems (VVSS-2007), Eindhoven, The Netherlands. March 2007.

2006

• Armijn Hemel. NixOS: the Nix based operating system. Master’s thesis, INF/SCR-2005-091, Institute of Information and Computing Sciences, Utrecht University, Utrecht, The Netherlands. August 2006. [pdf] [abstract]

  Abstract: The subject of this thesis is how the Nix package management system can be applied to manage a whole Linux distribution. Many conventional package management systems have drawbacks that Nix fixes. But, Nix has never been used to deploy and manage a whole system.

  In this thesis a proof of concept Linux distribution called NixOS is described. NixOS uses the Nix package management system to manage all software that is installed on the system, including the Linux kernel, all software and system services.

  Using Nix to manage all software on a system, as is done on NixOS, has several advantages. Developers don’t need to be worried that unwanted dependencies are picked up during the build of a software package, since these are completely eliminated. System administrators get the possibility to deploy services using Nix and how they can immediately use all benefits from Nix, including atomic upgrades and rollbacks, without going through a painful cycle of rolling back a service, with all its, possibly also updated, dependencies.

  This thesis describes the implementation NixOS, including pitfalls that were encountered and choices that were made. Also shown are some concrete results of running NixOS and how NixOS can be bettered.

• Eelco Dolstra. The Purely Functional Software Deployment Model. PhD thesis, Faculty of Science, Utrecht, The Netherlands. January 2006. ISBN 90-393-4130-3. [pdf] [igitur] [abstract]

  Abstract: Software deployment is the set of activities related to getting software components to work on the machines of end users. It includes activities such as installation, upgrading, uninstallation, and so on. Many tools have been developed to support deployment, but they all have serious limitations with respect to correctness. For instance, the installation of a component can lead to the failure of previously installed components; a component might require other components that are not present; and it is generally difficult to undo deployment actions. The fundamental causes of these problems are a lack of isolation between components, the difficulty in identifying the dependencies between components, and incompatibilities between versions and variants of components.

  This thesis describes a better approach based on a purely functional deployment model, implemented in a deployment system called Nix. Components are stored in isolation from each other in a Nix store. Each component has a name that contains a cryptographic hash of all inputs that contributed to its build process, and the content of a component never changes after it has been built. Hence the model is purely functional.

  This storage scheme provides several important advantages. First, it ensures isolation between components: if two components differ in any way, they will be stored in different locations and will not overwrite each other. Second, it allows us to identify component dependencies. Undeclared build time dependencies are prevented due to the absence of “global” component directories used in other deployment systems. Runtime dependencies can be found by scanning for cryptographic hashes in the binary contents of components, a technique analogous to conservative garbage collection in programming language implementation. Since dependency information is complete, complete deployment can be performed by copying closures of components under the dependency relation.

  Developers and users are not confronted with components’ cryptographic hashes directly. Components are built automatically from Nix expressions, which describe how to build and compose arbitrary software components; hashes are computed as part of this process. Components are automatically made available to users through “user environments”, which are synthesised sets of activated components. User environments enable atomic upgrades and rollbacks, as well as different sets of activated components for different users.

  Nix expressions provide a source-based deployment model. However, source-based deployment can be transparently optimised into binary deployment by making pre-built binaries (keyed on their cryptographic hashes) available in a shared location such as a network server. This is referred to as transparent source/binary deployment.

  The purely functional deployment model has been validated by applying it to the deployment of more than 278 existing Unix packages. In addition, this thesis shows that the model can be applied naturally to the related activities of continuous integration using build farms, service deployment and build management.

2005

• Eelco Dolstra. Secure Sharing Between Untrusted Users in a Transparent Source/Binary Deployment Model. In 20th IEEE/ACM International Conference on Automated Software Engineering (ASE 2005), pages 154–163, Long Beach, California, USA. ACM Press, November 2005. [pdf] [acm] [abstract]

  Abstract: The Nix software deployment system is based on the paradigm of transparent source/binary deployment: distributors deploy descriptors that build components from source, while client machines can transparently optimise such source builds by downloading pre-built binaries from remote repositories. This model combines the simplicity and flexibility of source deployment with the efficiency of binary deployment. A desirable property is sharing of components: if multiple users install from the same source descriptors, ideally only one remotely built binary should be installed. The problem is that users must trust that remotely downloaded binaries were built from the sources they are claimed to have been built from, while users in general do not have a trust relation with each other or with the same remote repositories.

  This paper presents three models that enable sharing: the extensional model that requires that all users on a system have the same remote trust relations, the intensional model that does not have this requirement but may be suboptimal in terms of space use, and the mixed model that merges the best properties of both. The latter two models are achieved through a novel technique of {\em hash rewriting} in content-addressable component stores, and were implemented in the context of the Nix system.

• Eelco Dolstra and Martin Bravenboer and Eelco Visser. Service Configuration Management. In E. James Whitehead, Jr. and Annita Persson Dahlqvist (Eds.), 12th International Workshop on Software Configuration Management (SCM-12), pages 83–98, Long Beach, California, USA. September 2005. [pdf] [abstract]
  Abstract: The deployment of services — sets of running programs that provide some useful facility on a system or network — is typically implemented through a manual, time-consuming and error-prone process. For instance, system administrators must deploy the necessary software components, edit configuration files, start or stop processes, and so on. This is often done in an ad hoc style with no reproducibility, violating proper configuration management practices. In this paper we show that build management, software deployment and service deployment can be integrated into a single formalism. We do this in the context of the Nix software deployment system, and show that its advantages — co-existence of versions and variants, atomic upgrades and rollbacks, and component closure — extend naturally to service deployment. The approach also elegantly extends to distributed services. In addition, we show that the Nix expression language can simplify the implementation of crosscutting variation points in services.
• Eelco Dolstra. Efficient Upgrading in a Purely Functional Component Deployment Model. In George Heineman et al. (Ed.), Eighth International SIGSOFT Symposium on Component-based Software Engineering (CBSE 2005), volume 3489 of Lecture Notes in Computer Science, pages 219–234, St. Louis, Missouri, USA. Springer-Verlag, May 2005. © Springer-Verlag. [pdf] [abstract]
  Abstract: Safe and efficient deployment of software components is an important aspect of CBSE. The Nix deployment system enables side-by-side deployment of different versions and variants of components, complete installation, safe upgrades, and safe uninstalls through garbage collection. It accomplishes this through a purely functional deployment model, meaning that the file system content of a component only depends on the inputs used to build it, and never changes afterwards. An apparent downside to this model is that upgrading “fundamental” components used as build inputs by many other components becomes expensive, since all of these must be rebuilt and redeployed. In this paper we show that binary patching between sets of components enables efficient deployment of upgrades in the purely functional model, transparently to users. Sequences of patches can be combined automatically to enable upgrading between arbitrary versions. The approach was empirically validated.

2004

• Eelco Dolstra and Merijn de Jonge and Eelco Visser. Nix: A Safe and Policy-Free System for Software Deployment. In Lee Damon (Ed.), 18th Large Installation System Administration Conference (LISA '04), pages 79–92, Atlanta, Georgia, USA. USENIX, November 2004. [pdf] [abstract]
  Abstract: Existing systems for software deployment are neither safe nor sufficiently flexible. Primary safety issues are the inability to enforce reliable specification of component dependencies, and the lack of support for multiple versions or variants of a component. This renders deployment operations such as upgrading or deleting components dangerous and unpredictable. A deployment system must also be flexible (i.e., policy-free) enough to support both centralised and local package management, and to allow a variety of mechanisms for transferring components. In this paper we present Nix, a deployment system that addresses these issues through a simple technique of using cryptographic hashes to compute unique paths for component instances.
• Eelco Dolstra and Eelco Visser and Merijn de Jonge. Imposing a Memory Management Discipline on Software Deployment. In 26th International Conference on Software Engineering (ICSE 2004), pages 583–592, Edinburgh, Scotland. IEEE Computer Society, May 2004. [pdf] [acm] [abstract]
  Abstract: The deployment of software components frequently fails because dependencies on other components are not declared explicitly or are declared imprecisely. This results in an incomplete reproduction of the environment necessary for proper operation, or in interference between incompatible variants. In this paper we show that these deployment hazards are similar to pointer hazards in memory models of programming languages and can be countered by imposing a memory management discipline on software deployment. Based on this analysis we have developed a generic, platform and language independent, discipline for deployment that allows precise dependency verification; exact identification of component variants; computation of complete closures containing all components on which a component depends; maximal sharing of components between such closures; and concurrent installation of revisions and variants of components. We have implemented the approach in the Nix deployment system, and used it for the deployment of a large number of existing Linux packages. We compare its effectiveness to other deployment systems.

2003

• Eelco Dolstra. Integrating Software Construction and Software Deployment. In Bernhard Westfechtel and André van der Hoek (Eds.), 11th International Workshop on Software Configuration Management (SCM-11), volume 2649 of Lecture Notes in Computer Science, pages 102–117, Portland, Oregon, USA. May 2003. [pdf] [abstract]
  Abstract: Classically, software deployment is a process consisting of building the software, packaging it for distribution, and installing it at the target site. This approach has two problems. First, a package must be annotated with dependency information and other meta-data. This to some extent overlaps with component dependencies used in the build process. Second, the same source system can often be built into an often very large number of variants. The distributor must decide which element(s) of the variant space will be packaged, reducing the flexibility for the receiver of the package. In this paper we show how building and deployment can be integrated into a single formalism. We describe a build manager called Maak that can handle deployment through a sufficiently general module system. Through the sharing of generated files, a source distribution transparently turns into a binary distribution, removing the dichotomy between these two modes of deployment. In addition, the creation and deployment of variants becomes easy through the use of a simple functional language as the build formalism.