From 2320a2f93e0aa61e8d45dcfb3fb536361a1fcb6f Mon Sep 17 00:00:00 2001 From: Artturin Date: Fri, 10 Jun 2022 17:46:57 +0300 Subject: [PATCH 01/15] systemd,launchd match nixos open files limit it was bumped in https://github.com/NixOS/nixpkgs/pull/176558 --- misc/launchd/org.nixos.nix-daemon.plist.in | 2 +- misc/systemd/nix-daemon.service.in | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/misc/launchd/org.nixos.nix-daemon.plist.in b/misc/launchd/org.nixos.nix-daemon.plist.in index da1970f69..5fa489b20 100644 --- a/misc/launchd/org.nixos.nix-daemon.plist.in +++ b/misc/launchd/org.nixos.nix-daemon.plist.in @@ -28,7 +28,7 @@ SoftResourceLimits NumberOfFiles - 4096 + 1048576 diff --git a/misc/systemd/nix-daemon.service.in b/misc/systemd/nix-daemon.service.in index e3ac42beb..f46413630 100644 --- a/misc/systemd/nix-daemon.service.in +++ b/misc/systemd/nix-daemon.service.in @@ -9,7 +9,7 @@ ConditionPathIsReadWrite=@localstatedir@/nix/daemon-socket [Service] ExecStart=@@bindir@/nix-daemon nix-daemon --daemon KillMode=process -LimitNOFILE=4096 +LimitNOFILE=1048576 [Install] WantedBy=multi-user.target From cc620d961f1fa68038ef44f5947fc053708ab3ad Mon Sep 17 00:00:00 2001 From: Liu Xiaoyi Date: Sun, 20 Nov 2022 20:04:22 +0800 Subject: [PATCH 02/15] Updated uninstall guide involving systemd Co-authored-by: sequencer --- .../src/installation/installing-binary.md | 32 +++++++++++++------ 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/doc/manual/src/installation/installing-binary.md b/doc/manual/src/installation/installing-binary.md index 31faeadc2..1b7db59cd 100644 --- a/doc/manual/src/installation/installing-binary.md +++ b/doc/manual/src/installation/installing-binary.md @@ -88,6 +88,28 @@ extension. The installer will also create `/etc/profile.d/nix.sh`. ### Linux +To begin, if you are on Linux with systemd, remove the Nix daemon service: + +```console +sudo systemctl stop nix-daemon.socket +sudo systemctl stop nix-daemon.service +sudo systemctl disable nix-daemon.socket +sudo systemctl disable nix-daemon.service +sudo systemctl daemon-reload +``` + +Then you can remove systemd service files: + +```console +sudo rm -f /etc/systemd/system/nix-daemon.service /etc/systemd/system/nix-daemon.socket +``` + +Also, the installer script uses systemd-tmpfiles (if presents) to create the socket directory. You may also want to remove the configuration for that: + +```console +sudo rm -f /etc/tmpfiles.d/nix-daemon.conf +``` + Remove files created by Nix: ```console @@ -103,16 +125,6 @@ done sudo groupdel 30000 ``` -If you are on Linux with systemd, remove the Nix daemon service: - -```console -sudo systemctl stop nix-daemon.socket -sudo systemctl stop nix-daemon.service -sudo systemctl disable nix-daemon.socket -sudo systemctl disable nix-daemon.service -sudo systemctl daemon-reload -``` - There may also be references to Nix in - `/etc/profile` From 4d55acf5154639cd1a54435ace28cd497b5a9eb8 Mon Sep 17 00:00:00 2001 From: Liu Xiaoyi Date: Mon, 21 Nov 2022 13:46:22 +0800 Subject: [PATCH 03/15] Apply suggestions from code review Co-authored-by: Sandro --- doc/manual/src/installation/installing-binary.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/doc/manual/src/installation/installing-binary.md b/doc/manual/src/installation/installing-binary.md index 1b7db59cd..8cdd64df6 100644 --- a/doc/manual/src/installation/installing-binary.md +++ b/doc/manual/src/installation/installing-binary.md @@ -91,23 +91,21 @@ extension. The installer will also create `/etc/profile.d/nix.sh`. To begin, if you are on Linux with systemd, remove the Nix daemon service: ```console -sudo systemctl stop nix-daemon.socket sudo systemctl stop nix-daemon.service -sudo systemctl disable nix-daemon.socket -sudo systemctl disable nix-daemon.service +sudo systemctl disable nix-daemon.socket nix-daemon.service sudo systemctl daemon-reload ``` Then you can remove systemd service files: ```console -sudo rm -f /etc/systemd/system/nix-daemon.service /etc/systemd/system/nix-daemon.socket +sudo rm /etc/systemd/system/nix-daemon.service /etc/systemd/system/nix-daemon.socket ``` Also, the installer script uses systemd-tmpfiles (if presents) to create the socket directory. You may also want to remove the configuration for that: ```console -sudo rm -f /etc/tmpfiles.d/nix-daemon.conf +sudo rm /etc/tmpfiles.d/nix-daemon.conf ``` Remove files created by Nix: From c4ce89f772d85e2a39727927903ca105a8c6a168 Mon Sep 17 00:00:00 2001 From: Liu Xiaoyi Date: Sat, 26 Nov 2022 22:01:51 +0800 Subject: [PATCH 04/15] Clarify uninstallation steps on Linux Co-authored-by: Valentin Gagarin --- .../src/installation/installing-binary.md | 31 ++++++++++--------- 1 file changed, 17 insertions(+), 14 deletions(-) diff --git a/doc/manual/src/installation/installing-binary.md b/doc/manual/src/installation/installing-binary.md index 8cdd64df6..a9378681d 100644 --- a/doc/manual/src/installation/installing-binary.md +++ b/doc/manual/src/installation/installing-binary.md @@ -88,25 +88,28 @@ extension. The installer will also create `/etc/profile.d/nix.sh`. ### Linux -To begin, if you are on Linux with systemd, remove the Nix daemon service: +If you are on Linux with systemd: -```console -sudo systemctl stop nix-daemon.service -sudo systemctl disable nix-daemon.socket nix-daemon.service -sudo systemctl daemon-reload -``` +1. Remove the Nix daemon service: -Then you can remove systemd service files: + ```console + sudo systemctl stop nix-daemon.service + sudo systemctl disable nix-daemon.socket nix-daemon.service + sudo systemctl daemon-reload + ``` -```console -sudo rm /etc/systemd/system/nix-daemon.service /etc/systemd/system/nix-daemon.socket -``` +1. Remove systemd service files: -Also, the installer script uses systemd-tmpfiles (if presents) to create the socket directory. You may also want to remove the configuration for that: + ```console + sudo rm /etc/systemd/system/nix-daemon.service /etc/systemd/system/nix-daemon.socket + ``` -```console -sudo rm /etc/tmpfiles.d/nix-daemon.conf -``` +1. The installer script uses systemd-tmpfiles to create the socket directory. + You may also want to remove the configuration for that: + + ```console + sudo rm /etc/tmpfiles.d/nix-daemon.conf + ``` Remove files created by Nix: From dfa27e6b2feb082b0a276338868b069458ec00db Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Thu, 1 Dec 2022 03:37:14 +0100 Subject: [PATCH 05/15] refactor rendering documentation of options this makes more obvious what the code produces, and the structure of the output easier to change --- doc/manual/generate-options.nix | 66 +++++++++++++++++++-------------- 1 file changed, 39 insertions(+), 27 deletions(-) diff --git a/doc/manual/generate-options.nix b/doc/manual/generate-options.nix index 680b709c8..814144c20 100644 --- a/doc/manual/generate-options.nix +++ b/doc/manual/generate-options.nix @@ -1,29 +1,41 @@ -with builtins; -with import ./utils.nix; +let + inherit (builtins) attrNames concatStringsSep isAttrs isBool; + inherit (import ./utils.nix) concatStrings squash splitLines; +in -options: +optionsInfo: +let + showOption = name: + let + inherit (optionsInfo.${name}) description documentDefault defaultValue aliases; + result = squash '' + - [`${name}`]{#conf-${name}} -concatStrings (map - (name: - let option = options.${name}; in - " - [`${name}`](#conf-${name})" - + "

\n\n" - + concatStrings (map (s: " ${s}\n") (splitLines option.description)) + "\n\n" - + (if option.documentDefault - then " **Default:** " + ( - if option.defaultValue == "" || option.defaultValue == [] - then "*empty*" - else if isBool option.defaultValue - then (if option.defaultValue then "`true`" else "`false`") - else - # n.b. a StringMap value type is specified as a string, but - # this shows the value type. The empty stringmap is "null" in - # JSON, but that converts to "{ }" here. - (if isAttrs option.defaultValue then "`\"\"`" - else "`" + toString option.defaultValue + "`")) + "\n\n" - else " **Default:** *machine-specific*\n") - + (if option.aliases != [] - then " **Deprecated alias:** " + (concatStringsSep ", " (map (s: "`${s}`") option.aliases)) + "\n\n" - else "") - ) - (attrNames options)) + ${indent " " body} + ''; + # separate body to cleanly handle indentation + body = '' + ${description} + + **Default:** ${showDefault documentDefault defaultValue} + + ${showAliases aliases} + ''; + showDefault = documentDefault: defaultValue: + if documentDefault then + # a StringMap value type is specified as a string, but + # this shows the value type. The empty stringmap is `null` in + # JSON, but that converts to `{ }` here. + if defaultValue == "" || defaultValue == [] || isAttrs defaultValue + then "*empty*" + else if isBool defaultValue then + if defaultValue then "`true`" else "`false`" + else "`${toString defaultValue}`" + else "*machine-specific*"; + showAliases = aliases: + if aliases == [] then "" else + "**Deprecated alias:** ${(concatStringsSep ", " (map (s: "`${s}`") aliases))}"; + indent = prefix: s: + concatStringsSep "\n" (map (x: if x == "" then x else "${prefix}${x}") (splitLines s)); + in result; +in concatStrings (map showOption (attrNames optionsInfo)) From 8c7661da0963a6a578605374cd9b309177563b3d Mon Sep 17 00:00:00 2001 From: Taeer Bar-Yam Date: Mon, 5 Dec 2022 23:21:58 -0500 Subject: [PATCH 06/15] check the store for input before failing (hopefully fix #6383) --- src/libstore/build/derivation-goal.cc | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 5aed51bcd..7dd39051b 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -1016,11 +1016,20 @@ void DerivationGoal::resolvedFinished() throw Error( "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolvedFinished,resolve)", worker.store.printStorePath(drvPath), wantedOutput); - auto realisation = get(resolvedResult.builtOutputs, DrvOutput { *resolvedHash, wantedOutput }); - if (!realisation) + + const Realisation * realisation = get(resolvedResult.builtOutputs, DrvOutput { *resolvedHash, wantedOutput }); + if (!realisation) { + /* The above `get` should work. But sateful tracking of + outputs in resolvedResult, this can get out of sync with the + store, which is our actual source of truth. For now we just + check the store directly if it fails. */ + realisation = worker.evalStore.queryRealisation(DrvOutput { *resolvedHash, wantedOutput }).get(); + if (!realisation) { throw Error( "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolvedFinished,realisation)", worker.store.printStorePath(resolvedDrvGoal->drvPath), wantedOutput); + } + } if (drv->type().isPure()) { auto newRealisation = *realisation; newRealisation.id = DrvOutput { initialOutput->outputHash, wantedOutput }; From 08dcd22582d65e73f29df79b3765e76cea8f3314 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Na=C3=AFm=20Favier?= Date: Tue, 6 Dec 2022 16:36:42 +0100 Subject: [PATCH 07/15] tests: don't refer to TMPDIR --- tests/check.nix | 2 +- tests/check.sh | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/check.nix b/tests/check.nix index ed91ff845..ddab8eea9 100644 --- a/tests/check.nix +++ b/tests/check.nix @@ -44,7 +44,7 @@ with import ./config.nix; }; hashmismatch = import { - url = "file://" + builtins.getEnv "TMPDIR" + "/dummy"; + url = "file://" + builtins.getEnv "TEST_ROOT" + "/dummy"; sha256 = "0mdqa9w1p6cmli6976v4wi0sw9r4p5prkj7lzfd1877wk11c9c73"; }; diff --git a/tests/check.sh b/tests/check.sh index 495202781..fbd784fc5 100644 --- a/tests/check.sh +++ b/tests/check.sh @@ -91,13 +91,13 @@ nix-build check.nix -A fetchurl --no-out-link --check nix-build check.nix -A fetchurl --no-out-link --repair [[ $(cat $path) != foo ]] -echo 'Hello World' > $TMPDIR/dummy +echo 'Hello World' > $TEST_ROOT/dummy nix-build check.nix -A hashmismatch --no-out-link || status=$? [ "$status" = "102" ] -echo -n > $TMPDIR/dummy +echo -n > $TEST_ROOT/dummy nix-build check.nix -A hashmismatch --no-out-link -echo 'Hello World' > $TMPDIR/dummy +echo 'Hello World' > $TEST_ROOT/dummy nix-build check.nix -A hashmismatch --no-out-link --check || status=$? [ "$status" = "102" ] From 1c8de7d3d03d7a6ba259387b0698874fa879428c Mon Sep 17 00:00:00 2001 From: Taeer Bar-Yam Date: Tue, 6 Dec 2022 11:25:38 -0500 Subject: [PATCH 08/15] improve style --- src/libstore/build/derivation-goal.cc | 29 +++++++++++++++------------ 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 7dd39051b..1f9bfaae0 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -1017,30 +1017,33 @@ void DerivationGoal::resolvedFinished() "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolvedFinished,resolve)", worker.store.printStorePath(drvPath), wantedOutput); - const Realisation * realisation = get(resolvedResult.builtOutputs, DrvOutput { *resolvedHash, wantedOutput }); - if (!realisation) { + auto realisation = [&]{ + auto take1 = get(resolvedResult.builtOutputs, DrvOutput { *resolvedHash, wantedOutput }); + if (take1) return *take1; + /* The above `get` should work. But sateful tracking of outputs in resolvedResult, this can get out of sync with the store, which is our actual source of truth. For now we just check the store directly if it fails. */ - realisation = worker.evalStore.queryRealisation(DrvOutput { *resolvedHash, wantedOutput }).get(); - if (!realisation) { - throw Error( - "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolvedFinished,realisation)", - worker.store.printStorePath(resolvedDrvGoal->drvPath), wantedOutput); - } - } + auto take2 = worker.evalStore.queryRealisation(DrvOutput { *resolvedHash, wantedOutput }); + if (take2) return *take2; + + throw Error( + "derivation '%s' doesn't have expected output '%s' (derivation-goal.cc/resolvedFinished,realisation)", + worker.store.printStorePath(resolvedDrvGoal->drvPath), wantedOutput); + }(); + if (drv->type().isPure()) { - auto newRealisation = *realisation; + auto newRealisation = realisation; newRealisation.id = DrvOutput { initialOutput->outputHash, wantedOutput }; newRealisation.signatures.clear(); if (!drv->type().isFixed()) - newRealisation.dependentRealisations = drvOutputReferences(worker.store, *drv, realisation->outPath); + newRealisation.dependentRealisations = drvOutputReferences(worker.store, *drv, realisation.outPath); signRealisation(newRealisation); worker.store.registerDrvOutput(newRealisation); } - outputPaths.insert(realisation->outPath); - builtOutputs.emplace(realisation->id, *realisation); + outputPaths.insert(realisation.outPath); + builtOutputs.emplace(realisation.id, realisation); } runPostBuildHook( From 6833ded76441a43b94ed0cf1827b9714862fdedc Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 7 Dec 2022 10:03:09 +0100 Subject: [PATCH 09/15] let installer blurb point to community page being too specific about it requires more maintenance (or otherwise produced more confusion and churn), since these points of contact change over time. --- scripts/install-multi-user.sh | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/scripts/install-multi-user.sh b/scripts/install-multi-user.sh index 96c0f302b..af8a9b278 100644 --- a/scripts/install-multi-user.sh +++ b/scripts/install-multi-user.sh @@ -99,11 +99,7 @@ is_os_darwin() { contact_us() { echo "You can open an issue at https://github.com/nixos/nix/issues" echo "" - echo "Or feel free to contact the team:" - echo " - Matrix: #nix:nixos.org" - echo " - IRC: in #nixos on irc.libera.chat" - echo " - twitter: @nixos_org" - echo " - forum: https://discourse.nixos.org" + echo "Or get in touch with the community: https://nixos.org/community" } get_help() { echo "We'd love to help if you need it." From 8e0946e8df968391d1430af8377bdb51204e4666 Mon Sep 17 00:00:00 2001 From: Linus Heckemann Date: Mon, 26 Sep 2022 20:55:56 +0200 Subject: [PATCH 10/15] Remove repeat and enforce-determinism options These only functioned if a very narrow combination of conditions held: - The result path does not yet exist (--check did not result in repeated builds), AND - The result path is not available from any configured substituters, AND - No remote builders that can build the path are available. If any of these do not hold, a derivation would be built 0 or 1 times regardless of the repeat option. Thus, remove it to avoid confusion. --- doc/manual/redirects.js | 3 - doc/manual/src/advanced-topics/diff-hook.md | 34 ----------- doc/manual/src/command-ref/nix-store.md | 4 -- doc/manual/src/release-notes/rl-next.md | 3 + src/libmain/progress-bar.cc | 10 ++-- src/libstore/build/derivation-goal.cc | 17 +----- src/libstore/build/derivation-goal.hh | 5 -- src/libstore/build/local-derivation-goal.cc | 62 +-------------------- src/libstore/daemon.cc | 1 - src/libstore/globals.hh | 22 -------- src/libstore/legacy-ssh-store.cc | 4 +- src/nix-store/nix-store.cc | 9 ++- 12 files changed, 21 insertions(+), 153 deletions(-) diff --git a/doc/manual/redirects.js b/doc/manual/redirects.js index 2e77edd0f..69f75d3a0 100644 --- a/doc/manual/redirects.js +++ b/doc/manual/redirects.js @@ -35,7 +35,6 @@ const redirects = { "conf-build-max-jobs": "command-ref/conf-file.html#conf-build-max-jobs", "conf-build-max-log-size": "command-ref/conf-file.html#conf-build-max-log-size", "conf-build-max-silent-time": "command-ref/conf-file.html#conf-build-max-silent-time", - "conf-build-repeat": "command-ref/conf-file.html#conf-build-repeat", "conf-build-timeout": "command-ref/conf-file.html#conf-build-timeout", "conf-build-use-chroot": "command-ref/conf-file.html#conf-build-use-chroot", "conf-build-use-sandbox": "command-ref/conf-file.html#conf-build-use-sandbox", @@ -47,7 +46,6 @@ const redirects = { "conf-connect-timeout": "command-ref/conf-file.html#conf-connect-timeout", "conf-cores": "command-ref/conf-file.html#conf-cores", "conf-diff-hook": "command-ref/conf-file.html#conf-diff-hook", - "conf-enforce-determinism": "command-ref/conf-file.html#conf-enforce-determinism", "conf-env-keep-derivations": "command-ref/conf-file.html#conf-env-keep-derivations", "conf-extra-binary-caches": "command-ref/conf-file.html#conf-extra-binary-caches", "conf-extra-platforms": "command-ref/conf-file.html#conf-extra-platforms", @@ -74,7 +72,6 @@ const redirects = { "conf-plugin-files": "command-ref/conf-file.html#conf-plugin-files", "conf-post-build-hook": "command-ref/conf-file.html#conf-post-build-hook", "conf-pre-build-hook": "command-ref/conf-file.html#conf-pre-build-hook", - "conf-repeat": "command-ref/conf-file.html#conf-repeat", "conf-require-sigs": "command-ref/conf-file.html#conf-require-sigs", "conf-restrict-eval": "command-ref/conf-file.html#conf-restrict-eval", "conf-run-diff-hook": "command-ref/conf-file.html#conf-run-diff-hook", diff --git a/doc/manual/src/advanced-topics/diff-hook.md b/doc/manual/src/advanced-topics/diff-hook.md index 161e64b2a..4a742c160 100644 --- a/doc/manual/src/advanced-topics/diff-hook.md +++ b/doc/manual/src/advanced-topics/diff-hook.md @@ -121,37 +121,3 @@ error: are not valid, so checking is not possible Run the build without `--check`, and then try with `--check` again. - -# Automatic and Optionally Enforced Determinism Verification - -Automatically verify every build at build time by executing the build -multiple times. - -Setting `repeat` and `enforce-determinism` in your `nix.conf` permits -the automated verification of every build Nix performs. - -The following configuration will run each build three times, and will -require the build to be deterministic: - - enforce-determinism = true - repeat = 2 - -Setting `enforce-determinism` to false as in the following -configuration will run the build multiple times, execute the build -hook, but will allow the build to succeed even if it does not build -reproducibly: - - enforce-determinism = false - repeat = 1 - -An example output of this configuration: - -```console -$ nix-build ./test.nix -A unstable -this derivation will be built: - /nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv -building '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' (round 1/2)... -building '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' (round 2/2)... -output '/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable' of '/nix/store/ch6llwpr2h8c3jmnf3f2ghkhx59aa97f-unstable.drv' differs from '/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable.check' from previous round -/nix/store/6xg356v9gl03hpbbg8gws77n19qanh02-unstable -``` diff --git a/doc/manual/src/command-ref/nix-store.md b/doc/manual/src/command-ref/nix-store.md index 1251888e9..9102aff2d 100644 --- a/doc/manual/src/command-ref/nix-store.md +++ b/doc/manual/src/command-ref/nix-store.md @@ -104,10 +104,6 @@ The following flags are available: previous build, the new output path is left in `/nix/store/name.check.` - See also the `build-repeat` configuration option, which repeats a - derivation a number of times and prevents its outputs from being - registered as “valid” in the Nix store unless they are identical. - Special exit codes: - `100`\ diff --git a/doc/manual/src/release-notes/rl-next.md b/doc/manual/src/release-notes/rl-next.md index bf51aa1f7..faead26c2 100644 --- a/doc/manual/src/release-notes/rl-next.md +++ b/doc/manual/src/release-notes/rl-next.md @@ -46,3 +46,6 @@ * `nix build --json` now prints some statistics about top-level derivations, such as CPU statistics when cgroups are enabled. + +* The `repeat` and `enforce-determinism` options have been removed + since they had been broken under many circumstances for a long time. diff --git a/src/libmain/progress-bar.cc b/src/libmain/progress-bar.cc index 961f4e18a..d160a83e9 100644 --- a/src/libmain/progress-bar.cc +++ b/src/libmain/progress-bar.cc @@ -180,10 +180,12 @@ public: auto machineName = getS(fields, 1); if (machineName != "") i->s += fmt(" on " ANSI_BOLD "%s" ANSI_NORMAL, machineName); - auto curRound = getI(fields, 2); - auto nrRounds = getI(fields, 3); - if (nrRounds != 1) - i->s += fmt(" (round %d/%d)", curRound, nrRounds); + + // Used to be curRound and nrRounds, but the + // implementation was broken for a long time. + if (getI(fields, 2) != 1 || getI(fields, 3) != 1) { + throw Error("log message indicated repeating builds, but this is not currently implemented"); + } i->name = DrvName(name).name; } diff --git a/src/libstore/build/derivation-goal.cc b/src/libstore/build/derivation-goal.cc index 5aed51bcd..98f9d681a 100644 --- a/src/libstore/build/derivation-goal.cc +++ b/src/libstore/build/derivation-goal.cc @@ -571,10 +571,6 @@ void DerivationGoal::inputsRealised() /* What type of derivation are we building? */ derivationType = drv->type(); - /* Don't repeat fixed-output derivations since they're already - verified by their output hash.*/ - nrRounds = derivationType.isFixed() ? 1 : settings.buildRepeat + 1; - /* Okay, try to build. Note that here we don't wait for a build slot to become available, since we don't need one if there is a build hook. */ @@ -589,12 +585,11 @@ void DerivationGoal::started() auto msg = fmt( buildMode == bmRepair ? "repairing outputs of '%s'" : buildMode == bmCheck ? "checking outputs of '%s'" : - nrRounds > 1 ? "building '%s' (round %d/%d)" : - "building '%s'", worker.store.printStorePath(drvPath), curRound, nrRounds); + "building '%s'", worker.store.printStorePath(drvPath)); fmt("building '%s'", worker.store.printStorePath(drvPath)); if (hook) msg += fmt(" on '%s'", machineName); act = std::make_unique(*logger, lvlInfo, actBuild, msg, - Logger::Fields{worker.store.printStorePath(drvPath), hook ? machineName : "", curRound, nrRounds}); + Logger::Fields{worker.store.printStorePath(drvPath), hook ? machineName : "", 1, 1}); mcRunningBuilds = std::make_unique>(worker.runningBuilds); worker.updateProgress(); } @@ -948,14 +943,6 @@ void DerivationGoal::buildDone() cleanupPostOutputsRegisteredModeNonCheck(); - /* Repeat the build if necessary. */ - if (curRound++ < nrRounds) { - outputLocks.unlock(); - state = &DerivationGoal::tryToBuild; - worker.wakeUp(shared_from_this()); - return; - } - /* It is now safe to delete the lock files, since all future lockers will see that the output paths are valid; they will not create new lock files with the same names as the old diff --git a/src/libstore/build/derivation-goal.hh b/src/libstore/build/derivation-goal.hh index 2d8bfd592..d33e04cbc 100644 --- a/src/libstore/build/derivation-goal.hh +++ b/src/libstore/build/derivation-goal.hh @@ -115,11 +115,6 @@ struct DerivationGoal : public Goal BuildMode buildMode; - /* The current round, if we're building multiple times. */ - size_t curRound = 1; - - size_t nrRounds; - std::unique_ptr> mcExpectedBuilds, mcRunningBuilds; std::unique_ptr act; diff --git a/src/libstore/build/local-derivation-goal.cc b/src/libstore/build/local-derivation-goal.cc index d2798888b..6fe3bc49c 100644 --- a/src/libstore/build/local-derivation-goal.cc +++ b/src/libstore/build/local-derivation-goal.cc @@ -2260,7 +2260,6 @@ DrvOutputs LocalDerivationGoal::registerOutputs() InodesSeen inodesSeen; Path checkSuffix = ".check"; - bool keepPreviousRound = settings.keepFailed || settings.runDiffHook; std::exception_ptr delayedException; @@ -2688,10 +2687,8 @@ DrvOutputs LocalDerivationGoal::registerOutputs() debug("unreferenced input: '%1%'", worker.store.printStorePath(i)); } - if (curRound == nrRounds) { - localStore.optimisePath(actualPath, NoRepair); // FIXME: combine with scanForReferences() - worker.markContentsGood(newInfo.path); - } + localStore.optimisePath(actualPath, NoRepair); // FIXME: combine with scanForReferences() + worker.markContentsGood(newInfo.path); newInfo.deriver = drvPath; newInfo.ultimate = true; @@ -2720,61 +2717,6 @@ DrvOutputs LocalDerivationGoal::registerOutputs() /* Apply output checks. */ checkOutputs(infos); - /* Compare the result with the previous round, and report which - path is different, if any.*/ - if (curRound > 1 && prevInfos != infos) { - assert(prevInfos.size() == infos.size()); - for (auto i = prevInfos.begin(), j = infos.begin(); i != prevInfos.end(); ++i, ++j) - if (!(*i == *j)) { - buildResult.isNonDeterministic = true; - Path prev = worker.store.printStorePath(i->second.path) + checkSuffix; - bool prevExists = keepPreviousRound && pathExists(prev); - hintformat hint = prevExists - ? hintfmt("output '%s' of '%s' differs from '%s' from previous round", - worker.store.printStorePath(i->second.path), worker.store.printStorePath(drvPath), prev) - : hintfmt("output '%s' of '%s' differs from previous round", - worker.store.printStorePath(i->second.path), worker.store.printStorePath(drvPath)); - - handleDiffHook( - buildUser ? buildUser->getUID() : getuid(), - buildUser ? buildUser->getGID() : getgid(), - prev, worker.store.printStorePath(i->second.path), - worker.store.printStorePath(drvPath), tmpDir); - - if (settings.enforceDeterminism) - throw NotDeterministic(hint); - - printError(hint); - - curRound = nrRounds; // we know enough, bail out early - } - } - - /* If this is the first round of several, then move the output out of the way. */ - if (nrRounds > 1 && curRound == 1 && curRound < nrRounds && keepPreviousRound) { - for (auto & [_, outputStorePath] : finalOutputs) { - auto path = worker.store.printStorePath(outputStorePath); - Path prev = path + checkSuffix; - deletePath(prev); - Path dst = path + checkSuffix; - renameFile(path, dst); - } - } - - if (curRound < nrRounds) { - prevInfos = std::move(infos); - return {}; - } - - /* Remove the .check directories if we're done. FIXME: keep them - if the result was not determistic? */ - if (curRound == nrRounds) { - for (auto & [_, outputStorePath] : finalOutputs) { - Path prev = worker.store.printStorePath(outputStorePath) + checkSuffix; - deletePath(prev); - } - } - /* Register each output path as valid, and register the sets of paths referenced by each of them. If there are cycles in the outputs, this will fail. */ diff --git a/src/libstore/daemon.cc b/src/libstore/daemon.cc index 48dd5c247..12596ba49 100644 --- a/src/libstore/daemon.cc +++ b/src/libstore/daemon.cc @@ -238,7 +238,6 @@ struct ClientSettings } else if (trusted || name == settings.buildTimeout.name - || name == settings.buildRepeat.name || name == settings.maxSilentTime.name || name == settings.pollInterval.name || name == "connect-timeout" diff --git a/src/libstore/globals.hh b/src/libstore/globals.hh index ca72ad31e..54a5d0fc7 100644 --- a/src/libstore/globals.hh +++ b/src/libstore/globals.hh @@ -373,11 +373,6 @@ public: )", {"build-max-log-size"}}; - /* When buildRepeat > 0 and verboseBuild == true, whether to print - repeated builds (i.e. builds other than the first one) to - stderr. Hack to prevent Hydra logs from being polluted. */ - bool printRepeatedBuilds = true; - Setting pollInterval{this, 5, "build-poll-interval", "How often (in seconds) to poll for locks."}; @@ -501,19 +496,6 @@ public: Setting sandboxFallback{this, true, "sandbox-fallback", "Whether to disable sandboxing when the kernel doesn't allow it."}; - Setting buildRepeat{ - this, 0, "repeat", - R"( - How many times to repeat builds to check whether they are - deterministic. The default value is 0. If the value is non-zero, - every build is repeated the specified number of times. If the - contents of any of the runs differs from the previous ones and - `enforce-determinism` is true, the build is rejected and the - resulting store paths are not registered as “valid” in Nix’s - database. - )", - {"build-repeat"}}; - #if __linux__ Setting sandboxShmSize{ this, "50%", "sandbox-dev-shm-size", @@ -577,10 +559,6 @@ public: configuration file, and cannot be passed at the command line. )"}; - Setting enforceDeterminism{ - this, true, "enforce-determinism", - "Whether to fail if repeated builds produce different output. See `repeat`."}; - Setting trustedPublicKeys{ this, {"cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="}, diff --git a/src/libstore/legacy-ssh-store.cc b/src/libstore/legacy-ssh-store.cc index dd34b19c6..4d398b21d 100644 --- a/src/libstore/legacy-ssh-store.cc +++ b/src/libstore/legacy-ssh-store.cc @@ -255,8 +255,8 @@ private: << settings.maxLogSize; if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 3) conn.to - << settings.buildRepeat - << settings.enforceDeterminism; + << 0 // buildRepeat hasn't worked for ages anyway + << 0; if (GET_PROTOCOL_MINOR(conn.remoteVersion) >= 7) { conn.to << ((int) settings.keepFailed); diff --git a/src/nix-store/nix-store.cc b/src/nix-store/nix-store.cc index b59a6d026..b854ef1e7 100644 --- a/src/nix-store/nix-store.cc +++ b/src/nix-store/nix-store.cc @@ -808,14 +808,17 @@ static void opServe(Strings opFlags, Strings opArgs) if (GET_PROTOCOL_MINOR(clientVersion) >= 2) settings.maxLogSize = readNum(in); if (GET_PROTOCOL_MINOR(clientVersion) >= 3) { - settings.buildRepeat = readInt(in); - settings.enforceDeterminism = readInt(in); + if (readInt(in) != 0) { + throw Error("client requested repeating builds, but this is not currently implemented"); + } + if (readInt(in) != 0) { + throw Error("client requested enforcing determinism, but this is not currently implemented"); + } settings.runDiffHook = true; } if (GET_PROTOCOL_MINOR(clientVersion) >= 7) { settings.keepFailed = (bool) readInt(in); } - settings.printRepeatedBuilds = false; }; while (true) { From cc27bd06339283a8189de5c4b1a981bd9c85c211 Mon Sep 17 00:00:00 2001 From: Robert Hensing Date: Tue, 6 Dec 2022 18:00:10 +0100 Subject: [PATCH 11/15] .version: Add newline --- .version | 2 +- flake.nix | 7 ++++--- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/.version b/.version index a3ebb9f51..fb2c0766b 100644 --- a/.version +++ b/.version @@ -1 +1 @@ -2.13.0 \ No newline at end of file +2.13.0 diff --git a/flake.nix b/flake.nix index d9d01da10..38b469fdb 100644 --- a/flake.nix +++ b/flake.nix @@ -9,14 +9,15 @@ let - version = builtins.readFile ./.version + versionSuffix; + officialRelease = false; + + version = strip (builtins.readFile ./.version) + versionSuffix; + strip = nixpkgs.lib.removeSuffix "\n"; versionSuffix = if officialRelease then "" else "pre${builtins.substring 0 8 (self.lastModifiedDate or self.lastModified or "19700101")}_${self.shortRev or "dirty"}"; - officialRelease = false; - linux64BitSystems = [ "x86_64-linux" "aarch64-linux" ]; linuxSystems = linux64BitSystems ++ [ "i686-linux" ]; systems = linuxSystems ++ [ "x86_64-darwin" "aarch64-darwin" ]; From c710aa1abd51231ce45fc419de2e0e82c8957fbd Mon Sep 17 00:00:00 2001 From: endgame Date: Wed, 7 Dec 2022 22:55:02 +1000 Subject: [PATCH 12/15] Post build hook signing (#7408) * docs: Use secret-key-files when demonstrating post-build-hooks The docs used to recommend calling `nix store sign` in a post-build hook, but on more recent versions of nix, this results in unsigned store paths being copied into binary caches. See https://github.com/NixOS/nix/issues/6960 for details. Instead, use the `secret-key-files` config option, which signs all locally-built derivations with the private key. Co-authored-by: Valentin Gagarin --- doc/manual/src/advanced-topics/post-build-hook.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/doc/manual/src/advanced-topics/post-build-hook.md b/doc/manual/src/advanced-topics/post-build-hook.md index fcb52d878..1479cc3a4 100644 --- a/doc/manual/src/advanced-topics/post-build-hook.md +++ b/doc/manual/src/advanced-topics/post-build-hook.md @@ -33,12 +33,17 @@ distribute the public key for verifying the authenticity of the paths. example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM= ``` -Then, add the public key and the cache URL to your `nix.conf`'s -`trusted-public-keys` and `substituters` options: +Then update [`nix.conf`](../command-ref/conf-file.md) on any machine that will access the cache. +Add the cache URL to [`substituters`](../command-ref/conf-file.md#conf-substituters) and the public key to [`trusted-public-keys`](../command-ref/conf-file.md#conf-trusted-public-keys): substituters = https://cache.nixos.org/ s3://example-nix-cache trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM= +Machines that build for the cache must sign derivations using the private key. +On those machines, add the path to the key file to the [`secret-key-files`](../command-ref/conf-file.md#conf-secret-key-files) field in their [`nix.conf`](../command-ref/conf-file.md): + + secret-key-files = /etc/nix/key.private + We will restart the Nix daemon in a later step. # Implementing the build hook @@ -52,14 +57,12 @@ set -eu set -f # disable globbing export IFS=' ' -echo "Signing paths" $OUT_PATHS -nix store sign --key-file /etc/nix/key.private $OUT_PATHS echo "Uploading paths" $OUT_PATHS -exec nix copy --to 's3://example-nix-cache' $OUT_PATHS +exec nix copy --to "s3://example-nix-cache" $OUT_PATHS ``` > **Note** -> +> > The `$OUT_PATHS` variable is a space-separated list of Nix store > paths. In this case, we expect and want the shell to perform word > splitting to make each output path its own argument to `nix From af8136afd4440bbabb4ea214bfd3dc15ffebf469 Mon Sep 17 00:00:00 2001 From: Robert Hensing Date: Wed, 7 Dec 2022 13:55:31 +0100 Subject: [PATCH 13/15] flake.nix: Use nixpkgs.lib.fileContents --- flake.nix | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/flake.nix b/flake.nix index 38b469fdb..5127ee2a3 100644 --- a/flake.nix +++ b/flake.nix @@ -11,8 +11,7 @@ officialRelease = false; - version = strip (builtins.readFile ./.version) + versionSuffix; - strip = nixpkgs.lib.removeSuffix "\n"; + version = nixpkgs.lib.fileContents ./.version + versionSuffix; versionSuffix = if officialRelease then "" From b8a1ff98c1019f36a6425f639bf4c694f42c7edf Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 7 Dec 2022 16:12:26 +0100 Subject: [PATCH 14/15] use HTML anchors for config parameters this avoids incorrect rendering on the man pages, since `lowdown` neither parses the anchor syntax nor HTML. this should rather be fixed in lowdown, as adding more anchors would otherwise produce ever more noise and error-prone repetition. --- doc/manual/generate-options.nix | 2 +- doc/manual/src/command-ref/nix-build.md | 12 +++++++----- doc/manual/src/command-ref/nix-store.md | 3 ++- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/doc/manual/generate-options.nix b/doc/manual/generate-options.nix index 814144c20..a4ec36477 100644 --- a/doc/manual/generate-options.nix +++ b/doc/manual/generate-options.nix @@ -9,7 +9,7 @@ let let inherit (optionsInfo.${name}) description documentDefault defaultValue aliases; result = squash '' - - [`${name}`]{#conf-${name}} + - [`${name}`](#conf-${name}) ${indent " " body} ''; diff --git a/doc/manual/src/command-ref/nix-build.md b/doc/manual/src/command-ref/nix-build.md index 49c6f3f55..3a47feaae 100644 --- a/doc/manual/src/command-ref/nix-build.md +++ b/doc/manual/src/command-ref/nix-build.md @@ -53,16 +53,18 @@ All options not listed here are passed to `nix-store --realise`, except for `--arg` and `--attr` / `-A` which are passed to `nix-instantiate`. - - [`--no-out-link`]{#opt-no-out-link}\ + - [`--no-out-link`](#opt-no-out-link) + Do not create a symlink to the output path. Note that as a result the output does not become a root of the garbage collector, and so - might be deleted by `nix-store - --gc`. + might be deleted by `nix-store --gc`. + + - [`--dry-run`](#opt-dry-run) - - [`--dry-run`]{#opt-dry-run}\ Show what store paths would be built or downloaded. - - [`--out-link`]{#opt-out-link} / `-o` *outlink*\ + - [`--out-link`](#opt-out-link) / `-o` *outlink* + Change the name of the symlink to the output path created from `result` to *outlink*. diff --git a/doc/manual/src/command-ref/nix-store.md b/doc/manual/src/command-ref/nix-store.md index 1251888e9..ec1da72a4 100644 --- a/doc/manual/src/command-ref/nix-store.md +++ b/doc/manual/src/command-ref/nix-store.md @@ -22,7 +22,8 @@ This section lists the options that are common to all operations. These options are allowed for every subcommand, though they may not always have an effect. - - [`--add-root`]{#opt-add-root} *path*\ + - [`--add-root`](#opt-add-root) *path* + Causes the result of a realisation (`--realise` and `--force-realise`) to be registered as a root of the garbage collector. *path* will be created as a symlink to the resulting From ebeaf03558caa62cba6f0bfbd1170dbd5c5944b8 Mon Sep 17 00:00:00 2001 From: Valentin Gagarin Date: Wed, 7 Dec 2022 16:20:25 +0100 Subject: [PATCH 15/15] do not render links in man pages this is a follow-up on e7dcacb. most links are relative and this should not be too much of a detriment. --- doc/manual/local.mk | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/manual/local.mk b/doc/manual/local.mk index 486dbd7a2..c0f69e00f 100644 --- a/doc/manual/local.mk +++ b/doc/manual/local.mk @@ -29,19 +29,19 @@ nix-eval = $(dummy-env) $(bindir)/nix eval --experimental-features nix-command - $(d)/%.1: $(d)/src/command-ref/%.md @printf "Title: %s\n\n" "$$(basename $@ .1)" > $^.tmp @cat $^ >> $^.tmp - $(trace-gen) lowdown -sT man -M section=1 $^.tmp -o $@ + $(trace-gen) lowdown -sT man --nroff-nolinks -M section=1 $^.tmp -o $@ @rm $^.tmp $(d)/%.8: $(d)/src/command-ref/%.md @printf "Title: %s\n\n" "$$(basename $@ .8)" > $^.tmp @cat $^ >> $^.tmp - $(trace-gen) lowdown -sT man -M section=8 $^.tmp -o $@ + $(trace-gen) lowdown -sT man --nroff-nolinks -M section=8 $^.tmp -o $@ @rm $^.tmp $(d)/nix.conf.5: $(d)/src/command-ref/conf-file.md @printf "Title: %s\n\n" "$$(basename $@ .5)" > $^.tmp @cat $^ >> $^.tmp - $(trace-gen) lowdown -sT man -M section=5 $^.tmp -o $@ + $(trace-gen) lowdown -sT man --nroff-nolinks -M section=5 $^.tmp -o $@ @rm $^.tmp $(d)/src/SUMMARY.md: $(d)/src/SUMMARY.md.in $(d)/src/command-ref/new-cli