wroclaw-git-backups/nix
1
0
Fork 0
mirror of https://github.com/NixOS/nix synced 2025-07-07 10:11:47 +02:00
Code Activity

Rename doc/manual{src -> source}

Browse source 
This is needed to avoid this
https://github.com/mesonbuild/meson/issues/13774 when we go back to
making our subproject directory `src`.
This commit is contained in:
John Ericson 2024-10-10 12:04:33 -04:00
parent d5c45952ac
commit eb7d7780b1
221 changed files with 75 additions and 74 deletions
Download patch file Download diff file Expand all files Collapse all files

39
doc/manual/source/advanced-topics/cores-vs-jobs.md Normal file
View file

@ -0,0 +1,39 @@
# Tuning Cores and Jobs
Nix has two relevant settings with regards to how your CPU cores will
be utilized: `cores` and `max-jobs`. This chapter will talk about what
they are, how they interact, and their configuration trade-offs.
- `max-jobs`\
Dictates how many separate derivations will be built at the same
time. If you set this to zero, the local machine will do no
builds. Nix will still substitute from binary caches, and build
remotely if remote builders are configured.
- `cores`\
Suggests how many cores each derivation should use. Similar to
`make -j`.
The `cores` setting determines the value of
`NIX_BUILD_CORES`. `NIX_BUILD_CORES` is equal to `cores`, unless
`cores` equals `0`, in which case `NIX_BUILD_CORES` will be the total
number of cores in the system.
The maximum number of consumed cores is a simple multiplication,
`max-jobs` \* `NIX_BUILD_CORES`.
The balance on how to set these two independent variables depends upon
each builder's workload and hardware. Here are a few example scenarios
on a machine with 24 cores:
| `max-jobs` | `cores` | `NIX_BUILD_CORES` | Maximum Processes | Result |
| --------------------- | ------------------ | ----------------- | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | 24 | 24 | 24 | One derivation will be built at a time, each one can use 24 cores. Undersold if a job can’t use 24 cores. |
| 4 | 6 | 6 | 24 | Four derivations will be built at once, each given access to six cores. |
| 12 | 6 | 6 | 72 | 12 derivations will be built at once, each given access to six cores. This configuration is over-sold. If all 12 derivations being built simultaneously try to use all six cores, the machine's performance will be degraded due to extensive context switching between the 12 builds. |
| 24 | 1 | 1 | 24 | 24 derivations can build at the same time, each using a single core. Never oversold, but derivations which require many cores will be very slow to compile. |
| 24 | 0 | 24 | 576 | 24 derivations can build at the same time, each using all the available cores of the machine. Very likely to be oversold, and very likely to suffer context switches. |
It is up to the derivations' build script to respect host's requested
cores-per-build by following the value of the `NIX_BUILD_CORES`
environment variable.

123
doc/manual/source/advanced-topics/diff-hook.md Normal file
View file

@ -0,0 +1,123 @@
# Verifying Build Reproducibility
You can use Nix's `diff-hook` setting to compare build results. Note
that this hook is only executed if the results differ; it is not used
for determining if the results are the same.
For purposes of demonstration, we'll use the following Nix file,
`deterministic.nix` for testing:
```nix
let
inherit (import <nixpkgs> {}) runCommand;
in {
stable = runCommand "stable" {} ''
touch $out
'';
unstable = runCommand "unstable" {} ''
echo $RANDOM > $out
'';
}
```
Additionally, `nix.conf` contains:
diff-hook = /etc/nix/my-diff-hook
run-diff-hook = true
where `/etc/nix/my-diff-hook` is an executable file containing:
```bash
#!/bin/sh
exec >&2
echo "For derivation $3:"
/run/current-system/sw/bin/diff -r "$1" "$2"
```
The diff hook is executed by the same user and group who ran the build.
However, the diff hook does not have write access to the store path just
built.
# Spot-Checking Build Determinism
Verify a path which already exists in the Nix store by passing `--check`
to the build command.
If the build passes and is deterministic, Nix will exit with a status
code of 0:
```console
$ nix-build ./deterministic.nix --attr stable
this derivation will be built:
/nix/store/z98fasz2jqy9gs0xbvdj939p27jwda38-stable.drv
building '/nix/store/z98fasz2jqy9gs0xbvdj939p27jwda38-stable.drv'...
/nix/store/yyxlzw3vqaas7wfp04g0b1xg51f2czgq-stable
$ nix-build ./deterministic.nix --attr stable --check
checking outputs of '/nix/store/z98fasz2jqy9gs0xbvdj939p27jwda38-stable.drv'...
/nix/store/yyxlzw3vqaas7wfp04g0b1xg51f2czgq-stable
```
If the build is not deterministic, Nix will exit with a status code of
1:
```console
$ nix-build ./deterministic.nix --attr unstable
this derivation will be built:
/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv
building '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv'...
/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable
$ nix-build ./deterministic.nix --attr unstable --check
checking outputs of '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv'...
error: derivation '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv' may
not be deterministic: output '/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable' differs
```
In the Nix daemon's log, we will now see:
```
For derivation /nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv:
1c1
< 8108
---
> 30204
```
Using `--check` with `--keep-failed` will cause Nix to keep the second
build's output in a special, `.check` path:
```console
$ nix-build ./deterministic.nix --attr unstable --check --keep-failed
checking outputs of '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv'...
note: keeping build directory '/tmp/nix-build-unstable.drv-0'
error: derivation '/nix/store/cgl13lbj1w368r5z8gywipl1ifli7dhk-unstable.drv' may
not be deterministic: output '/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable' differs
from '/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable.check'
```
In particular, notice the
`/nix/store/krpqk0l9ib0ibi1d2w52z293zw455cap-unstable.check` output. Nix
has copied the build results to that directory where you can examine it.
> []{#check-dirs-are-unregistered} **Note**
>
> Check paths are not protected against garbage collection, and this
> path will be deleted on the next garbage collection.
>
> The path is guaranteed to be alive for the duration of
> the `diff-hook`'s execution, but may be deleted any time after.
>
> If the comparison is performed as part of automated tooling, please
> use the diff-hook or author your tooling to handle the case where the
> build was not deterministic and also a check path does not exist.
`--check` is only usable if the derivation has been built on the system
already. If the derivation has not been built Nix will fail with the
error:
error: some outputs of '/nix/store/hzi1h60z2qf0nb85iwnpvrai3j2w7rr6-unstable.drv'
are not valid, so checking is not possible
Run the build without `--check`, and then try with `--check` again.

71
doc/manual/source/advanced-topics/distributed-builds.md Normal file
View file

@ -0,0 +1,71 @@
# Remote Builds
Nix supports remote builds, where a local Nix installation can forward
Nix builds to other machines. This allows multiple builds to be
performed in parallel and allows Nix to perform multi-platform builds in
a semi-transparent way. For instance, if you perform a build for a
`x86_64-darwin` on an `i686-linux` machine, Nix can automatically
forward the build to a `x86_64-darwin` machine, if available.
To forward a build to a remote machine, it’s required that the remote
machine is accessible via SSH and that it has Nix installed. You can
test whether connecting to the remote Nix instance works, e.g.
```console
$ nix store info --store ssh://mac
```
will try to connect to the machine named `mac`. It is possible to
specify an SSH identity file as part of the remote store URI, e.g.
```console
$ nix store info --store ssh://mac?ssh-key=/home/alice/my-key
```
Since builds should be non-interactive, the key should not have a
passphrase. Alternatively, you can load identities ahead of time into
`ssh-agent` or `gpg-agent`.
If you get the error
```console
bash: nix-store: command not found
error: cannot connect to 'mac'
```
then you need to ensure that the `PATH` of non-interactive login shells
contains Nix.
The [list of remote build machines](@docroot@/command-ref/conf-file.md#conf-builders) can be specified on the command line or in the Nix configuration file.
For example, the following command allows you to build a derivation for `x86_64-darwin` on a Linux machine:
```console
$ uname
Linux
$ nix build --impure \
--expr '(with import <nixpkgs> { system = "x86_64-darwin"; }; runCommand "foo" {} "uname > $out")' \
--builders 'ssh://mac x86_64-darwin'
[1/0/1 built, 0.0 MiB DL] building foo on ssh://mac
$ cat ./result
Darwin
```
It is possible to specify multiple build machines separated by a semicolon or a newline, e.g.
```console
--builders 'ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd'
```
Remote build machines can also be configured in [`nix.conf`](@docroot@/command-ref/conf-file.md), e.g.
builders = ssh://mac x86_64-darwin ; ssh://beastie x86_64-freebsd
Finally, remote build machines can be configured in a separate configuration
file included in `builders` via the syntax `@/path/to/file`. For example,
builders = @/etc/nix/machines
causes the list of machines in `/etc/nix/machines` to be included.
(This is the default.)

1
doc/manual/source/advanced-topics/index.md Normal file
View file

@ -0,0 +1 @@
This section lists advanced topics related to builds and builds performance

127
doc/manual/source/advanced-topics/post-build-hook.md Normal file
View file

@ -0,0 +1,127 @@
# Using the `post-build-hook`
# Implementation Caveats
Here we use the post-build hook to upload to a binary cache. This is a
simple and working example, but it is not suitable for all use cases.
The post build hook program runs after each executed build, and blocks
the build loop. The build loop exits if the hook program fails.
Concretely, this implementation will make Nix slow or unusable when the
internet is slow or unreliable.
A more advanced implementation might pass the store paths to a
user-supplied daemon or queue for processing the store paths outside of
the build loop.
# Prerequisites
This tutorial assumes you have configured an [S3-compatible binary cache](@docroot@/command-ref/new-cli/nix3-help-stores.md#s3-binary-cache-store) as a [substituter](../command-ref/conf-file.md#conf-substituters),
and that the `root` user's default AWS profile can upload to the bucket.
# Set up a Signing Key
Use `nix-store --generate-binary-cache-key` to create our public and
private signing keys. We will sign paths with the private key, and
distribute the public key for verifying the authenticity of the paths.
```console
# nix-store --generate-binary-cache-key example-nix-cache-1 /etc/nix/key.private /etc/nix/key.public
# cat /etc/nix/key.public
example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM=
```
Then update [`nix.conf`](../command-ref/conf-file.md) on any machine that will access the cache.
Add the cache URL to [`substituters`](../command-ref/conf-file.md#conf-substituters) and the public key to [`trusted-public-keys`](../command-ref/conf-file.md#conf-trusted-public-keys):
substituters = https://cache.nixos.org/ s3://example-nix-cache
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= example-nix-cache-1:1/cKDz3QCCOmwcztD2eV6Coggp6rqc9DGjWv7C0G+rM=
Machines that build for the cache must sign derivations using the private key.
On those machines, add the path to the key file to the [`secret-key-files`](../command-ref/conf-file.md#conf-secret-key-files) field in their [`nix.conf`](../command-ref/conf-file.md):
secret-key-files = /etc/nix/key.private
We will restart the Nix daemon in a later step.
# Implementing the build hook
Write the following script to `/etc/nix/upload-to-cache.sh`:
```bash
#!/bin/sh
set -eu
set -f # disable globbing
export IFS=' '
echo "Uploading paths" $OUT_PATHS
exec nix copy --to "s3://example-nix-cache" $OUT_PATHS
```
> **Note**
>
> The `$OUT_PATHS` variable is a space-separated list of Nix store
> paths. In this case, we expect and want the shell to perform word
> splitting to make each output path its own argument to `nix
> store sign`. Nix guarantees the paths will not contain any spaces,
> however a store path might contain glob characters. The `set -f`
> disables globbing in the shell.
> If you want to upload the `.drv` file too, the `$DRV_PATH` variable
> is also defined for the script and works just like `$OUT_PATHS`.
Then make sure the hook program is executable by the `root` user:
```console
# chmod +x /etc/nix/upload-to-cache.sh
```
# Updating Nix Configuration
Edit `/etc/nix/nix.conf` to run our hook, by adding the following
configuration snippet at the end:
post-build-hook = /etc/nix/upload-to-cache.sh
Then, restart the `nix-daemon`.
# Testing
Build any derivation, for example:
```console
$ nix-build --expr '(import <nixpkgs> {}).writeText "example" (builtins.toString builtins.currentTime)'
this derivation will be built:
/nix/store/s4pnfbkalzy5qz57qs6yybna8wylkig6-example.drv
building '/nix/store/s4pnfbkalzy5qz57qs6yybna8wylkig6-example.drv'...
running post-build-hook '/home/grahamc/projects/github.com/NixOS/nix/post-hook.sh'...
post-build-hook: Signing paths /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
post-build-hook: Uploading paths /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
/nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
```
Then delete the path from the store, and try substituting it from the
binary cache:
```console
$ rm ./result
$ nix-store --delete /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
```
Now, copy the path back from the cache:
```console
$ nix-store --realise /nix/store/ibcyipq5gf91838ldx40mjsp0b8w9n18-example
copying path '/nix/store/m8bmqwrch6l3h8s0k3d673xpmipcdpsa-example from 's3://example-nix-cache'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/m8bmqwrch6l3h8s0k3d673xpmipcdpsa-example
```
# Conclusion
We now have a Nix installation configured to automatically sign and
upload every local build to a remote binary cache.
Before deploying this to production, be sure to consider the
[implementation caveats](#implementation-caveats).