I can't find a good way to benchmark in isolation from the
git cache, but common sense dictates that creating (and destroying)
a 131KiB std::vector for each regular file from the archive imposes
quite a significant overhead regardless of the IO bound git cache.
AFAICT there is no reason to keep a copy of the data since
it always gets fed into the sink and there are no coroutines/threads
in sight.
warning: installing Nix as root is not supported by this script!
performing a single-user installation of Nix...
copying Nix to /nix/store.......................................................
......
warning: the group 'nixbld' specified in 'build-users-group' does not exist
warning: the group 'nixbld' specified in 'build-users-group' does not exist
installing 'nix-2.28.3'
error: the group 'nixbld' specified in 'build-users-group' does not exist
/tmp/nix-binary-tarball-unpack.2j3lCU0A89/unpack/nix-2.28.3-x86_64-linux/install: unable to install Nix into your default profile
The nixos/nix docker image is built using `buildLayeredImage`, which spreads the nix store over a configured number of layers. This number was set to create an image with 100 layers. Because there is a limit of (typically) 127 layers in AUFS, this only left 27 layers to build on top. At the same time, nearly half of the created layers were only <100kb in size, many even <10kb, negating the intended advantage in cachability.
This commit moves the tradeoff a bit by reducing the number of layers to 70.
Layer sizes for the 2.28.3 nixos/nix image: https://hub.docker.com/layers/nixos/nix/2.28.3/images/sha256-d078d7153763895fce17c5fbbdeb86fcfcac414ca0ba875d413c1df57be19931
As it turns out using `std::regex` is actually the bottleneck
for root discovery. Just substituting `std::` -> `boost::`
makes root discovery twice as fast (3x if counting only userspace time).
Some rather ad-hoc measurements to motivate the switch:
(On master)
```
nix build github:nixos/nix/1e822bd4149a8bce1da81ee2ad9404986b07914c#nix-cli --out-link result-1e822bd4149a8bce1da81ee2ad9404986b07914c
taskset -c 2,3 hyperfine "result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0"
Benchmark 1: result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run --max 0
Time (mean ± σ): 481.6 ms ± 3.9 ms [User: 336.2 ms, System: 142.0 ms]
Range (min … max): 474.6 ms … 487.7 ms 10 runs
```
(After this patch)
```
taskset -c 2,3 hyperfine "result/bin/nix store gc --dry-run --max 0"
Benchmark 1: result/bin/nix store gc --dry-run --max 0
Time (mean ± σ): 254.7 ms ± 9.7 ms [User: 111.1 ms, System: 141.3 ms]
Range (min … max): 246.5 ms … 281.3 ms 10 runs
```
`boost::regex` is a drop-in replacement for `std::regex`, but much faster.
Doing a simple before/after comparison doesn't surface any change in behavior:
```
result/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
result-1e822bd4149a8bce1da81ee2ad9404986b07914c/bin/nix store gc --dry-run -vvvvv --max 0 |& grep "got additional" | wc -l
```
