From 86a3fad0856a1755a128c645057c69d2b11b6bec Mon Sep 17 00:00:00 2001
From: Sergei Zimmerman <xokdvium@proton.me>
Date: Thu, 1 May 2025 23:10:04 +0000
Subject: [PATCH] libexpr: Improve lexer performance by using full scanner
 tables (-Cf)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This trades off some executable size for measurable lexer performance
improvements.

Note on the explicitly enabling 8bit scanner.
This is needed due to the default behavior of flex (excerpt from the manual [1]):

> Flex’s default behavior is to generate an 8-bit scanner unless you
> use the ‘-Cf’ or ‘-CF’, in which case flex defaults to generating
> 7-bit scanners unless your site was always configured to generate 8-bit
> scanners.

Some quantifyable metrics:

Nixpkgs revision: a6e3f45acf4e817532a861ab0eda4ab5485fecc1
Parsing the largest file in nixpkgs: pkgs/development/haskell-modules/hackage-packages.nix.

(Before this patch)

```
$ nix build github:nixos/nix/9fe3077d4#nix-expr
$ du --apparent-size result/lib/libnixexpr.so
2518    result/lib/libnixexpr.so
$ nix build github:nixos/nix/9fe3077d4#nix-cli
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g \
    result/bin/nix-instantiate --parse               \
    ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix > /dev/null"
  Time (mean ± σ):     375.5 ms ±   6.3 ms    [User: 316.9 ms, System: 56.7 ms]
  Range (min … max):   368.5 ms … 388.3 ms    10 runs
```

(After the patch)

```
$ nix build .#nix-expr
$ du --apparent-size result/lib/libnixexpr.so
2685    result/lib/libnixexpr.so
$ nix build .#nix-cli
$ taskset -c 2,3 hyperfine "GC_INITIAL_HEAP_SIZE=16g \
    result/bin/nix-instantiate --parse               \
    ../nixpkgs/pkgs/development/haskell-modules/hackage-packages.nix > /dev/null"
  Time (mean ± σ):     326.8 ms ±   4.9 ms    [User: 269.5 ms, System: 55.3 ms]
  Range (min … max):   319.7 ms … 335.5 ms    10 runs
```

Overall, the change is roughly:

- 2518KiB -> 2685KiB ~ 150 KiB of machine code
- 375ms -> 325ms ~ 50ms

The perf uplift for eval-heavy test cases is obviously less noticeable,
but it doesn't make sense not to take this free perf win.

[1]: https://westes.github.io/flex/manual/Options-Affecting-Scanner-Behavior.html#Options-Affecting-Scanner-Behavior
---
 src/libexpr/lexer.l     | 1 +
 src/libexpr/meson.build | 1 +
 2 files changed, 2 insertions(+)

diff --git a/src/libexpr/lexer.l b/src/libexpr/lexer.l
index 1e196741d..1005f9f7e 100644
--- a/src/libexpr/lexer.l
+++ b/src/libexpr/lexer.l
@@ -1,3 +1,4 @@
+%option 8bit
 %option reentrant bison-bridge bison-locations
 %option align
 %option noyywrap
diff --git a/src/libexpr/meson.build b/src/libexpr/meson.build
index 2e773938d..2b465b85a 100644
--- a/src/libexpr/meson.build
+++ b/src/libexpr/meson.build
@@ -112,6 +112,7 @@ lexer_tab = custom_target(
   ],
   command : [
     'flex',
+    '-Cf', # Use full scanner tables
     '--outfile',
     '@OUTPUT0@',
     '--header-file=' + '@OUTPUT1@',