BUG: fix the kernel version table scripts, and fill in the kernel versions

doc/admin/SYSCALL_KVER_TABLES.md

@@ -0,0 +1,119 @@
+Generating the Syscall Kernel Version Tables
+===============================================================================
+https://github.com/seccomp/libseccomp
+
+The `*_kver` columns in `src/syscalls.csv` record the first kernel version in
+which each syscall was available on each architecture.  They are generated from
+per-kernel-version syscall tables, which are built from the kernel sources using
+the [syscalls-table](https://github.com/hrw/syscalls-table) project.  The syscall
+numbers in `src/syscalls.csv` are not taken from these tables; they are
+maintained using `src/arch-syscall-validate`, which preserves the kernel version
+columns.
+
+This document describes how to regenerate the kernel version columns from
+scratch, and how to update them for new kernel versions.
+
+### Requirements
+
+* A clone of the Linux kernel git repository with the release tags.  The scripts
+  check out each kernel version being processed, so do not use a tree with any
+  work in progress.
+
+		# git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
+
+* A clone of the syscalls-table repository.  Its `data/` directory is modified
+  while the tables are built.
+
+		# git clone https://github.com/hrw/syscalls-table.git
+
+* A toolchain able to build the kernel headers for each kernel version being
+  processed, and the glibc i386 and x32 headers.
+
+Modern toolchains are not able to build the host tools of old kernels; for
+example, kernels v3.0 and v3.15 fail with GCC 16.  Some distributions, e.g.
+Fedora, do not ship the glibc x32 headers.  `src/arch-build-kver-tables.py`
+checks for both problems and fails if it detects any of them.
+
+### Building the Toolchain Container
+
+The easiest way to get a suitable toolchain is to use a container based on an
+older distribution; the one below was tested with kernels v3.0 to v7.2:
+
+	# podman build -t libseccomp-kver - <<'EOF'
+	FROM docker.io/library/ubuntu:20.04
+	RUN apt-get update && \
+	    DEBIAN_FRONTEND=noninteractive apt-get install -y \
+	        --no-install-recommends make gcc libc6-dev libc6-dev-i386 \
+	        libc6-dev-x32 perl rsync python3 git ca-certificates bc && \
+	    rm -rf /var/lib/apt/lists/*
+	EOF
+
+The same command works with Docker, just replace `podman` with `docker`.
+
+### Building the Tables
+
+Building the tables for all kernel versions known to the scripts takes about a
+minute per kernel version.  The tables are written to the `tables-<version>`
+subdirectories of the current directory.
+
+	# KERNEL=/path/to/linux
+	# SCTABLE=/path/to/syscalls-table
+	# TABLES=/path/to/tables
+	# LIBSECCOMP=/path/to/libseccomp
+	# mkdir -p $TABLES
+	# podman run --rm --userns=keep-id --security-opt label=disable \
+	      -v $KERNEL:$KERNEL -v $SCTABLE:$SCTABLE -v $TABLES:$TABLES \
+	      -v $LIBSECCOMP:$LIBSECCOMP:ro -w $TABLES libseccomp-kver \
+	      python3 $LIBSECCOMP/src/arch-build-kver-tables.py \
+	          -d $SCTABLE -k $KERNEL
+
+With Docker, replace `--userns=keep-id` with `--user $(id -u):$(id -g)`.  If the
+kernel tree is a git worktree, the repository it belongs to must be mounted as
+well.
+
+To only build the tables for some kernel versions, use the `-V` option, e.g.
+`-V 7.1,7.2`.
+
+### Filling In the Kernel Versions
+
+Once the tables are built, fill in the kernel versions in the CSV file from the
+libseccomp source directory (this does not require the container):
+
+	# ./src/arch-update-syscalls-csv.py -d $TABLES -c src/syscalls.csv \
+	      -V $(ls $TABLES | sed 's/^tables-//' | paste -sd,)
+
+This only sets the kernel versions which are `SCMP_KV_UNDEF`, so to regenerate
+them from scratch, reset them first:
+
+	# sed -i 's/SCMP_KV_[0-9][0-9_]*/SCMP_KV_UNDEF/g' src/syscalls.csv
+
+The syscalls-table project does not generate the s390 (31-bit) tables, so the
+s390 kernel versions remain `SCMP_KV_UNDEF`.
+
+### Adding New Kernel Versions
+
+There is no need to rebuild the tables for all kernel versions when a new
+kernel is released.  To add a new kernel version, e.g. v7.3:
+
+1. Add the `SCMP_KV_7_3` value to the `scmp_kver` enumeration in
+   `include/seccomp-kvers.h`, and to the Python bindings in
+   `src/python/libseccomp.pxd` and `src/python/seccomp.pyx`.
+
+2. Add the new version to the `kernel_versions` list in
+   `src/arch-build-kver-tables.py`.
+
+3. Update the syscall numbers in the CSV file (this requires libseccomp to be
+   built first); the syscalls new to this kernel version get `SCMP_KV_UNDEF`
+   as their kernel version:
+
+		# cd src; ./arch-syscall-validate -c syscalls.csv $KERNEL; cd ..
+
+4. Build the tables for the new kernel version as described above, adding
+   `-V 7.3`.  A recent toolchain is fine for recent kernels, so the container
+   is only needed if the host lacks the glibc x32 headers.  If the container's
+   toolchain becomes too old for new kernels, update its base image.
+
+5. Fill in the kernel versions of the new syscalls:
+
+		# ./src/arch-update-syscalls-csv.py -d $TABLES -c src/syscalls.csv \
+		      -V 7.3

include/seccomp-kvers.h

@@ -108,5 +108,10 @@ enum scmp_kver {
 	SCMP_KV_6_15 = 78,
 	SCMP_KV_6_16 = 79,
 	SCMP_KV_6_17 = 80,
+	SCMP_KV_6_18 = 81,
+	SCMP_KV_6_19 = 82,
+	SCMP_KV_7_0 = 83,
+	SCMP_KV_7_1 = 84,
+	SCMP_KV_7_2 = 85,
 	__SCMP_KV_MAX,
 };

src/arch-build-kver-tables.py

@@ -20,18 +20,14 @@
 # along with this library; if not, see <http://www.gnu.org/licenses>.
 #
 
-#######################################################
-#### WARNING - to generate proper headers for x32, you
-####           must install the glibc 32-bit headers
-####
-####           apt install libc6-dev-x32
-####
-#######################################################
+# See doc/admin/SYSCALL_KVER_TABLES.md for the requirements and usage.
 
 from subprocess import TimeoutExpired
 import subprocess
 import argparse
+import shutil
 import os
+import re
 
 kernel_versions = ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5', '3.6', '3.7',
                    '3.8', '3.9', '3.10', '3.11', '3.12', '3.13', '3.14',
@@ -43,7 +39,12 @@
                    '5.11', '5.12', '5.13', '5.14', '5.15', '5.16', '5.17',
                    '5.18', '5.19', '6.0', '6.1', '6.2', '6.3', '6.4', '6.5',
                    '6.6', '6.7', '6.8', '6.9', '6.10', '6.11', '6.12',
-                   '6.13', '6.14', '6.15', '6.16', '6.17']
+                   '6.13', '6.14', '6.15', '6.16', '6.17', '6.18',
+                   '6.19', '7.0', '7.1', '7.2']
+
+# kernel architecture directories used by libseccomp
+kernel_arches = ['arm', 'arm64', 'loongarch', 'm68k', 'mips', 'parisc',
+                 'powerpc', 'riscv', 's390', 'sh', 'x86']
 
 def parse_args():
     parser = argparse.ArgumentParser('Script to populate the syscalls.csv kernel versions',
@@ -130,15 +131,44 @@ def main(args):
         update_cmd = 'cd {};bash scripts/update-tables.sh {}'.format(
                      args.datapath, args.kernelpath)
         ret, out, err = run(update_cmd, shell=True)
-        if ret != 0:
+
+        # NOTE: this relies on the update-tables.sh fixes from syscalls-table
+        #       commits c58bc160537f, e5b5b182d2c9 and b019db68a150
+
+        # update-tables.sh keeps the old table if it fails to generate a
+        # new one
+        failed = re.findall(r'Failed to compile list-syscalls for (\S+)', out)
+        if failed:
+            raise RuntimeError('Failed to build the {} tables for kernel '
+                               'v{}'.format(', '.join(failed), kver))
+
+        # update-tables.sh skips the architectures whose headers can not be
+        # installed.  This is fine for the architectures not used by
+        # libseccomp (e.g. the cris headers can not be installed in Linux
+        # v3.x).
+        failed = re.findall(r'Failed to install headers for (\S+)', out)
+        if [arch for arch in failed if arch in kernel_arches]:
+            raise RuntimeError('Failed to install the {} headers for kernel '
+                               'v{}'.format(', '.join(failed), kver))
+        if ret != 0 and not failed:
             raise RuntimeError('Failed to update tables: {}'.format(ret))
 
+        # update-tables.sh only regenerates the tables for the architectures
+        # present in this kernel version.  The other tables in data/tables
+        # are leftovers (e.g. from a newer kernel version), so only copy the
+        # tables that were generated for this kernel.
+        arch_path = os.path.join(args.datapath,
+                                 'data/architectures-present-in-kernel.text')
+        with open(arch_path, 'r') as archf:
+            arches = [arch.strip() for arch in archf if arch.strip()]
+
         src_path = os.path.join(args.datapath, 'data/tables')
         dest_path = os.path.join(os.getcwd(), 'tables-{}'.format(kver))
-        cp_cmd = 'cp -r {} {}'.format(src_path, dest_path)
-        ret, out, err = run(cp_cmd, shell=True)
-        if ret != 0:
-            raise RuntimeError('Table copy failed: {}'.format(ret))
+        shutil.rmtree(dest_path, ignore_errors=True)
+        os.makedirs(dest_path)
+        for arch in arches:
+            shutil.copy(os.path.join(src_path, 'syscalls-{}'.format(arch)),
+                        dest_path)
 
 if __name__ == '__main__':
     args = parse_args()