Install node, yarn and python from the distribution (closes #146)

Follows sneak's ruling at
#146 (comment). Runtimes
and the installer tools bundled with python come from the distribution,
with no version and no hash. Third-party packages stay pinned by hash.

script/bootstrap drops nvm and corepack. It installs node, yarn and the
node packages only when yarn.lock is in the tree, and stops there when
yarn's major version is not 1. This repository's Dockerfile starts from
Alpine 3.24. The Python example starts from debian:trixie-slim, keeps
pipenv at $HOME/.pipenv on a host and in an image, and sets
PIP_ONLY_BINARY=:all:, so pip downloads no build tools without a hash
check.

Model: opus-5-5

prompts/REPO_POLICIES.md

@@ -25,6 +25,7 @@ style conventions are in separate documents:
   a remote source, including:
     - Docker base images
     - Go modules
+    - Python packages
     - npm packages
     - GitHub Actions
 
@@ -33,29 +34,45 @@ style conventions are in separate documents:
     dependency by its content hash:
     - Docker: `@sha256:...`
     - Go: the module hash in `go.sum`
+    - Python: the package hash in `Pipfile.lock`
     - npm: the integrity hash in the lockfile
     - GitHub Actions: `@<commit-sha>`
 
-    This also means never `curl | bash` to install tools like pyenv, nvm,
-    rustup, etc. Instead, download a specific release archive from GitHub,
-    verify its hash, and only then install. The hash is hardcoded in the
-    Dockerfile or script. Unverified install scripts are arbitrary remote code
-    execution. Double-check every external reference in every file before
-    committing.
+    This also means never `curl | bash` to install a tool such as rustup.
+    Instead, download a specific release archive from GitHub, verify its hash,
+    and only then install. The hash is hardcoded in the Dockerfile or script.
+    Unverified install scripts are arbitrary remote code execution. Double-check
+    every external reference in every file before committing.
 
     There is one exception: a package from the distribution's own package index
     needs no version and no hash. Pulling that index is also allowed. The
     distribution is trusted. In a `Dockerfile`, it is the pinned base image's
-    distribution. On a host, it is the host's. These commands are examples:
+    distribution. On a host, it is the host's. The host's package manager stands
+    for its distribution, and that includes nix and brew. These commands are
+    examples:
     - `apt-get update`
     - `apt-get install`
     - `apk update`
     - `apk add`
 
+    These come from the distribution wherever possible:
+    - node
+    - yarn
+    - npm
+    - python
+    - pip and venv, the installer tools bundled with python
+
+    If the distribution's version is too old, use a newer distribution, never a
+    download. That rules out all of these:
+    - nvm
+    - pyenv
+    - corepack
+    - `npm install -g yarn`
+
     These are not trusted, so everything from them stays pinned by hash:
     - Docker Hub
     - PyPI
-    - npm
+    - the npm registry
     - `go get`
 
     A package source that a `Dockerfile` adds itself is not the distribution's
@@ -99,18 +116,35 @@ style conventions are in separate documents:
     initialization.
 
     `script/bootstrap` installs all dependencies idempotently and assumes
-    nothing is present. Base tools come from nix, apt, brew, or apk, detected in
-    that order; apt runs noninteractive. For node it uses the installed node
-    only when its major version is the pinned one. When node is missing or has
-    another major, it installs the PINNED node version via nvm, and installs
-    yarn and the packages under it. The CI runner image ships node 24, for
-    example. nvm itself is installed first if missing, from a hash-verified
-    GitHub release archive, never `curl | sh`. bash is installed as an explicit
-    prerequisite, since nvm requires bash. Only the major version is compared
-    because the `Dockerfile` stages start from a node image whose exact version
-    is not the pin, and nvm cannot install a prebuilt node on alpine. yarn is
-    pinned via `corepack prepare yarn@<version> --activate`. Never install
-    "latest" or "lts"; always exact versions.
+    nothing is present. Base tools, node and yarn come from nix, apt, brew, or
+    apk, detected in that order; apt runs noninteractive. Each one is installed
+    only when it is missing. A tool from outside the distribution is never
+    installed as "latest" or "lts"; it is always an exact version.
+
+    `script/bootstrap` installs node, yarn and the node packages only when
+    `yarn.lock` is in the tree. With no `yarn.lock`, it installs no node and no
+    yarn, and it does not check yarn's version. Make a repository's first
+    `yarn.lock` with yarn 1, from a distribution that packages it, such as
+    Alpine (`apk add nodejs yarn`).
+
+    Debian and Ubuntu name the yarn command `yarnpkg`. `script/bootstrap`,
+    `script/fmt` and `script/fmt-check` each carry the same small function that
+    runs whichever name is present. With neither name on `PATH`, `script/fmt`
+    and `script/fmt-check` print one line saying to run `script/bootstrap`, and
+    exit 1.
+
+    With `yarn.lock` in the tree, `script/bootstrap` stops with a one-line
+    message when yarn's major version is not 1, before it installs any node
+    packages. Debian 13 and Ubuntu 26.04 package yarn 4. With
+    `--frozen-lockfile`, yarn 4 refuses a yarn 1 `yarn.lock`. Without that flag,
+    it rewrites the file in its own format, with yarn 4 hashes in place of the
+    yarn 1 ones. A host whose distribution packages yarn 4 moves to one that
+    packages yarn 1, such as Alpine or Ubuntu 24.04.
+
+    A `Dockerfile` stage copies `yarn.lock` before it runs `script/bootstrap`
+    only if that stage needs yarn. A stage that needs yarn starts from an image
+    whose distribution packages yarn 1, such as Alpine. The Python and Go stages
+    below need no yarn and copy no `yarn.lock`, so they never reach the stop.
 
     `script/cibuild` runs the CI build, and the Gitea workflow calls it. The
     script changes to the repo root, runs `script/bootstrap`, runs
@@ -118,15 +152,7 @@ style conventions are in separate documents:
     runs `script/bootstrap` first**, because the workflow checks out the repo
     and runs nothing else, while `script/fmt-check` runs the formatter on the
     host. On a pristine checkout with nothing installed, the run dies there,
-    after the containerised gates have passed. **The bootstrap alone is not
-    enough.** When `script/bootstrap` installs node and yarn under nvm, it
-    leaves neither on the `PATH` of the shell that called it. A bare `yarn` then
-    either exits 127 or runs under another node. The host entrypoints that need
-    yarn, `script/fmt` and `script/fmt-check`, therefore use the same test as
-    `script/bootstrap`. When the node on `PATH` has the pinned major version,
-    they run the `yarn` on `PATH`. Otherwise they load nvm through
-    `$HOME/.nvm/nvm.sh` and run yarn under the pinned node. A runner carrying
-    nothing but docker and git then gets through `script/check`.
+    after the containerised gates have passed.
 
     Four further scripts are our own extensions to the standard:
     - `script/check` runs `script/test`, `script/lint` and `script/fmt-check`.
@@ -175,10 +201,11 @@ style conventions are in separate documents:
       does for its yarn packages
 
     The development environment stage installs development prerequisites by
-    running `script/bootstrap` rather than duplicating its installs inline. A
-    stage that runs `script/bootstrap` COPYs `script/` and the dependency
-    manifests (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before
-    running it.
+    running `script/bootstrap`. Only system packages may be installed inline
+    above it, so that check builds keep them; the `prompts` repo's own
+    `Dockerfile` does this. A stage that runs `script/bootstrap` COPYs `script/`
+    and the dependency manifests it needs (`package.json` + `yarn.lock`,
+    `go.mod` + `go.sum`, etc.) before running it.
 
 - **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
   no separate lint file. `script/lint` and `script/test` each build one phase
@@ -491,10 +518,18 @@ style conventions are in separate documents:
         - pipenv and its dependencies
         - the packages from `Pipfile.lock`
 
-        `script/bootstrap` installs no Python there: the tests use the base
-        image's Python, which the image's hash pins. If it installs anything
-        else there, the step goes after `ARG CACHEBUST`.
+        `script/bootstrap` installs no Python there: the tests use the
+        distribution's Python, which a step above it installs. If it installs
+        anything else there, the step goes after `ARG CACHEBUST`.
 
+    - Debian's Python refuses a system-wide `pip install`, so pipenv goes in a
+      virtual environment at `$HOME/.pipenv`. `script/bootstrap` creates it with
+      `python3 -m venv` when it is missing, then uses that environment's `pip`
+      and `pipenv`. On a host, it first installs the distribution's python and
+      venv packages when they are missing. One script then works on a host and
+      in a `Dockerfile`. `Pipfile` names no `python_version`, because
+      `pipenv install --deploy` stops when the distribution's Python differs
+      from it.
     - PyPI is not trusted, so `script/bootstrap` installs pipenv and its
       dependencies by hash. It does so with `pip install --require-hashes -r`
       and a requirements file in `script/` that lists each of them with `==` and
@@ -504,37 +539,65 @@ style conventions are in separate documents:
         - installs only what `Pipfile.lock` names
         - checks every package against the lockfile's hashes
         - fails when the lockfile is out of date with `Pipfile`
+    - `script/bootstrap` sets `PIP_ONLY_BINARY=:all:` for both installs, and
+      pipenv passes it on to pip. A package with no ready-built wheel for the
+      image then stops the install. Without the setting, pip builds that package
+      from source and downloads the build tools without a hash check.
     - A kept layer holds nothing a new install would refuse: pip checked the
       hashes of pipenv and its dependencies, and pipenv checked the hashes of
       the lockfile's packages. pipenv creates its virtual environment with the
       `virtualenv` package. The packages that `virtualenv` adds come from wheels
-      inside that package, so the hash pip checked for it covers them. Two
-      things in the layer are not fixed by a hash:
-        - A package with no ready-built wheel for the image is built from
-          source. pip downloads the build tools without a hash check, so a kept
-          layer's build of such a package may differ from a new one.
-        - System packages keep the versions the package manager gave when the
-          layer was built; a new install may get newer ones.
+      inside that package, so the hash pip checked for it covers them. System
+      packages, Python among them, are not fixed by a hash. They keep the
+      versions the package manager gave when the layer was built; a new install
+      may get newer ones.
     - As with Go's modules, the packages are kept in a layer, not a cache mount.
       A cache mount is shared by every build on the host, and nothing checks its
       files again.
 
-    In the example below, `script/bootstrap` does all of these:
-    - finds `make` and `git` already installed
-    - installs pipenv and its dependencies by hash
-    - runs `pipenv install --dev --deploy`
+    A Python repository's `script/bootstrap` adds these lines in `main`, above
+    the last `echo`. The `ensurepip` test fails when python is missing, and also
+    when Debian's venv package is missing:
+
+    ```sh
+    if [ ! -x "$HOME/.pipenv/bin/pip" ]; then
+        if ! python3 -m ensurepip --version >/dev/null 2>&1; then
+            pkg_install python3 python3-venv python3 python3
+        fi
+        python3 -m venv "$HOME/.pipenv"
+    fi
+    export PIP_ONLY_BINARY=:all:
+    "$HOME/.pipenv/bin/pip" install --require-hashes -r script/requirements.txt
+    "$HOME/.pipenv/bin/pipenv" install --dev --deploy
+    ```
+
+    Make `script/requirements.txt` once, with
+    `echo pipenv | pip-compile --generate-hashes --allow-unsafe -o script/requirements.txt -`.
+    `pip-compile` comes from Alpine's `py3-pip-tools` package; Debian and Ubuntu
+    do not package `pip-tools`. Make the first `Pipfile.lock` with
+    `$HOME/.pipenv/bin/pipenv lock`, after the `pip install` line has installed
+    pipenv.
+
+    In the example below, `script/bootstrap` finds `make`, `git`, python and
+    venv already installed. The example copies no `yarn.lock`, so
+    `script/bootstrap` installs no node and no yarn.
 
     ```dockerfile
     # Test phase. Both pytest runs share one 90-second timeout, so a hang
     # uses it once and gets no rerun.
-    # python:3.x-slim, YYYY-MM-DD
-    FROM python@sha256:... AS test
-    # make and git are base tools that script/bootstrap installs when they
-    # are missing. Installed here, they stay in their own layer when
-    # script/ changes.
+    # debian:trixie-slim, YYYY-MM-DD
+    FROM debian@sha256:... AS test
+    # Python, pip and venv come from the distribution, which is trusted, so
+    # they need no version and no hash. make and git are base tools that
+    # script/bootstrap installs when they are missing. Installed here, they
+    # stay in their own layer when script/ changes.
     RUN apt-get update \
-        && apt-get install -y --no-install-recommends make git \
+        && apt-get install -y --no-install-recommends ca-certificates make git \
+            python3 python3-venv \
         && rm -rf /var/lib/apt/lists/*
+    # script/bootstrap installs pipenv in a virtual environment at
+    # $HOME/.pipenv, and the tests below run that pipenv.
+    ENV PATH=/root/.pipenv/bin:$PATH
     WORKDIR /app
     COPY script/ script/
     COPY Pipfile Pipfile.lock ./
@@ -669,16 +732,16 @@ style conventions are in separate documents:
 
     ```makefile
     test:
-    	@timeout 90 sh -c 'python -m pytest || { \
+    	@timeout 90 sh -c 'pipenv run pytest || { \
     		echo "--- Rerunning with -v for details ---"; \
-    		python -m pytest -v; exit 1; }'
+    		pipenv run pytest -v; exit 1; }'
     ```
 
     Both pytest runs share that one 90-second timeout, as in the `Dockerfile`
     example above, so a hang uses it once and gets no rerun. Ctrl-C does not
     stop a running `make test`, which goes on until the tests end or the 90
     seconds run out. On a Mac without `timeout`, the target fails until it is
-    installed.
+    installed. On a host, the target needs `$HOME/.pipenv/bin` on `PATH`.
 
     The `exit 1` ensures the target always fails after a rerun. The first run
     already proved the tests are broken, so the build must not pass even if a
@@ -857,10 +920,8 @@ style conventions are in separate documents:
       function defined and never invoked has the same exit status and the same
       empty output as one that worked.
 
-    Node is the exception to the whole-token comparison. The canonical
-    `script/bootstrap` compares only node's major version. That is because the
-    `Dockerfile` stages start from a node 22 alpine image, which is not the
-    exact pin, and nvm cannot install a prebuilt node on alpine.
+    A tool from the distribution has no pin, so `script/bootstrap` installs it
+    when it is missing. make, git, node and yarn are such tools.
 
     Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.