test/install errors pass silently on PG 9.5-15: pg_regress ignores psql's exit status before PG16 #148

Claude here, filing this at the maintainer's request.

Problem: on PostgreSQL 15 and earlier, a test/install/*.sql file that hits an error with ON_ERROR_STOP set still reports ok, and the error does not fail make test. pgxntool's docs say the opposite. base.mk's test/install section says psql's non-zero exit is something "pg_regress does report as a failure". README.asc's test/install warning ("The only thing that still fails the build is a hard SQL error, and only if the file sets ON_ERROR_STOP") and CLAUDE.md's test/install note say the same. That's only true on PG16 and later.

Why: before PG16, pg_regress decides pass/fail from the output diff alone. A non-zero exit from a test's psql process only appends a note to that test's status line, e.g. ok ... (test process exited with exit code 3). PostgreSQL commit 337903a16fb ("Consider a failed process as a failed test in pg_regress", first released in 16.0) changed this. It was not backpatched; REL_15_STABLE still has the old logic, and 9.5 behaves the same as 15. test/install's expected and actual output are the same file, so its diff can never fail. On PG 15 and earlier, nothing in the install step can fail the run.

Effect: an install-file error that keeps the extension from being installed is still caught on older versions, indirectly, because the test/sql files that follow it then fail. An error raised after the install succeeded is not caught at all, e.g. an assertion at the end of the file such as a version check after ALTER EXTENSION UPDATE, or a check that installing left session settings unchanged. The ON_ERROR_STOP check from #97 makes sure such an error stops psql, but on PG 15 and earlier that stop never reaches the build result. Postgres-Extensions/trunklet#10 and #11 have assertions of exactly this kind in test/install/load.sql.

Possible fixes (not tested):

• After pg_regress runs, fail if its log (regression.out in the output directory) contains exited with exit code or terminated by signal. pg_regress writes that note on every supported version, so this would give PG16's behavior to older versions, for every test rather than only test/install.
• Alternatively, check each test/install .out file for psql error lines. pg_regress runs psql with stderr redirected into the .out file, so the error text lands there.
• Either way, the three doc passages above should say what actually happens on PG 15 and earlier.

Related: #97 (the ON_ERROR_STOP check), #104 (test/install .out cleanup), and open PR #144, which edits the same README text.