Merge pull request #171 from Univers42/164-bug-incomplete-output-from-curl-installations

.github/workflows/ci.yml

@@ -254,6 +254,8 @@ jobs:
       # debug build runs this sweep from the pty job; this is its release run.
       - name: ${...} quoting sweep against the RELEASE build
         run: python3 tests/expansion_quote_sweep_test.py build/bin/hellish
+      - name: curl installer output regression (#164)
+        run: python3 tests/curl_install_output_test.py build/bin/hellish
       # The suite diffs hellish against bash byte-for-byte, so anything that
       # steers bash steers the verdict -- and the people most likely to run it
       # are the people whose rc exports the most. A ~/.hellishrc framework

.gitignore

@@ -9,7 +9,6 @@ out.dot
 tests/results.log
 tests/bench_results.txt
 tests/agnostic_results.tsv
-.claude/
 
 # secrets / credentials — never commit
 .env
@@ -44,8 +43,5 @@ tests/glob-zoo/tab	file
 
 # Host-runnable static binaries produced by `make static`.
 dist/
-.claude/*
-.claude
-.CLAUDE
-*CLAUDE*
 site/
+*.env

.gitmodules

@@ -12,3 +12,6 @@
 	path = tests/inception
 	url = ../Inception.git
 	branch = feat/scripts-run-with-launching-shell
+[submodule ".claude"]
+	path = .claude
+	url = ../claude-deal-with-the-devil.git

CLAUDE.md

@@ -0,0 +1,77 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+hellish is a from-scratch POSIX(+bashisms, + an opt-in zsh dialect) shell in C, written as a 42 project. Its behaviour is defined by diffing against a **pinned bash 5.3.9** (`bash --posix`) and, for the zsh dialect, a pinned zsh 5.9. In-depth docs: `DEV_DOC.md`, `CONTRIBUTING.md`, and `src/<module>/README.md` for each module.
+
+## Setup
+
+- Submodules are required: `vendor/libft` (libft + the `ft_malloc` allocator) and `vendor/scripts` (dev tooling). If the build can't find libft, run `git submodule update --init --recursive`.
+- Build the oracles once: `make oracle` (→ `~/bash-5.3.9`) and `make zsh-oracle` (→ `~/zsh-5.9`). The golden suite compares against these, not the system bash.
+- `make` with no target prints the help page, which lists every target with a description.
+
+## Build
+
+| Command | Result |
+|---|---|
+| `make all` | `MODE=debug`: `-O0 -g3`, ASan+LSan, libc malloc (`SAFE=1`) → `build/bin/hellish` |
+| `make OPT=1 all` / `MODE=release` | `-O3 -flto`, `ft_malloc` (`SAFE=0`), no sanitizers |
+| `MODE=relwithdebinfo` | `-O2 -g`, `ft_malloc` |
+| `make re` / `make re OPT=1` | full rebuild; both must build cleanly before a PR |
+
+- `-Wall -Wextra -Werror`: a warning fails the build. Add flags with `EXTRA_CFLAGS=…`. Passing `CFLAGS=…` replaces the whole flag set.
+- Each mode/SAFE combination has its own object dir (`build/obj-<mode>-<alloc>`), but they all link to the same binary path.
+- Sources are found with `find src` (excluding `src/platform/`), so a new `.c` file needs no Makefile edit. Only `src/platform/$(TARGET)/` (currently `posix`) is compiled in. Platform-specific code goes in per-file implementations there, not in `#ifdef`s.
+- Debug knobs: `--debug=lexer|parser|ast`, `HELLISH_ALLOC_STATS=1` (live bytes at exit, the leak oracle on `SAFE=0`), `HELLISH_BANNER=0`, `HELLISH_NO_UPDATE_CHECK=1`.
+
+## Test
+
+A green `make test` alone isn't enough. The test layers are:
+
+- **Golden suite**: `make test`. Each category in `tests/` is a file with one command per line. Every case runs through `hellish -c` and the pinned `bash --posix`, and stdout, exit status and any written files are diffed.
+  - Run one category: `cd tests && ./tester <category>` (`-v` for verbose).
+  - **A new category file must be added to `test_lists` in `tests/tester`**, or it silently never runs.
+  - The tester `chmod 000`s `tests/test_files/invalid_permission`. If a run is interrupted, `chmod 755` it before git operations.
+- **pty/interactive tests**: `make pty-test`. It discovers every `tests/*.py` automatically. Run a subset with `tests/pty_suite.sh -- <pattern>`. Each `.py` takes the shell path as its only `argv[1]`; keep it that way.
+  - Many `make *-test` targets run one pty test; see `make help`.
+- **Whole scripts**: `tests/run_scripts.sh`. **Hard corpus**: `tests/hard/run.sh`.
+- **Allocator parity**: `cd tests && ./verify_alloc.sh` builds and diffs both `SAFE=1` and `SAFE=0` against bash. Run `make test-release` after touching memory code (ASan and `-O3` can disagree).
+- `make arena-stress`: parse arena under ASan with tiny chunks. `make help-test`: every builtin must have a `help` entry.
+- `make norm` runs 42 norminette (it only reports, and always exits 0).
+- Pre-PR gates: `make re`, `make re OPT=1`, `make test`, `tests/run_scripts.sh`, `tests/hard/run.sh`, `make pty-test`, `cd tests && ./verify_alloc.sh`, `make norm`.
+
+Leak checking: ASan/LSan only work on `SAFE=1`. On `SAFE=0` use `HELLISH_ALLOC_STATS=1`.
+
+## Architecture
+
+One command's path through the shell (`src/core/shell.c`: `main` → `on()` → `repl_shell` → `off()`):
+
+1. **Input** (`src/infrastructure/input*.c`, readline in `rl_*`/`history*`): one unified line buffer (`t_rl`) for tty, file, `-c` and pipe input, plus multi-line joining and heredoc collection (`src/heredoc/`).
+2. **Alias expansion** (`src/alias/`) produces `state->alias_exp`, which is what gets lexed.
+3. **Lexer** (`src/lexer/`) and **parser** (`src/parsing/`) build the AST (`incs/ast.h`, `src/infrastructure/ast*.c`).
+4. **Reparser** (`src/word_splitting/`) runs between parsing and expansion. It splits coarse `AST_WORD` tokens into quote- and variable-aware sub-tokens and re-tags assignment words.
+5. **Expander** (`src/expander/`) handles parameter/arith/command/process substitution, IFS splitting, brace expansion, globbing (`src/glob/`) and redirect construction. `expand_zsh_*` is the zsh dialect.
+6. **Execution** (`src/execution/`, fork/job parts in `src/platform/posix/`, `src/job_control/`) runs builtins (`src/builtins/`, dispatch table plus `help_data*.c`), functions, pipelines and compound commands.
+7. After each REPL turn, the per-command AST, redirects and input are freed, so memory stays flat over long scripts.
+
+**`t_shell`** (`incs/shell.h`) holds all the shell's state and is passed everywhere. New state goes there, or in a struct you pass down. **Never add a global variable.** The few legacy globals (word slab, parse arena, env index, readline completion iterators, dir stack) are scheduled for cleanup and aren't precedent.
+
+**Memory model** (`src/helpers/README.md`):
+- All allocation goes through `xmalloc`/`xfree`. They route to either libc or `ft_malloc` at compile time, and a pointer must never cross heaps.
+- The **parse arena** (`parena_*`) is a bump allocator, open only around the main cycle's parse. Nested parses (`eval`, `source`, cmdsub bodies) fall through to the heap. `parena_free` is a no-op for arena pointers. Heap memory attached to an arena tree must call `parena_note_attach()`. Before sharing a struct between nodes, check `parena_owns()`; issue #94 was a double free caused by skipping that check.
+- The **word slab** (`word_slab_push`) holds argv strings. Values that can escape into the environment (assignments, `for` values) must live on the general heap.
+- Shutdown goes through `free_all_state`. Its ordering is deliberate.
+- Bugs in libft/ft_malloc get fixed in the submodule's own repo, then the pointer gets bumped here. Don't work around them from the shell.
+
+## Code style (42 Norm, enforced)
+
+- At most 5 functions per file and 80 columns per line. No `for` loops and no ternaries. Every file starts with the standard 42 header. Because files hit the 5-function limit, modules have many `foo2.c`, `foo3.c` files; continue that pattern.
+- Comments are block comments placed **before** a function or at file scope, never inside a function body. They explain the *why* (root cause, invariant, issue number). Complex functions get a Doxygen block (`@brief`/`@param`/`@return`, plus the hard part).
+
+## Workflow
+
+- Every fix ships with a test that covers the bug and nearby cases.
+- Branch from `develop` (named `type/short-description`). PRs target `develop`; `main` and `develop` are protected.
+- Commits use Conventional Commits (`type(scope): imperative subject`, types `feat|fix|refactor|perf|test|docs|chore|style`), enforced by the hook from `./vendor/scripts/install-hooks.sh`. The body explains the root cause and what the test proves.
+- **Don't add `Co-authored-by:` trailers for AI tools.**
+- After changing builtins, regenerate the wiki page with `make docs-builtins`.

USER_DOC.md

@@ -97,6 +97,52 @@ A syntax error in any file hellish loads is reported once, with the file
 and line, and the line itself — the same two lines bash prints — and the
 lines before it still run.
 
+## The prompt
+
+`PS1` is the prompt, and it is bilingual: bash's backslash escapes (`\u`,
+`\w`, `\$`, `\[ \]`…) and zsh's percent escapes (`%~`, `%F{cyan}`,
+`%(?.a.b)`…) work in the same string. `PROMPT` is the zsh spelling with
+zsh's exact rules, and wins when both are set; `RPROMPT` is the right
+prompt. The full escape list is section 1 of `hellishrc.example`, and
+[wiki/interactive.md](wiki/interactive.md#prompt-) has the rest.
+
+What is live at every render, and what is not:
+
+| in the prompt | rendered? |
+|---|---|
+| `$VAR`, `${VAR}`, `${VAR:-x}`, `$?` and the other specials | yes, every time the prompt is drawn |
+| `$((…))` | yes |
+| `$(…)` and backquotes | **no** — the text stays on screen as written; it would fork on every redraw |
+
+For a computed segment, set a variable from a hook and put the variable in
+the prompt. `HELLISH_PRECMD_FUNCS` runs once before each prompt
+(`hellishrc.example`, section 5b):
+
+```sh
+_branch() { BR=$(git branch --show-current 2>/dev/null); }
+HELLISH_PRECMD_FUNCS=(_branch)
+PS1='${BR:+($BR) }\w \$ '
+```
+
+**`prompt` is a shell function, not a builtin.** The installer seeds it
+into `~/.config/hellish/rc.d/40-prompt-switch.hsh`, next to the 29 themes
+in `~/.config/hellish/themes/`. `type prompt` says whether it is loaded;
+from a checkout, `HELLISH_THEMES=share/themes; . share/rc.d/40-prompt-switch.hsh`
+loads it by hand.
+
+**`\A` is not bash's 24-hour clock here.** It is the animated glyph, and it
+was there first. A bash `PS1` pasted in with `\A` shows the glyph — or
+nothing, since the animation is off by default — where bash shows `14:30`.
+Write `\D{%H:%M}` or `%T` for the clock.
+
+**The animation is off by default, on purpose.** `HELLISH_ANIM=spinner`
+(or `pulse`, `ember`) makes `\A` repaint the prompt row about ten times a
+second while the shell is idle. That repaint has been reported to corrupt
+the prompt on some terminals: an escape sequence printed as text
+(`8;2;90;96;106m`), or a cut glyph shown as `�`. It could not be
+reproduced on demand, so it ships off. If you turn it on and see that,
+`HELLISH_ANIM=off` is the fix.
+
 ## Check that it works
 
 ```sh
@@ -111,11 +157,62 @@ conf list                    # plugin framework: what is on and off
 If a terminal ever misbehaves after an experiment: `reset` — and the
 escape hatches above get you back to your previous shell at any time.
 
+## Completing `make` targets
+
+`make <TAB>` / `make cl<TAB>` completes target names from the Makefile in the
+current directory (also `gmake`, `/usr/bin/make`, and behind `sudo`, `env X=1`,
+`&&`, `(`; a wrapper option that takes an argument, as in `sudo -u root make`,
+falls back to filenames). The file is parsed in the shell itself; make is never run.
+
+- Files read: the first regular file of `GNUmakefile`, `makefile`, `Makefile`
+  (GNU make's order, symlinks followed), at most 4 MiB. No such file, a
+  directory, or an unreadable file means plain filename completion.
+- Offered: the names on `target:` / `a b c:` / `a::` lines. Not offered:
+  variable assignments, directives, recipe lines, `define` blocks, names with
+  `%` or `$`, names starting with `.`; text after the colon is ignored.
+- `include` / `-include` / `sinclude` of literal file names are followed ONE
+  level deep (what an included file includes is ignored). Changing only an
+  included file does not refresh the cache; touch the Makefile.
+- Cache: the targets are reused while the Makefile's name, inode, mtime and
+  size are unchanged (one `stat` per TAB).
+- A word starting with `-` or containing `=` or `/` keeps filename completion,
+  and so does any `make` line with `-C`, `--directory`, `-f`, `--file` or
+  `--makefile` before the cursor, or a prefix that matches no target
+  (`make out.txt`).
+- A `complete` spec for `make` (under `shopt -s progcomp`) wins over all of
+  this; `complete -r make` brings the targets back.
+
 ## Knobs
 
 | variable | effect |
 |---|---|
 | `HELLISH_BANNER=0\|1` | welcome panel off / on |
+| `HELLISH_HISTORY_PREFIX=0\|off` | Up/Down plain history; default is prefix search, read at every prompt |
+| `HELLISH_AUTOSUGGEST=1\|on` | inline suggestion, off by default: the newest history line that starts with what you typed, shown dim after the cursor (see below); read at every prompt |
+| `HELLISH_SUGGEST_MAX=N` | how many of the newest history lines the suggestion looks at (default 1000; not a number, or below 1, is the default); read at every prompt |
 | `HELLISH_NO_UPDATE_CHECK=1` | never check for updates, no badge |
 | `HELLISH_NO_ANIM=1` | no startup animation |
 | `HELLISH_NO_EXEC=1` | user-install hook: skip for this one session |
+
+### Inline suggestion
+
+With `HELLISH_AUTOSUGGEST=1` (set it at the prompt or in the rc file) the
+newest history line that starts with what you have typed appears dim after
+the cursor. It is a hint, not part of the line: **Right**, **End** or
+**Ctrl-F** at the end of the line insert it; anywhere else, and when there
+is no hint, those keys do what they always did. Enter runs only what you
+typed, and the hint is wiped from the screen. `unset HELLISH_AUTOSUGGEST`
+(or `=0`) turns it off at the next prompt. A key you bound yourself
+(`bind` / `bindkey` / `.inputrc`) is never taken over.
+
+Never suggested: a line with a newline, a tab or any other control character
+(an old `printf '\e[31m'` cannot repaint your prompt), and nothing at all
+while you search (Ctrl-R), in vi command mode, on a continuation prompt, with
+the cursor inside the line, or on a line that would not fit the terminal's
+width (the hint is cut at the edge, never wrapped). A right prompt
+(`RPROMPT`) is hidden while the hint reaches it and comes back when it does
+not.
+
+For zsh widgets (`zle -N`), the variable `POSTDISPLAY` is the text shown
+after the cursor: read it in a widget to get the current hint, assign it
+(`POSTDISPLAY=' <hint>'`) to show your own until the line next changes.

docker/Dockerfile

@@ -44,17 +44,25 @@ FROM ${BASE}
 # in was a red rung on a commit the rung had built fine an hour earlier.
 # A mirror that keeps failing four times still fails the build -- it is
 # the transient, not the broken, that is retried.
+# apt gets the same loop, update included: an Ubuntu mirror mid-sync serves
+# an index that disagrees with its files ("File has unexpected size ...
+# Mirror sync in progress?", or a 404 on a .deb the index already moved
+# past), and only a fresh update a little later cures it. It turned the
+# ubuntu 22.04 rung red on PR #141 before a line was compiled.
 ARG CC=cc
 
 RUN set -eux; \
 	if command -v apk >/dev/null 2>&1; then \
 		apk add --no-cache build-base readline-dev ncurses-dev git bash \
 			coreutils python3 clang; \
 	elif command -v apt-get >/dev/null 2>&1; then \
-		apt-get update; \
-		DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
-			build-essential libreadline-dev git ca-certificates bash \
-			python3 clang; \
+		n=0; until apt-get update -o Acquire::Retries=3 \
+			&& DEBIAN_FRONTEND=noninteractive apt-get install -y \
+				-o Acquire::Retries=3 --no-install-recommends \
+				build-essential libreadline-dev git ca-certificates bash \
+				python3 clang; do \
+			n=$((n + 1)); [ "$n" -ge 4 ] && exit 1; sleep $((n * 15)); \
+		done; \
 		rm -rf /var/lib/apt/lists/*; \
 	elif command -v pacman >/dev/null 2>&1; then \
 		pacman -Sy --noconfirm --needed base-devel readline git bash \

hellishrc.example

@@ -70,6 +70,15 @@
 #
 # \S is usually the nicer answer than $? anyway: it shows " ✘N" only after a
 # failure and stays invisible when everything is fine.
+#
+# $((...)) is live too. $(...) is NOT: command substitution in a prompt is
+# never run -- it stays on screen as the text you wrote -- because it would
+# fork on every redraw. Compute the value in a hook (section 5b) and put the
+# variable in the prompt; the hook runs once per prompt, not per redraw:
+#
+#     _branch() { BR=$(git branch --show-current 2>/dev/null); }
+#     HELLISH_PRECMD_FUNCS=(_branch)
+#     PS1='${BR:+($BR) }\w \$ '
 
 # ── ANIMATION: OFF BY DEFAULT, AND THAT IS DELIBERATE ──────────────────
 # A live style makes \A repaint the prompt rows in place ~10x/second