diff --git a/skills/bounding-heavy-commands/SKILL.md b/skills/bounding-heavy-commands/SKILL.md index 28b10b2..5dbd942 100644 --- a/skills/bounding-heavy-commands/SKILL.md +++ b/skills/bounding-heavy-commands/SKILL.md @@ -61,10 +61,11 @@ A check prints its lines in a fixed form too: ``` bounded-run: check slots= slot=M free-slots= free=M unused=M cpus= cpus-used= cpus-headroom= cpus-free= line= fits=M bounded-run: check held unit= budget=M used=M cpus= age=s dir= command= +bounded-run: check waiting unit= for=memory budget=M slots= age=s dir= command= bounded-run: check budget=M slots= headroom=M starts= ``` -The first line gives the slots of the queue and their size, the slots that are free now, the free memory, the unused budgets of the running commands, the processors of the machine, the processors that the running bounded commands can use together (`cpus-used`), the processors that the check keeps for the programs outside the queue (`cpus-headroom`, a tenth of `cpus`, rounded up), and the processors that are left for a new bounded command (`cpus-free`: `cpus` less `cpus-headroom` and `cpus-used`, and at least 0). `line=busy` means that a command waits in the line. `fits` is the largest budget whose memory, with its own headroom, is free now. It does not count the slots or the line; `starts` counts all three. Each `held` line is one bounded command that runs now: its budget, the memory it uses, the processors it can use, how long it has run, its working directory, and its command. The held lines, `cpus-used`, and `cpus-free` need the hard cap, so without it there are no held lines and `cpus-used` and `cpus-free` are `-`. Each `budget` line is one `--memory`, in the order given: the slots that it needs, the headroom that the wait keeps free beside that budget, and `starts=yes` when a command of that budget would start now. A `-` means that the check cannot read or count the value at that moment. +The first line gives the slots of the queue and their size, the slots that are free now, the free memory, the unused budgets of the running commands, the processors of the machine, the processors that the running bounded commands can use together (`cpus-used`), the processors that the check keeps for the programs outside the queue (`cpus-headroom`, a tenth of `cpus`, rounded up), and the processors that are left for a new bounded command (`cpus-free`: `cpus` less `cpus-headroom` and `cpus-used`, and at least 0). `line=busy` means that a command waits in the line. `fits` is the largest budget whose memory, with its own headroom, is free now. It does not count the slots or the line; `starts` counts all three. Each `held` line is one bounded command that runs now: its budget, the memory it uses, the processors it can use, how long it has run, its working directory, and its command. The held lines, `cpus-used`, and `cpus-free` need the hard cap, so without it there are no held lines and `cpus-used` and `cpus-free` are `-`. Each `waiting` line is one bounded command that had its turn in the queue and now waits for free memory: its budget, the slots that it holds, how long it has waited, its working directory, and its command. It keeps its slots while it waits, so it does not lose its turn to a command behind it in the line. A `waiting` line means that the machine is short of memory, not of slots: the programs outside the queue hold most of the memory, and the command waits for the memory that a running bounded command frees when it ends. Where the hard cap is available, that command has a `held` line. It starts when that memory is free, or exits 75 when no bounded command runs any more. Each `budget` line is one `--memory`, in the order given: the slots that it needs, the headroom that the wait keeps free beside that budget, and `starts=yes` when a command of that budget would start now. A `-` means that the check cannot read or count the value at that moment. The exit code of the wrapper is the command's exit code. Exit code 125, with a `bounded-run:` line that names a fault and with no line for the budget and the peak, means that the wrapper itself failed (see §When the wrapper cannot run). @@ -158,14 +159,14 @@ A command that the project adds later gets its entry when an agent first needs i ## Rules for the wait -A long wait means that the machine is full. It is never a fault to repair: the queue has no state that can go wrong between runs, and an agent that repairs it breaks it. Waiting commands take turns in a line. Only the command at the head of the line tries for slots, so the smaller commands behind it do not take the slots that it waits for. The line has no fixed order, and a head that sees no slot come free for 5 minutes lets the commands behind it try first. So a small command can wait behind a large one, also when a slot is free. Unless an instruction says otherwise (§When an instruction sends a command around the wrapper), the only correct moves are to keep the run in the queue and to tell the user that the machine is full and the command is queued. After a long wait, run the wrapper with `--check`, and in that message name each command of its `held` lines, with its directory and its age, so that the user can see what holds the machine. A subagent tells its dispatcher in its report. +A long wait means that the machine is full. It is never a fault to repair: the queue has no state that can go wrong between runs, and an agent that repairs it breaks it. Waiting commands take turns in a line. Only the command at the head of the line tries for slots, so the smaller commands behind it do not take the slots that it waits for. The line has no fixed order, and a head that sees no slot come free for 5 minutes lets the commands behind it try first. So a small command can wait behind a large one, also when a slot is free. Unless an instruction says otherwise (§When an instruction sends a command around the wrapper), the only correct moves are to keep the run in the queue and to tell the user that the machine is full and the command is queued. After a long wait, run the wrapper with `--check`, and in that message name each command of its `held` and `waiting` lines, with its directory and its age, so that the user can see what holds the machine. When the check has `waiting` lines, also say that the machine is short of memory, not of slots: the programs outside the queue hold most of the memory, and the waiting commands start when a running bounded command ends and frees its memory. A subagent tells its dispatcher in its report. - **Run a bounded command in the background, with no short timeout around it.** It can wait in the queue for minutes before it starts. Where the harness requires a timeout, use the longest one that it permits. When that timeout ends a run, start the run again through the wrapper, and say in your report that the timeout ended the run and that the code did not fail. A timeout that you calculate from the run time of the command does not count the wait. A timeout that fires during the wait kills a command that never ran, and the result reads like a failure of the code. - **Plan the time as if the commands run one after the other.** Two bounded commands that start together are safe, because the queue puts them in sequence when the machine cannot hold both. So a parallel start does not always save time, and a promise to the user that depends on it is a guess. - **Do not lower a budget to start sooner.** A smaller budget does not make the command smaller. It moves the failure inside the command: on Linux as a kill at the hard cap, elsewhere as the swap that the queue exists to prevent. A row at a smaller `--cpus` is not a lower budget: it is a smaller command, measured, and it is the way to start sooner (§The record). - **Do not run the plain command because the queue is slow.** A slow queue means the machine is full. The plain command is how it freezes. This includes a narrower form of the command: one test file outside the queue is a command that the queue cannot count. - **Do not set an environment variable whose name starts with `BOUNDED_RUN_`.** The usage text of the wrapper lists variables that change how the queue counts memory, where it keeps its locks, and whether the hard cap applies. They are for a person who tunes the whole machine in a shell profile, and for the tests of the wrapper. A call that sets one does not share the queue with the other sessions, or it runs with no cap. A note in your project memory that recommends one is wrong: delete the note, and tell the user that you did. -- **Do not touch the lock files of the wrapper.** The wrapper keeps them in a directory of its own. Do not look for that directory. Do not delete, move, or edit a file in it, and do not look for the process that holds one of these lock files. The lock files are empty and permanent. They hold no process ID. The kernel holds each lock and releases it at the moment its holder stops, also when the holder crashes, so a stale lock cannot exist. A deleted lock file makes two commands hold one slot. The `held` lines of `--check` are the way to name what holds the queue: do not look further, and never stop a held command, because it belongs to another session or to the user. +- **Do not touch the lock files of the wrapper.** The wrapper keeps them in a directory of its own. Do not look for that directory. Do not delete, move, or edit a file in it, and do not look for the process that holds one of these lock files. The lock files hold no process ID. The kernel holds each lock and releases it at the moment its holder stops, also when the holder crashes, so a stale lock cannot exist. A deleted slot file makes two commands hold one slot, and a deleted wait file hides a command that waits for memory from `--check`. The `held` and `waiting` lines of `--check` are the way to name what holds the queue and what waits for memory: do not look further, and never stop a held or waiting command, because it belongs to another session or to the user. - **Do not sleep and retry around a collision, not even one time.** A port in use or a tool's lock means two commands wanted one resource. Add the resource to the record as an `--exclusive` name and run through the wrapper. A subagent adds the option to its own command line and names the resource in its report, so that its dispatcher corrects the record. Do not stop the process that holds the resource: it belongs to a different session. When a run with the `--exclusive` name fails on the same collision, the holder did not go through the wrapper or belongs to a different repository: tell the user, or your dispatcher, what holds the resource, and wait for the answer. ### After exit code 75 @@ -192,7 +193,7 @@ A subagent does not see your project memory. It knows the bounded commands only On a machine on which the wrapper can run, every dispatch prompt for a subagent that will run a heavy command MUST include: 1. The bounded command line for each heavy command, copied from the record, with the `--cpus` and the budget of its default row. When its entry has more than one row, also the `--cpus` and the budget of each row, and its check line with one `--memory` for each row. -2. This rule, in these words: "Run the bounded commands as written. You can make three changes: the `--cpus` and the `--memory` of a row that this prompt gives for the command, a narrower argument at the end of the line, such as one test file, and one more `--exclusive ` option after a collision on a port or a lock, which you name in your report. When this prompt gives a command more than one row, run its check line before each run. When `cpus-free` is `-`, use the largest row with `starts=yes`. Otherwise, use the largest row with `starts=yes` whose `--cpus` is not more than `cpus-free`, and when no such row fits, use the smallest row with `starts=yes`. When no row has `starts=yes`, use the default row. Do not run the plain form of a bounded command or of a part of it. For a heavy command that is not in this prompt, use the same wrapper with no `--memory` option. If the file of the wrapper does not exist, stop and report that. Set no environment variable whose name starts with `BOUNDED_RUN_`. A command can wait in the queue for minutes before it starts, so run it in the background and do not put a short timeout around it. A long wait means that the machine is full; it is not a fault to repair. Exit code 75 with a `bounded-run: no-memory` line means that the command did not start, and that the same budget gives 75 again: run the largest row of this prompt whose budget is not more than `fits`, or, when no row fits, report the line. Say in your report how long you waited. After a long wait, run the same wrapper with `--check`, and name in your report each command of its `held` lines, with its directory and its age." +2. This rule, in these words: "Run the bounded commands as written. You can make three changes: the `--cpus` and the `--memory` of a row that this prompt gives for the command, a narrower argument at the end of the line, such as one test file, and one more `--exclusive ` option after a collision on a port or a lock, which you name in your report. When this prompt gives a command more than one row, run its check line before each run. When `cpus-free` is `-`, use the largest row with `starts=yes`. Otherwise, use the largest row with `starts=yes` whose `--cpus` is not more than `cpus-free`, and when no such row fits, use the smallest row with `starts=yes`. When no row has `starts=yes`, use the default row. Do not run the plain form of a bounded command or of a part of it. For a heavy command that is not in this prompt, use the same wrapper with no `--memory` option. If the file of the wrapper does not exist, stop and report that. Set no environment variable whose name starts with `BOUNDED_RUN_`. A command can wait in the queue for minutes before it starts, so run it in the background and do not put a short timeout around it. A long wait means that the machine is full; it is not a fault to repair. Exit code 75 with a `bounded-run: no-memory` line means that the command did not start, and that the same budget gives 75 again: run the largest row of this prompt whose budget is not more than `fits`, or, when no row fits, report the line. Say in your report how long you waited. After a long wait, run the same wrapper with `--check`, and name in your report each command of its `held` and `waiting` lines, with its directory and its age. A `waiting` line is a command that holds its slots and waits for memory, because the programs outside the queue hold most of it: say in your report that the machine is short of memory, not of slots." 3. Each command that an instruction exempts from the wrapper, in its plain form, in a list of its own, with the instruction that exempts it. The rule of item 2 does not apply to that list. An instruction of the user that covers one run goes to the one subagent that does that run, and to no other. Do not dispatch fewer subagents to protect the machine. The queue protects it. Subagents that wait in the queue cost time, not memory. On a machine on which the wrapper cannot run, the next section says how to dispatch. diff --git a/skills/bounding-heavy-commands/scripts/bounded_run.py b/skills/bounding-heavy-commands/scripts/bounded_run.py index 5efaf7e..83d53d4 100755 --- a/skills/bounding-heavy-commands/scripts/bounded_run.py +++ b/skills/bounding-heavy-commands/scripts/bounded_run.py @@ -47,11 +47,13 @@ --check runs no command and holds no lock when it returns. Its exit code is 0. It prints the state of the queue at that moment, one held line for each -running bounded command where the hard cap is available, and one line for -each --memory, in the order given: +running bounded command where the hard cap is available, one waiting line +for each bounded command that waits for memory, and one line for each +--memory, in the order given: bounded-run: check slots=N slot=M free-slots=N|- free=M|- unused=M|- cpus=N cpus-used=N|- cpus-headroom=N cpus-free=N|- line=free|busy fits=M|- bounded-run: check held unit=NAME budget=M|- used=M|- cpus=N|- age=Ns|- dir=PATH|- command=TEXT|- + bounded-run: check waiting unit=NAME for=memory budget=M slots=N age=Ns dir=PATH|- command=TEXT|- bounded-run: check budget=M slots=N headroom=M starts=yes|no A '-' stands for a value that the check cannot read or count here. cpus is @@ -62,7 +64,8 @@ cpus-headroom and cpus-used, and at least 0. starts=yes means that a run of that budget would start now. It is not a reservation. fits is the largest budget whose memory, with its own headroom, is free now, in steps of 256M, -or 0M when none is; it does not count slots or the line. +or 0M when none is; it does not count slots or the line. A waiting line is +a command that holds its slots and waits for memory. A normal call sets none of the environment variables below. A person can set the tuning values in the profile of the shell, so that every session @@ -93,6 +96,7 @@ import fcntl import hashlib +import json import math import os import random @@ -512,6 +516,77 @@ def running_commands(directory: str) -> Optional[int]: return running +def wait_file_name(unit: str) -> str: + """Returns the name of the file that a bounded command holds locked while it waits for memory.""" + return "wait-%s.lock" % unit + + +def write_wait_file( + directory: str, unit: str, budget_mib: int, slots: int, working: str, command: Sequence[str], + now: Callable[[], float] = time.time, +) -> int: + """Writes the wait file of the unit and holds it locked. Returns its descriptor for remove_wait_file. + + The file names the budget, the slots that the command holds, the directory, and the command, for memory_waiters to read. + It holds no process ID. + """ + path = os.path.join(directory, wait_file_name(unit)) + partial = path + ".new" + text = json.dumps({"budget": budget_mib, "slots": slots, "since": now(), "dir": working, "command": list(command)}) + descriptor = _open_lock_file(partial) + try: + fcntl.flock(descriptor, fcntl.LOCK_EX) + os.write(descriptor, text.encode("utf-8")) + # A reader takes a wait file that no process holds as one that a killed wrapper left, so the file gets its name only once it is locked and full. + os.rename(partial, path) + except OSError as error: + os.close(descriptor) + raise WrapperError("cannot write %s: %s" % (path, error.strerror if error.strerror else error)) + return descriptor + + +def remove_wait_file(directory: str, unit: str, descriptor: int) -> None: + try: + os.unlink(os.path.join(directory, wait_file_name(unit))) + except OSError: + pass + os.close(descriptor) + + +def memory_waiters(directory: str, now: Callable[[], float] = time.time) -> List[str]: + """Returns one text for each bounded command that waits for memory, as the fields of a check line. + + Returns nothing when the directory cannot be read. A wait file that no process holds is left over from a wrapper + that was killed, and is removed. + """ + try: + names = sorted(os.listdir(directory)) + except OSError: + return [] + waiters = [] + for name in names: + if not (name.startswith("wait-") and name.endswith(".lock")): + continue + path = os.path.join(directory, name) + descriptor = _try_lock(path) + if descriptor is not None: + try: + os.unlink(path) + except OSError: + pass + os.close(descriptor) + continue + try: + fields = json.loads(_read_text(path) or "") + waiters.append("unit=%s for=memory budget=%dM slots=%d age=%ds dir=%s command=%s" % ( + name[len("wait-"):-len(".lock")], fields["budget"], fields["slots"], max(0, int(now() - fields["since"])), + _one_line(fields["dir"]) or "-", _one_line(" ".join(fields["command"]))[:HELD_COMMAND_CHARACTERS] or "-")) + except (ValueError, KeyError, TypeError): + # The waiter removed its file between the lock and the read. + continue + return waiters + + def _lock_error(path: str, error: OSError) -> WrapperError: error_text = error.strerror if error.strerror else str(error) return WrapperError("cannot lock %s: %s. The lock directory must be on a filesystem that supports file locks." % (path, error_text)) @@ -703,6 +778,7 @@ def wait_for_memory( lock: Callable[[], None] = lambda: None, unlock: Callable[[], None] = lambda: None, clock: Callable[[], float] = time.monotonic, + on_wait: Callable[[], None] = lambda: None, ) -> None: """Returns when the free memory, less the unused budgets of running commands and the headroom, holds the budget. @@ -711,6 +787,7 @@ def wait_for_memory( read_running gives the number of bounded commands that run, or None where it is not known: the call waits. Returns at once when the free memory cannot be read. read_outstanding gives None where the unused budgets cannot be counted. The call returns with lock held: the caller calls unlock once the new command counts in read_outstanding. + on_wait is called, with lock released, before each sleep of the wait. """ def read() -> Tuple[Optional[int], Optional[int]]: # A running command that grows or shrinks during the reads moves both values. The larger of the two unused @@ -753,6 +830,7 @@ def state(available: int, outstanding: Optional[int]) -> str: log(state(available, outstanding) + ("; waiting" if next_line is None else "; still waiting after %ds" % waited)) next_line = waited + interval_seconds unlock() + on_wait() sleep(poll_seconds) @@ -1588,6 +1666,16 @@ def read_running() -> Optional[int]: return max(running or 0, found) return running + try: + working = os.getcwd() + except OSError: + working = "-" + waiting: List[int] = [] + + def mark_waiting() -> None: + if not waiting: + waiting.append(write_wait_file(directory, unit, wait_budget, needed, working, options.command)) + reported: Optional[int] = None reservation = reserve(directory, count, needed, exclusive_files, settings.poll_seconds) try: @@ -1597,12 +1685,15 @@ def read_running() -> Optional[int]: headroom_mib=headroom_mib(settings.total_mib, wait_budget, most), read_available=read_available, read_outstanding=read_outstanding, read_running=read_running, lock=lock_memory, unlock=unlock_memory, + on_wait=mark_waiting, ) except NoMemory as stop: log("no-memory budget=%dM free=%dM headroom=%dM fits=%dM" % ( wait_budget, stop.available - (stop.outstanding or 0), headroom_mib(settings.total_mib, wait_budget, most), largest_fit_mib(stop.available, stop.outstanding, settings.total_mib, most))) raise + while waiting: + remove_wait_file(directory, unit, waiting.pop()) code, maxrss = run_command(command, child_environ, on_start) tracked = trackers[0].finish() if trackers else None held = trackers[0].held if trackers else None @@ -1617,6 +1708,8 @@ def read_running() -> Optional[int]: os.close(run_lock.pop()) if capped: reported = read_peak_file(peak_file) + while waiting: + remove_wait_file(directory, unit, waiting.pop()) reservation.release() exact = reported is not None @@ -1686,6 +1779,8 @@ def mib(value: Optional[int]) -> str: "-" if available is None else "%dM" % largest_fit_mib(available, outstanding, settings.total_mib, most))) for held in held_commands(scopes, ()) if scopes is not None else []: log("check held " + held) + for waiter in memory_waiters(directory): + log("check waiting " + waiter) for budget in budgets: needed = slots_needed(budget, settings.slot_mib, count) headroom = headroom_mib(settings.total_mib, min(budget, most), most) diff --git a/skills/bounding-heavy-commands/scripts/tests/test_bounded_run.py b/skills/bounding-heavy-commands/scripts/tests/test_bounded_run.py index 38b2ee7..d30157e 100644 --- a/skills/bounding-heavy-commands/scripts/tests/test_bounded_run.py +++ b/skills/bounding-heavy-commands/scripts/tests/test_bounded_run.py @@ -724,6 +724,70 @@ def test_a_run_file_that_no_process_holds_is_removed_and_not_counted(self): self.assertFalse(os.path.exists(path)) +class MemoryWaiterTest(unittest.TestCase): + def setUp(self): + self.base = tempfile.TemporaryDirectory() + self.addCleanup(self.base.cleanup) + self.directory = self.base.name + + def mark(self, unit, budget=6144, slots=3, working="/work/project", command=("make", "test"), since=1000.0): + descriptor = bounded_run.write_wait_file(self.directory, unit, budget, slots, working, list(command), now=lambda: since) + self.addCleanup(lambda: os.close(descriptor) if self.is_open(descriptor) else None) + return descriptor + + @staticmethod + def is_open(descriptor): + try: + os.fstat(descriptor) + except OSError: + return False + return True + + def waiters(self, now=1042.0): + return bounded_run.memory_waiters(self.directory, now=lambda: now) + + def test_names_the_budget_the_slots_the_age_the_directory_and_the_command_of_each_waiter(self): + self.mark("bounded-run-5-000005") + self.mark("bounded-run-6-000006", budget=2048, slots=1, working="/other", command=("tool",), since=1040.0) + self.assertEqual(self.waiters(), [ + "unit=bounded-run-5-000005 for=memory budget=6144M slots=3 age=42s dir=/work/project command=make test", + "unit=bounded-run-6-000006 for=memory budget=2048M slots=1 age=2s dir=/other command=tool", + ]) + + def test_no_waiter_after_the_wait_file_is_removed(self): + descriptor = self.mark("bounded-run-5-000005") + bounded_run.remove_wait_file(self.directory, "bounded-run-5-000005", descriptor) + self.assertEqual(self.waiters(), []) + self.assertEqual(os.listdir(self.directory), []) + + def test_a_wait_file_that_no_process_holds_is_removed_and_not_named(self): + os.close(self.mark("bounded-run-5-000005")) + self.assertEqual(self.waiters(), []) + self.assertEqual(os.listdir(self.directory), []) + + def test_the_wait_file_holds_no_process_id(self): + self.mark("bounded-run-5-000005") + text = Path(self.directory, bounded_run.wait_file_name("bounded-run-5-000005")).read_text() + self.assertNotIn(str(os.getpid()), text) + + def test_a_directory_that_cannot_be_read_gives_no_waiter(self): + self.assertEqual(bounded_run.memory_waiters(os.path.join(self.directory, "missing")), []) + + def test_a_directory_and_a_command_with_line_breaks_stay_on_one_line(self): + self.mark("bounded-run-5-000005", working="/a\nb", command=("sh", "-c", "a\nb")) + self.assertEqual(self.waiters(), ["unit=bounded-run-5-000005 for=memory budget=6144M slots=3 age=42s dir=/a b command=sh -c a b"]) + + def test_a_long_command_is_cut(self): + self.mark("bounded-run-5-000005", command=("x" * 500,)) + self.assertEqual(len(self.waiters()[0].split(" command=")[1]), bounded_run.HELD_COMMAND_CHARACTERS) + + def test_slot_and_run_files_are_not_waiters(self): + for name in ("slot-000.lock", bounded_run.run_file_name("bounded-run-1-000001")): + descriptor = bounded_run._try_lock(os.path.join(self.directory, name)) + self.addCleanup(os.close, descriptor) + self.assertEqual(self.waiters(), []) + + class MemoryReadTest(unittest.TestCase): def test_reads_the_available_memory_of_meminfo(self): text = "MemTotal: 32000000 kB\nMemFree: 1000000 kB\nMemAvailable: 8388608 kB\n" @@ -927,6 +991,18 @@ def read(): ) self.assertEqual(events, ["lock", "read", "unlock", "sleep", "lock", "read"]) + def test_calls_on_wait_before_each_sleep_with_the_lock_released(self): + events = [] + readings = [1000, 1000, 5000] + bounded_run.wait_for_memory( + 4096, 300, 5.0, read_available=lambda: readings.pop(0), sleep=lambda seconds: events.append("sleep"), + lock=lambda: events.append("lock"), unlock=lambda: events.append("unlock"), on_wait=lambda: events.append("wait"), + ) + self.assertEqual(events, ["lock", "unlock", "wait", "sleep", "lock", "unlock", "wait", "sleep", "lock"]) + + def test_does_not_call_on_wait_when_the_memory_is_free_at_once(self): + bounded_run.wait_for_memory(4096, 300, 5.0, read_available=lambda: 8000, on_wait=lambda: self.fail("waits")) + def test_reads_the_unused_budgets_before_and_after_the_free_memory(self): events = [] @@ -2080,6 +2156,30 @@ def test_a_scope_that_never_appears_frees_the_memory_check_after_a_time_limit(se self.assertEqual(free, {"before the scope": False, "before the limit": False, "at the start": True}) self.assertLess(time.monotonic() - started, 5) + def test_the_wait_file_names_the_held_slots_while_the_command_waits_and_is_gone_when_it_starts(self): + seen = [] + + def wait(*arguments, **keywords): + keywords["on_wait"]() + keywords["on_wait"]() + seen.append(bounded_run.memory_waiters(self.directory)) + + def fake_run_command(command, environ, on_start=None): + seen.append(bounded_run.memory_waiters(self.directory)) + return 0, 0 + + options = bounded_run.Options() + options.memory = "1G" + options.command = [sys.executable, "-c", "pass"] + with mock.patch.object(bounded_run, "probe_cap", return_value=None), \ + mock.patch.object(bounded_run, "wait_for_memory", wait), \ + mock.patch.object(bounded_run, "run_command", fake_run_command): + self.assertEqual(bounded_run.bounded(options, self.environ), 0) + self.assertEqual(len(seen[0]), 1) + self.assertIn(" for=memory budget=1024M slots=1 ", seen[0][0]) + self.assertEqual(seen[1], []) + self.assertEqual([name for name in os.listdir(self.directory) if name.startswith("wait-")], []) + def test_the_wait_counts_no_unused_budget_without_a_cap(self): self.assertIsNone(self.wait_arguments(None)["read_outstanding"]()) @@ -2631,6 +2731,13 @@ def free_slots(*arguments, **keywords): self.assertEqual(events[-1], "slots") self.assertIn("probe", events) + def test_names_each_command_that_waits_for_memory_also_without_a_cap(self): + descriptor = bounded_run.write_wait_file(self.directory, "bounded-run-5-000005", 6144, 3, "/w", ["tool"], now=time.time) + self.addCleanup(os.close, descriptor) + lines = self.check(["2G"]) + self.assertRegex(lines[1], r"^check waiting unit=bounded-run-5-000005 for=memory budget=6144M slots=3 age=\d+s dir=/w command=tool$") + self.assertTrue(lines[2].startswith("check budget=2048M "), lines) + def test_no_held_line_without_a_cap(self): self.hold("slot-000.lock") self.assertFalse(any(line.startswith("check held") for line in self.check(["2G"]))) @@ -3003,6 +3110,38 @@ def test_a_waiting_command_holds_no_slot_and_is_not_overtaken(self): self.assertLess(order.index(["end", "holder"]), order.index(["start", "large"])) self.assertLess(order.index(["start", "large"]), order.index(["start", "small"])) + def memory_waiter(self): + """Starts a holder, then a large command whose budget does not fit in the free memory, with five slots of 2 GiB. + + Each wrapper reads 6000 MiB of free memory and keeps a headroom of 1228 MiB, so the large command needs 3 slots and + waits for memory while the holder runs. Returns the release file, the holder, and the large command, once the large + command waits for memory. + """ + self.environ.update({"BOUNDED_RUN_TOTAL_MIB": "12288", "BOUNDED_RUN_FREE_MIB": "6000"}) + release = os.path.join(self.base.name, "release") + holder = self.worker("holder", "2G", hold="300", release=release) # runs until the test writes the release file + self.wait_for_event("start", "holder") + large = self.worker("large", "6G", hold="0") # does not start: its budget never fits in the free memory + self.wait_for_stderr(large, "; waiting") + # The wrapper logs the wait before it writes its wait file. + deadline = time.time() + 60 + while not any(name.startswith("wait-") and name.endswith(".lock") for name in os.listdir(self.environ["BOUNDED_RUN_LOCK_DIR"])): + self.assertLess(time.time(), deadline, "the large command wrote no wait file") + time.sleep(0.02) + return release, holder, large + + def test_a_check_names_the_command_that_waits_for_memory(self): + release, holder, large = self.memory_waiter() + done = subprocess.run([sys.executable, WRAPPER, "--check"], env=self.environ, capture_output=True, text=True, timeout=60) + self.assertRegex(done.stderr, r"\nbounded-run: check waiting unit=bounded-run-%d-[0-9a-f]{6} for=memory budget=6144M slots=3 age=\d+s dir=%s command=%s -c import os, sys, time def event" % ( + large.pid, re.escape(os.getcwd()), re.escape(sys.executable))) + # The holder holds 1 slot and the waiter 3 of the 5. + self.assertIn(" free-slots=1 ", done.stderr) + Path(release).write_text("") + self.assertEqual(holder.wait(timeout=60), 0) + self.assertEqual(large.wait(timeout=60), bounded_run.EXIT_NO_MEMORY) + self.assertEqual([name for name in os.listdir(self.environ["BOUNDED_RUN_LOCK_DIR"]) if name.startswith("wait-")], []) + def test_a_hard_kill_of_the_wrapper_frees_the_slots_at_once(self): holder = self.worker("holder", "4G", hold="300") # runs until the kill below ends it self.wait_for_event("start", "holder")