Skip to content

fix: preserve dependency order and resource ownership in experiment execution - #342

Open
Cookie0103 wants to merge 1 commit into
ServiceNow:mainfrom
Cookie0103:fix/dependency-aware-experiment-execution
Open

Cookie0103 wants to merge 1 commit into
ServiceNow:mainfrom
Cookie0103:fix/dependency-aware-experiment-execution

Conversation

@Cookie0103

Copy link
Copy Markdown

A dependency-ordered study can currently run children before parents in the sequential backend, submit part of an invalid Ray graph, and associate Ray results with the wrong experiment when references finish out of order. Shared agent resources can also be prepared repeatedly or left open after partial preparation and cleanup failures.

This repairs the experiment execution lifecycle across graph validation, dispatch, monitoring, and resource cleanup:

  • Validate IDs, missing dependencies, cycles, and episode budgets before preparing agents or experiment directories; use a stable, iterative topological order for sequential and Ray execution.
  • Keep result identity attached to each Ray reference and enforce each episode's own runtime budget. Dependency/worker queue time is excluded, and cancellation escalates once after a monotonic grace period.
  • Prepare and close shared agent objects once, attempt all cleanup callbacks, preserve the primary failure, and clean partially initialized Ray runtimes only when owned by this invocation.
  • Reject dependency graphs on joblib before side effects; retain its independent-experiment behavior.

Validation on the frozen lock (Python 3.12.13, Ray 2.48.0, pytest 7.3.2): 133 passed, 1 skipped, 6 deselected in the experiment suite. The 120 new cases cover deterministic failure regressions, native ExpArgs persistence/relaunch, and real local Ray execution, including failed-parent isolation and per-episode deadlines. The independent review rerun passed all 112 deterministic cases. Black 25.1.0, darglint, compileall, and diff checks pass.

Browser/provider-dependent tests were excluded; no real LLM or remote benchmark was exercised. Ray runtime deadlines still depend on task-state availability; sequential/joblib deadlines remain unenforced. These limitations and the changed joblib/shared-agent behavior are documented.

Prepared with AI assistance.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant