Repository navigation
Add Override, to replace the task which provides a key #6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
sengjea
wants to merge
1
commit into
master
Choose a base branch
from
override-bindings
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+982
−1
Open
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| package taskgraph | ||
|
|
||
| import ( | ||
| "context" | ||
| "errors" | ||
| "fmt" | ||
| "sort" | ||
| "strings" | ||
| ) | ||
|
|
||
| var ( | ||
| // ErrUnknownOverride is returned by RunWithOverrides and CheckWithOverrides if an override is | ||
| // given for a key which no task in the graph provides. This is almost always a stale override, | ||
| // left over from a task which has since been renamed or removed. | ||
| ErrUnknownOverride = errors.New("override of a key which no task provides") | ||
|
|
||
| // ErrPartialOverride is returned by RunWithOverrides and CheckWithOverrides if overrides cover | ||
| // some, but not all, of the keys which a task provides. The task would still have to run, with | ||
| // all of its effects, only to have some of its results discarded, which is very unlikely to be | ||
| // what was intended. Override all of a task's keys, or none of them. | ||
| ErrPartialOverride = errors.New("override of only some of the keys which a task provides") | ||
| ) | ||
|
|
||
| // RunWithOverrides runs the graph, as Graph.Run does with the inputs, with the tasks which provide | ||
| // the keys of the overrides replaced by them. It is the way to substitute what the tasks of a graph | ||
| // would otherwise create for themselves, such as clients of other services, in a test: | ||
| // | ||
| // taskgraph.RunWithOverrides(ctx, graph, inputs, []taskgraph.Binding{keyClient.Bind(fakeClient)}) | ||
| // | ||
| // The task providing the key is not run. If the task provides other keys as well, they must all be | ||
| // overridden too (ErrPartialOverride), so that the real task's effects can never happen unnoticed. | ||
| // Tasks which have no other purpose than to produce the inputs of a task which is not run are still | ||
| // run, as are tasks which depend on the overridden key; they see the override as if the task which | ||
| // it replaces had produced it. The overrides are included in the Binder returned, as the replaced | ||
| // tasks' results would have been, and inputs which only a replaced task needed are not required. | ||
| // | ||
| // An override may be absent, which replaces the task with a failure: | ||
| // | ||
| // keyClient.BindError(errUnreachable) | ||
| // | ||
| // Overriding a key which no task provides is an error (ErrUnknownOverride), as is overriding the same | ||
| // key twice, or overriding a key which is also given as an input. An input for a key which a task | ||
| // provides is still an error (ErrDuplicateBinding), so a key can only ever be replaced on purpose. | ||
| // | ||
| // Overrides apply to the graph's own tasks. A graph run as a task with Graph.AsTask is replaced as a | ||
| // whole, by overriding every key it exposes, but its tasks can't be overridden individually. | ||
| // | ||
| // This is a function, not a method of Graph, so that the Graph interface, which other code may | ||
| // implement, is not changed. It only works on graphs made by New. | ||
| func RunWithOverrides( | ||
| ctx context.Context, | ||
| g Graph, | ||
| inputs, overrides []Binding, | ||
| ) (Binder, error) { | ||
| graph, err := asGraph(g) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| replaced, err := graph.withOverrides(inputs, overrides) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| return replaced.Run(ctx, inputs...) | ||
| } | ||
|
|
||
| // CheckWithOverrides is Graph.Check, with the overrides RunWithOverrides would be given: it finds | ||
| // unknown, duplicated and partial overrides, and which inputs are still needed, without running | ||
| // anything. | ||
| func CheckWithOverrides(g Graph, inputs, overrides []Binding) error { | ||
| graph, err := asGraph(g) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| replaced, err := graph.withOverrides(inputs, overrides) | ||
| if err != nil { | ||
| return err | ||
| } | ||
| return replaced.Check(inputs...) | ||
| } | ||
|
|
||
| // An Overrider runs graphs with one set of overrides. It is for something which runs several graphs, | ||
| // such as an operator with a graph to create and one to delete, and which can only be given one set: | ||
| // each graph is run with those of the overrides which are for keys it provides, and Check makes sure | ||
| // that every override is for some graph. Run does not report an override which is for none of the | ||
| // graphs, as it can't tell that from one for another graph, so Check is how to find a stale one. | ||
| type Overrider struct { | ||
| overrides []Binding | ||
| } | ||
|
|
||
| // NewOverrider returns an Overrider with the overrides. | ||
| func NewOverrider(overrides ...Binding) *Overrider { | ||
| return &Overrider{overrides: overrides} | ||
| } | ||
|
|
||
| // Run runs the graph with the inputs, and those of the Overrider's overrides which are for keys the | ||
| // graph provides. Its signature is that of a function for running a graph, so it can be handed to | ||
| // code which runs graphs. | ||
| func (o *Overrider) Run(ctx context.Context, g Graph, inputs ...Binding) (Binder, error) { | ||
| graph, err := asGraph(g) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| var own []Binding | ||
| for _, override := range o.overrides { | ||
| if graph.allProvided.Contains(override.ID()) { | ||
| own = append(own, override) | ||
| } | ||
| } | ||
| return RunWithOverrides(ctx, g, inputs, own) | ||
| } | ||
|
|
||
| // Check returns an error if any of the overrides is for a key which none of the graphs provide, which | ||
| // is nearly always one left behind by a task which has been renamed or removed. | ||
| func (o *Overrider) Check(graphs ...Graph) error { | ||
| var unknown []string | ||
| for _, override := range o.overrides { | ||
| provided := false | ||
| for _, g := range graphs { | ||
| if graph, err := asGraph(g); err == nil && graph.allProvided.Contains(override.ID()) { | ||
| provided = true | ||
| break | ||
| } | ||
| } | ||
| if !provided { | ||
| unknown = append(unknown, override.ID().String()) | ||
| } | ||
| } | ||
| if len(unknown) > 0 { | ||
| sort.Strings(unknown) | ||
| return fmt.Errorf("%w: %s", ErrUnknownOverride, strings.Join(unknown, ", ")) | ||
| } | ||
| return nil | ||
| } | ||
|
|
||
| func asGraph(g Graph) (*graph, error) { | ||
| graph, ok := g.(*graph) | ||
| if !ok { | ||
| return nil, fmt.Errorf( | ||
| "overrides can only be applied to a Graph made by taskgraph.New, not %T", | ||
| g, | ||
| ) | ||
| } | ||
| return graph, nil | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,98 @@ | ||
| package taskgraph | ||
|
|
||
| import ( | ||
| "context" | ||
| "fmt" | ||
| "sort" | ||
| "strings" | ||
|
|
||
| set "github.com/deckarep/golang-set/v2" | ||
| ) | ||
|
|
||
| // withOverrides returns a graph like g, but with each task whose keys are all overridden replaced by a | ||
| // task which provides the overriding bindings. The graph is g itself if there are no overrides. | ||
| // | ||
| // A replacement has no dependencies, so the tasks which fed the task it replaces have nothing to | ||
| // signal, and inputs which only the replaced task needed are not required. Those tasks still run, as | ||
| // the graph can't know that its caller doesn't want their results. | ||
| // | ||
| // An override is an error if it is for a key which is also an input or is overridden twice, as that | ||
| // is a mistake in which one is meant; if no task provides its key, which is nearly always a task | ||
| // which has been renamed or removed; or if it is for only some of the keys which a task provides, | ||
| // as the task would then have to run for the others and its effects could not be ruled out. | ||
| func (g *graph) withOverrides(inputs, overrides []Binding) (*graph, error) { | ||
| if len(overrides) == 0 { | ||
| return g, nil | ||
| } | ||
|
|
||
| given := set.NewSet[ID]() | ||
| for _, input := range inputs { | ||
| given.Add(input.ID()) | ||
| } | ||
| overriddenBy := map[ID]Binding{} | ||
| for _, o := range overrides { | ||
| if given.Contains(o.ID()) { | ||
| return nil, wrapStackErrorf( | ||
| "override of a key which is also an input: %w: %q", ErrDuplicateBinding, o.ID()) | ||
| } | ||
| if _, twice := overriddenBy[o.ID()]; twice { | ||
| return nil, wrapStackErrorf("key overridden twice: %w: %q", ErrDuplicateBinding, o.ID()) | ||
| } | ||
| overriddenBy[o.ID()] = o | ||
| } | ||
|
|
||
| var unknown []string | ||
| for id := range overriddenBy { | ||
| if !g.allProvided.Contains(id) { | ||
| unknown = append(unknown, id.String()) | ||
| } | ||
| } | ||
| if len(unknown) > 0 { | ||
| sort.Strings(unknown) | ||
| return nil, wrapStackErrorf("%w: %s", ErrUnknownOverride, strings.Join(unknown, ", ")) | ||
| } | ||
|
|
||
| var tasks []TaskSet | ||
| var partial []string | ||
| for _, t := range g.tasks { | ||
| var covered []Binding | ||
| var uncovered []string | ||
| for _, id := range t.Provides() { | ||
| if o, ok := overriddenBy[id]; ok { | ||
| covered = append(covered, o) | ||
| } else { | ||
| uncovered = append(uncovered, id.String()) | ||
| } | ||
| } | ||
| switch { | ||
| case len(covered) == 0: | ||
| tasks = append(tasks, t) | ||
| case len(uncovered) == 0: | ||
| tasks = append(tasks, replacement(t, covered)) | ||
| default: | ||
| partial = append(partial, fmt.Sprintf( | ||
| "task %s also provides [%s], which are not overridden", | ||
| t.Name(), strings.Join(uncovered, ", "))) | ||
| } | ||
| } | ||
| if len(partial) > 0 { | ||
| sort.Strings(partial) | ||
| return nil, wrapStackErrorf("%w: %s", ErrPartialOverride, strings.Join(partial, "; ")) | ||
| } | ||
|
|
||
| replaced, err := New(g.name, WithTasks(tasks...), WithTracer(g.tracer), WithLogger(g.logger)) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| return replaced.(*graph), nil | ||
| } | ||
|
|
||
| // replacement is a task in place of t which provides the overrides for all of the keys of t. | ||
| func replacement(t Task, overrides []Binding) Task { | ||
| return NewTask( | ||
| t.Name()+" (overridden)", | ||
| func(context.Context, Binder) ([]Binding, error) { return overrides, nil }, | ||
| nil, | ||
| t.Provides(), | ||
| ) | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.