Skip to content

feat(grammar): matrices and tensors with shape checks - #47

Open
Trosper3 wants to merge 86 commits into
ReduxISU:mainfrom
Trosper3:feat/30-tensors
Open

Trosper3 wants to merge 86 commits into
ReduxISU:mainfrom
Trosper3:feat/30-tensors

Conversation

@Trosper3

Copy link
Copy Markdown
Contributor

Adds matrices and tensors: a definition can declare a rectangular grid of ints, numbers or strings, with its sizes checked across parts.

Stack: builds on #28 and should merge after it. Review only the last three commits; the diff shrinks as #11–#28 merge. It merges cleanly with #39 (all 401 tests pass on the combination).

Features

  • C is matrix int declares a 2-dimensional grid, and T is tensor 3 int one with any number of dimensions (tensor 1 is a vector; matrix is tensor 2). The element type is int, number or string.
  • d is list int (also list number, list string) checks a list whose elements all have one type. A plain d is list still accepts any list. This is the form chosen for "list with a size", since is list int wasn't valid SPADE before.
  • Instances are nested lists, row by row: ((1,2,3),(4,5,6)) is 2 rows of 3. No new separators.
  • By default sizes come from the instance, and every row at the same depth must be the same length: ((1,2,3),(4,5)) fails with row 2 of C has 2 values, expected 3.
  • size names each dimension, outermost first, with a name or a whole-number constant: {(C,d) | C is matrix int size m n, d is list int size m} (0-1 ILP), {G | G is matrix int size 9 9} (Sudoku-shaped), {G | G is matrix int size n n} (square). A name must have the same value everywhere it is used, in one part or across parts: C has 3 rows but m is 4 (from d).
  • A size named after a part that holds a whole number takes its value: {(n,G) | n is int, G is matrix int size n n}.
  • Errors are SpadeParseExceptions with Part and Element set, naming the position from 1: element (2,1) of C is not an int, row (2,2) of T has 1 value, expected 2, row 2 of C is not a list.
  • Empty cases: () as a matrix has shape 0×0 (a dimension after an empty one can't be read, so it's reported as 0 and not checked), ((),()) is 2×0, ((7)) is 1×1.
  • Definitions that can't be checked are rejected when the parser is made, with a FormatException: set int, tensor 0 int, matrix list, matrix int size m (wrong number of sizes), size 2.5 3, and a shape after subset or inside union.
  • parser.GetTensor("C") (and TryGetTensor) returns a Tensor: ElementType, Shape, Rank, Count, Values, elements by index (t[1, 2], t.GetInt(1, 2), t.GetNumber(1, 2)), ToCollection() and ToString(). It is immutable, and equality and hashing agree (element type, shape and each element's text). parser["C"] still returns the nested lists.
  • The definition description reports these parts: PartKind.Tensor for a matrix or tensor (a matrix is rank 2), and ElementType, Rank and Sizes (as written, e.g. ["m","n"] or ["9","9"], empty when free) on DefinitionPart, for Redux to label and check each part (Redux#851).
  • matrix, tensor and size are now keywords, so they can't name a part. No definition string in Redux uses them as names.
  • Documented in DefinitionString.md, InstanceString.md and example.md. The grammar was regenerated with ANTLR 4.13.2.

Issue Handling

Since d7e982e, an unorderedcross membership check rejected every
normal undirected edge such as {a,b}. The added line checked whether
each node inside the edge was a list, and plain nodes never are.

The check now looks at the edge itself: a plain value is not an
edge, so it fails the constraint instead of crashing while being
enumerated. List edges such as (a,b) are still accepted, as in the
released 0.1.2, because existing grammars for directed graphs rely
on it.

Closes ReduxISU#12
An unorderedcross member is an unordered pair, so it must be a set
such as {a,b}. A list such as (a,b) is an ordered pair, which is what
cross describes, so it is now rejected here too. This is what
d7e982e set out to do.

Grammars for directed graphs that use unorderedcross with list edges
need to switch to cross.

Refs ReduxISU#12
"N is int" checked nothing at all: int reaches the constraint as a set
descriptor, and the "is" branch only handled the set, list and int
keywords. So abc, {1} and (1,2) all passed. It now requires a single
value that parses as an integer.

"S is set" accepted a plain value, because a value is not ordered. It
now requires an unordered collection.

"is list" already rejected values and sets; it is now covered by tests.

Closes ReduxISU#10
Replaces the substring-based reader in UtilCollection(string) with
InstanceScanner, which walks the input once.

Whitespace (ReduxISU#2): spaces, tabs and line breaks around { } ( ) , and at
either end of the input are skipped, so {2, 3} now holds 3 and a
trailing new line no longer breaks an instance. Inside a value,
whitespace next to [ ] ; : = is dropped, plain spaces are kept (New
York), and a run with a tab or line break becomes one space. These
match the rules Redux applies before parsing today.

Errors (ReduxISU#4): malformed input throws SpadeParseException with the
position, what was expected and what was found, e.g. "expected , or }
at position 5, found end of input" for {1,2. Input that used to be
misread without an error, such as {1,2 or {1,2}junk, is now rejected.

Time and depth (ReduxISU#7): parsing is linear in the input's length, and
ToString builds its text in one buffer. Nesting deeper than 256 levels
is rejected instead of overflowing the stack and crashing the process.
40,000 elements took 104 seconds before; 100,000 now take well under
a second.

Closes ReduxISU#2
Closes ReduxISU#7
Refs ReduxISU#4
…tion

Input that is well formed but doesn't fit the definition string threw
whatever happened to fail: InvalidOperationException ("Cannot index
into a set"), ArgumentOutOfRangeException, or a bare Exception whose
message said PLACEHOLDER. All of these now throw SpadeParseException
with a message that names what failed:

- A format such as (N,E) checks that it got a list of exactly that
  many elements. A longer list used to be accepted and the extra
  elements dropped.
- Constraint failures quote the constraint as written, e.g.
  "Constraint E subset N unorderedcross N failed: {1,9} is not a
  member". Long values are shortened.
- subset on a plain value, and cross on anything that isn't a pair,
  fail the constraint instead of crashing.

Closes ReduxISU#4
A set drops repeated elements, so {3,3,2,2} was read as {3,2}. Problems
defined on multisets, such as Partition and Subset Sum, then solved a
different instance from the one written.

A definition string can now mark a collection with "is multiset":

    {N | N is multiset, N subset int}

reads {3,3,2,2} as four numbers. It applies before the other
constraints wherever it is written. Without it, braces still mean a
set, so nothing changes for existing definitions or callers.

To make this possible, a set read from an instance string keeps its
elements as written, and AsMultiset() turns them into a multiset.
Multisets count every occurrence in Count(), enumeration and
ToString(), compare by count in any order, and have order-independent
hash codes. Union and Intersect use the larger and smaller count. New
CountOf() and IsSubsetOf() compare by count, so {7,7} is in {7,7,1}
but not in {7,1}. A set and a multiset are never equal, and "is set"
rejects a multiset.

Adds the multiset keyword to coll.g4; the lexer is regenerated with
ANTLR 4.13.2, the version that generated the existing files.

Closes ReduxISU#1
…stance

StringParser.Definition reads the definition string on first use and returns
its outline and each named part's kind, object, path and constraints, so a
caller can build an editor for a problem before any instance exists.
…xception

SpadeParseException gets Part, Element, MissingValue and ExpectedIn, so a
caller can find the part an error belongs to without reading the message.
Failures inside a nested set belong to the outer part and member. Messages
are unchanged.

Closes ReduxISU#19
"x is int", "S subset int" and UtilCollection.parseInt used the machine's culture, so a culture with another minus sign could refuse "-5". They now share IntValue, which parses with CultureInfo.InvariantCulture as NumberValue does for number (ReduxISU#23).
Whitespace only: dotnet format SPADE.slnx on this branch's code.
Whitespace only: dotnet format SPADE.slnx on this branch's code.
Whitespace only: dotnet format SPADE.slnx on this branch's code.
Whitespace only: dotnet format SPADE.slnx on this branch's code.
Whitespace only: dotnet format SPADE.slnx on this branch's code.
Whitespace only: dotnet format SPADE.slnx on this branch's code.
# Conflicts:
#	Documentation/utilcollection.md
StringParser kept ANTLR's default error listener on the definition
string, which printed the error to the console and recovered with a best
guess, so "{N | N is set" (missing brace), "{N | N is set#}" and
"{N | N is set}}" were accepted. ReduxISU#4 rules out best-guess parsing, and
Definition already rejected the same strings.

The throwing listener moves out of DefinitionDescription into its own
internal class, with one Parse method both use. The StringParser
constructor now reads the definition string through it, so a malformed
one throws the same FormatException as Definition, naming the string and
the position of the first error, before any instance is parsed. Nothing
is printed. parse() walks the tree read at construction instead of
reading the definition string again on every call.
"C is matrix int", "T is tensor 3 number" and "d is list int" read nested
lists row-major, check that every row at the same depth has the same length
and every element has the element type, and return the part as a Tensor
(element type, shape, row-major elements). "size" binds each dimension to a
name or a whole-number constant; a name keeps one value across the parts of
an instance, and a size named after an int part takes its value. Errors name
the part and the row or element, e.g. "row 3 of C has 2 values, expected 3"
and "C has 3 rows but m is 4 (from d)". The definition description reports
each such part's element type, rank and sizes. (ReduxISU#30)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Matrices and tensors: declare n-dimensional values with shape checks

1 participant