[pietercolpaert]

Status: discussion draft, not meant to be merged as-is. Every open point is also marked as an Issue: in the spec text. This draft introduces breaking changes, hence why we’d call this 2.0. A client can however choose to be compliant to both LDES 1.0 and LDES 2.0.

Files: eventstreams.bs (consumer spec, Level: 2), vocabulary.bs, vocabulary.ttl, context.jsonld, server-primer.bs.

Existing convention: eventstreams.bs only uses MUST/SHOULD/MAY for client behaviour. Producer rules moved to server-primer.bs.

What changes

1. Members are RDF Messages, pages are RDF Message logs

• Each member is one RDF Message. Each page with members is an RDF Message log (RDF Messages, W3C RSP CG).
• The Member Extraction Algorithm and the TREE profile are removed: the message boundary delimits the member.
• A page can have an optional description message (relations, ldes:immutable, and in the root page the LDES context). It is the first message, and a client recognises it by content: it has the page IRI as a subject, or describes the LDES. No <> a ldes:Page triple is needed (ldes:Page follows from the domain of ldes:relation).
• ldes:immutable can now also be stated about a linked page in the
  description message of the page that links to it ( ldes:immutable
  true in P1), so a client can stop revalidating before fetching it.
• Pages without members can be any RDF serialization and do not need RDF Messages support.
• Empty RDF messages are disallowed. A client returns an error on one.
• Pages can be much bigger, because clients parse them as a stream.
• This automatically adds support for RDF Messages based formats, such as Jelly-RDF (cc @Ostrzyciel)
• This makes LDES an RDF Message Stream Profile

2. Everything is in the LDES namespace now. Alignments to TREE and DCAT where relevant

• tree: is removed from the spec text. New terms: ldes:page, ldes:relation, ldes:shape, ldes:member, ldes:Page, ldes:Relation.
• ldes:Page, ldes:Relation, ldes:relation and ldes:page are subclass/subproperty axioms (OWL 2 RL) of their TREE counterparts. Checked with owlrl: after reasoning, a TREE client sees tree:Node, tree:relation, tree:Relation and tree:node and can traverse an LDES. It cannot prune, as the relation has no specific TREE type.
• Other alignments (tree:Collection, tree:member, tree:shape) are skos:closeMatch. See the discussion below.

4. Relations

• One relation per target page: ldes:relation [ ldes:page <p> ; … bounds … ]. Bounds are [from, until):
  • ldes:timeFrom / ldes:timeUntil (timezone mandatory)
  • ldes:sequenceFrom / ldes:sequenceUntil
  • ldes:pageNumber / ldes:pageNumberUntil (see point 6)
• TREE relation types, tree:path and tree:value are replaced by the bounds.
• Other kinds of relations (e.g. TREE relation types) can be added to a relation. A client does not have to understand them, but if it does not, it MUST follow the link (treated as a relation with no bounds, so it is fetched before later members can be emitted).

5. New traversal algorithm, small state

• The client merges pages like sorted logs: it always continues with the smallest lower key, and never fetches a relation whose upper bound is at or before the bookmark.
• State is a bookmark (last ordering key, plus the few member ids that share it). No visited-pages list and no emitted-members list.

6. Implicit order is still possible, but now need page numbers (which still can be a tree)

• An LDES without ldes:timestampPath and ldes:sequencePath is ordered by the log. The ordering key is (page number, index in page).
• Relations carry ldes:pageNumber (and optionally ldes:pageNumberUntil for a range). The root page is number 0 unless stated otherwise.
• Page numbers are stable: producers never renumber, and retention removes whole pages. Keys are unique, so the bookmark needs no member ids.
• Considered and rejected: position bounds (shift when retention removes members), an rdf:List of pages, or next links (a second paging construct, and the state becomes page IRIs).

7. Direct and indirect membership (new chapter)

• Direct: <ES> ldes:member <m> in the message.
• Indirect: at most one of ldes:memberGraph true (the only named graph), ldes:memberType <C> (the only subject of that type), or ldes:memberPath <P> (the only subject with that path, e.g. the timestampPath).

8. The LDES is one specific view, and a dcat:Distribution

• ldes:EventStream is a dcat:Distribution of a dcat:Dataset, no longer the dataset.
• tree:view, tree:viewDescription and ldes:EventSource are removed.
• The LDES IRI always dereferences to the root page. A hash-identifier or 303 See Other keeps the LDES IRI and the page IRI apart. A client finds the LDES via ?s rdf:type ldes:EventStream in the root page's description message.
• dcat:accessURL is only used in the DCAT-AP catalog (it carries the LDES IRI), not within LDES pages.
• Why: LDES 1.0 caused a lot of confusion about which IRI carries what, and this makes an LDES addable to data portals like any distribution.

9. Retention policy moves to the LDES

• ldes:retentionPolicy is now described on the ldes:EventStream instead of the root node or a view description.
• The former retention classes are removed: ldes:DurationAgoPolicy, ldes:LatestVersionSubset, ldes:PointInTimePolicy, ldes:amount and ldes:pointInTime.

10. Timezones

• Timezones: producers MUST publish a timezone on every xsd:dateTime
  (server primer), but clients must still cope without one. A client
  uses a configured default timezone, otherwise the new
  ldes:defaultTimezone of the LDES, otherwise assumes one shared,
  unknown timezone within the LDES. Only when a literal without a
  timezone is compared with one that has a timezone does it use the
  -14h/+14h period and worst-case comparison. Skipping a member or
  relation, or reporting a mis-ordered page, only happens when the
  comparison is certain.
• Add ldes:defaultTimezone to the vocabulary, the context and the
  primer.

Discussion topics

1. Description message: does it have to be the first message of a page, or do we add a flag that marks any message as context rather than a member? First keeps clients simple. A flag lets a "next" relation be appended to a growing page, but a client can't emit members of a page before it has read all of it.
2. LDES vs RDF Messages terminology: how do stream, log and profile in the RSP spec map to an LDES? The RSP profile section is still a TODO.
3. LDES as DCAT distribution: what happens to what was on the dataset-level LDES in 1.0 (e.g. the shape)? Two paginations of the same members are now two LDESs. Is that fine?
4. Retention on the LDES: how does it relate to a dataset with several distributions with different retention?
5. Implicit order with page numbers:
   • Is the root page being number 0 by default OK?
   • Do we need ldes:pageNumberUntil, or only flat pagination?
   • Is "retention removes whole pages only" too strict for version-based retention?
6. Bound naming: six bound properties (timeFrom/Until, sequenceFrom/Until, pageNumber/Until), or one generic ldes:from / ldes:until whose datatype picks the path? ldes:page as the relation target name?
7. TREE alignment: ldes:member and ldes:shape are not subproperties of tree:member / tree:shape, because their domain is tree:Collection (a dcat:Dataset), so a reasoner would infer the LDES is also a dataset. Do we accept that, or add an optional alignment file? tree:view cannot be inferred at all.
8. Page structure and profile announcement: profile parameter on the media type and .ldes file infix, as in the TREE profile. RDF Messages has no media types yet.
9. Member identity: combinations of indirect membership rules, or a SHACL selector? What if messages have no member identifier (fallback to the ordering key)?
10. Other relation kinds: a relation without bounds is followed before later members can be emitted. Is that the behaviour we want for unknown relations?
11. Streaming protocols: specify push notification here, or in a separate spec?

Not done yet

• Figures searchtree.svg and assets/ldes-vocabulary-overview.svg still show TREE.
• vocabulary.bs still uses MUST in term definitions; not aligned with the client-only convention yet.
• README.md has no 2.0 changelog entry; no migration note from 1.0 to 2.0.