Skip to content

README: a conformance suite for HOCON - #889

Open
lejmr wants to merge 1 commit into
lightbend:mainfrom
lejmr:readme-conformance-suite
Open

lejmr wants to merge 1 commit into
lightbend:mainfrom
lejmr:readme-conformance-suite

Conversation

@lejmr

@lejmr lejmr commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

A test suite for HOCON implementations: 446 cases, one for each sentence of HOCON.md, each citing the sentence it tests, with a runner and the scores of the implementations it has adapters for, this library among them. It is where the question settled in #887 came from.

@kastoestoramadus

kastoestoramadus commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

TLDR: The gray-zone evidence behind the suite this README links: I re-ran it (589 cases, zig-hocon main @ 3e4ddc6, 2026-11-03) against typesafe/config 1.4.9, ekrich/sconfig 2.1.0, my sHOCON fork and our 13 open PRs from #868 to #882 (eight independent, plus the #878 → #882 stack). No PR makes any case worse than the one before it (0 regressions across the stack); the cases the stack fixes are attributed in #882.

26 cases stay gray-zone today — implementations legitimately disagree, and each row records one way how:

  • 7 × unsupported — typesafe/config 1.4.9 fails outright on input the spec allows: an internal error (BugOrBroken) on six rows, a hard parse error on the .properties-include row
  • 10 × lenient — typesafe/config 1.4.9 silently accepts or drops data where the spec requires an error
  • 8 × diverges — substitution/merge resolution differs (a substitution sees the field's merged value; a null stops merging only at its own key)
  • 1 × — — outside the java-marked zone: self-referential-examples/051 (hidden list concatenation) passes on typesafe/config 1.4.9 and sconfig 2.1.0 but fails on the fork and on the stack (already on main, before any of our PRs); a known gap of the fork, not addressed here

The two unicode-escape rows are the bug reported in #884; a fix is open in #890, outside this stack. Full per-row status (✓ pass / ✗ fail; "stack @ #882" is the top of the stack):

26-row table
case typesafe 1.4.9 sconfig 2.1.0 sHOCON fork stack @ #882 class
array-and-object-concatenation/018 ✗ ✓ ✓ ✓ unsupported
array-and-object-concatenation/019 ✗ ✓ ✓ ✓ unsupported
array-and-object-concatenation/020 ✗ ✓ ✓ ✓ unsupported
array-and-object-concatenation/021 ✗ ✓ ✓ ✓ unsupported
array-and-object-concatenation/022 ✗ ✓ ✓ ✓ unsupported
array-and-object-concatenation/023 ✗ ✗ ✓ ✓ diverges
concatenation-whitespace-and-substitutions/007 ✗ ✗ ✓ ✓ lenient
duplicate-keys-and-object-merging/010 ✗ ✗ ✓ ✓ diverges
duplicate-keys-and-object-merging/011 ✗ ✗ ✓ ✓ diverges
duplicate-keys-and-object-merging/012 ✗ ✗ ✓ ✓ diverges
duplicate-keys-and-object-merging/013 ✗ ✗ ✓ ✓ diverges
duplicate-keys-and-object-merging/015 ✗ ✗ ✓ ✓ lenient
duplicate-keys-and-object-merging/016 ✗ ✗ ✓ ✓ diverges
duplicate-keys-and-object-merging/029 ✗ ✗ ✓ ✓ diverges
include-file-formats/006 ✗ ✓ ✓ ✗ lenient
include-file-formats/007 ✗ ✓ ✓ ✗ unsupported
plus-equals-field-separator/008 ✗ ✓ ✓ ✓ unsupported
self-referential-examples/043 ✗ ✗ ✓ ✓ diverges
self-referential-examples/051 ✓ ✓ ✗ ✗ —
string-value-concatenation/010 ✗ ✗ ✓ ✓ lenient
string-value-concatenation/011 ✗ ✗ ✓ ✓ lenient
string-value-concatenation/014 ✗ ✗ ✓ ✓ lenient
string-value-concatenation/015 ✗ ✗ ✓ ✓ lenient
string-value-concatenation/016 ✗ ✗ ✓ ✓ lenient
unchanged-from-json/029 ✗ ✗ ✓ ✗ lenient
unchanged-from-json/030 ✗ ✗ ✓ ✗ lenient

Per-row explanations (the why note for each case) go with the java_since offer on #882 — happy to open the zig-hocon issue with them on request.

Generated with glm-5.3-flash/h through ZCode
Reviewed by opus-5.5/m
Revised by glm-5.3/max

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants