Shell Syntax

Shell Syntax

Shell command parsing with MuParser and argument types.

Overview

Shell Syntax is the parsing and evaluation infrastructure that powers JNode's command-line interface and scripting capabilities. At its core, it consists of two main systems: a high-level declarative syntax framework (Syntax, Argument, SyntaxBundle) that commands use to describe expected arguments, and a low-level backtracking parser (MuParser) that validates user input against those descriptions. This design separates the "what arguments are expected" from the "how to parse them", making it easy to define complex command-line interfaces without writing parsing code.

The system is centered in the org.jnode.shell.syntax package (~200 references) and is built around the plugin system, allowing commands and their syntaxes to be registered declaratively. When a user invokes a shell command, the flow is: SyntaxManager locates the command's SyntaxBundle, the bundle's Syntax trees are compiled into MuSyntax graphs, MuParser performs a backtracking parse of the user's input tokens, and the resulting ArgumentBundle is populated with validated values before execute() is called.

Key Components

Class / File	Role
shell/src/shell/org/jnode/shell/syntax/SyntaxManager.java	Maps command aliases to SyntaxBundles; manages registration and lookup
shell/src/shell/org/jnode/shell/syntax/MuParser.java	Backtracking parser; validates and populates arguments from token stream
shell/src/shell/org/jnode/shell/syntax/Syntax.java	Abstract syntax tree node (SequenceSyntax, OptionSyntax, RepeatSyntax, etc.)
shell/src/shell/org/jnode/shell/syntax/Argument.java	Base class for typed argument holders (FileArgument, StringArgument, FlagArgument, etc.)
shell/src/shell/org/jnode/shell/syntax/SyntaxBundle.java	Container holding one or more Syntax alternatives for a command alias
shell/src/shell/org/jnode/shell/syntax/ArgumentBundle.java	Container holding Argument instances for a command; populated by MuParser
shell/src/shell/org/jnode/shell/syntax/SyntaxSpecLoader.java	Builds Syntax/ArgumentSpec trees from the org.jnode.shell.syntaxes extension point XML
shell/src/shell/org/jnode/shell/syntax/XMLSyntaxSpecAdapter.java, PluginSyntaxSpecAdapter.java	The two SyntaxSpecAdapter implementations (NanoXML element vs. plugin ConfigurationElement)
shell/src/shell/org/jnode/shell/syntax/MuPreset.java, MuBackReference.java, SharedStack.java	mu-syntax internals: injected argument values, label references, and the LIFO-over-FIFO backtracking stack
shell/src/shell/org/jnode/shell/alias/DefaultAliasManager.java	Alias lookup with parent-chain resolution and a read-only system-alias view

How It Works

Command Registration

Commands register their syntax via the plugin system using the org.jnode.shell.syntaxes extension point. The SyntaxManager interface (NAME = SyntaxManager.class) is registered in the InitialNaming namespace and provides:

• add(SyntaxBundle bundle) — register syntax bundles embedded in commands
• add(String alias, ArgumentSpec<?>[] argSpecs) — register specs for non-native commands
• getSyntaxBundle(String alias) / getArgumentBundle(String alias) — lookup by command alias
• createSyntaxManager() — create child managers for nested scopes

Syntax Compilation

Each Syntax subclass (e.g., SequenceSyntax, OptionSyntax, AlternativesSyntax, RepeatSyntax) implements prepare(ArgumentBundle bundle) which produces a MuSyntax tree:

• MuSymbol — matches literal tokens (e.g., -o, --force)
• MuArgument — matches and validates arguments against an Argument instance
• MuPreset — injects preset values for hidden/fixed arguments
• MuSequence — sequences of elements processed left-to-right
• MuAlternation — alternatives explored via backtracking

MuParser Backtracking

MuParser is the runtime engine. It uses a deque-based syntax stack and a backtrack stack of ChoicePoint entries. The parsing loop:

1. Pops the next MuSyntax from the stack
2. Matches symbol tokens (MuSymbol) or validates against arguments (MuArgument)
3. For MuArgument, calls Argument.accept(token, flags) which delegates to doAccept() for type conversion and validation
4. On failure, pops the next alternative from the current ChoicePoint, restores the syntax stack and source position, and undoes argument modifications via undoLastValue()
5. On success, all arguments are populated in the ArgumentBundle

The DEFAULT_STEP_LIMIT is 10000 parse steps to guard against pathological grammars.

Argument Types

The Argument<V> class hierarchy provides typed, validated argument holders. Key types include StringArgument, IntegerArgument, LongArgument, FileArgument, HostNameArgument, PortNumberArgument, FlagArgument, and EnumArgument. Each implements doAccept(Token) for type conversion and complete() for tab-completion suggestions.

Argument flags control parsing behavior:

Flag	Meaning
MANDATORY	At least one value must be supplied
OPTIONAL	May be omitted (default)
MULTIPLE	Zero or more values allowed
EXISTING	Value must denote an existing entity
NONEXISTENT	Value must NOT exist

Tab-Completion

The same MuParser operates in completion mode when completions is non-null. Instead of throwing on parse failure, it explores all alternatives and calls Argument.complete() on partial tokens to produce suggestions (e.g., FileArgument lists directory entries).

Note the split between Argument.complete(CompletionInfo, Token, String, int) and the protected doComplete(...). complete() first calls isSet(), which throws SyntaxFailureException("This Argument is not associated with an ArgumentBundle") when the argument was constructed standalone. Any code path that builds a helper Argument on the fly — as URLArgument does for the file: scheme — must call doComplete() directly, or completion fails unconditionally.

XML Spec Loading — SyntaxSpecLoader

The org.jnode.shell.syntaxes extension point delivers syntaxes as XML through a SyntaxSpecAdapter, so the loader is adapter-agnostic (one implementation reads NanoXML XMLElements, another reads plugin ConfigurationElements). SyntaxSpecLoader.doLoad dispatches on element name:

Element	Produces
argument	the alias' ArgumentSpec[] + SyntaxBundle
sequence, powerset, repeat, option, optionSet, verb, symbol, alternatives, optional	the corresponding Syntax subclass

getFlag(adapter, name, default) reads eager on <optional> / <repeat> / <powerset>; a value that is neither true nor false throws SyntaxFailureException("'<name>' attribute is not 'true' or 'false'").

Serialization round trips are the contract that keeps this honest: every Syntax.toXML() must emit exactly the attribute names doLoad reads back, because the syntax shell command dumps a syntax as XML and re-loading that dump has to reproduce the same tree.

Host Test Coverage (2026-10-04 to 2026-10-11)

The shell syntax subsystem grew from ~211 to ~500 host JUnit tests in one week, all wired through shell/src/test/org/jnode/test/shell/syntax/AllTests.java (29 classes), .../alias/AllTests.java (new, 1 class) and .../io/AllTests.java (4 classes), which the top-level org.jnode.test.shell.AllTests nests.

Group	Test classes	Subject
Managers and loaders	DefaultAliasManagerTest (18), DefaultSyntaxManagerTest (13), SyntaxSpecLoaderTest (34), XMLSyntaxSpecAdapterTest (12), PluginSyntaxSpecAdapterTest (7)	extension-point registration, every loader element kind, both spec adapters
mu-syntax internals	MuPresetTest (17), SharedStackTest (18)	preset backtracking/undo, LIFO-over-FIFO stack, copy-on-write drain, unsupported Deque ops
Composition syntaxes	VerbSyntaxTest (7), GroupSyntaxTest (12), OptionalSyntaxTest (20), SymbolSyntaxTest (15), EmptySyntaxTest (13)	matching, labelling, format(), toXML() round trips
Argument types	HostArgumentTypesTest (10), NumericArgumentTypesTest (12), SystemArgumentTypesTest (16)	host-bound, numeric, and log4j/shell-property arguments
Shell IO	PipelineStreamsAndFanoutWriterTest (31)	PipelineInputStream, PipelineOutputStream, FanoutWriter

SharedStackTest deliberately lives at shell/src/test/org/jnode/shell/syntax/SharedStackTest.java — a different package from the rest — because SharedStack is package-private. The suite imports it explicitly.

TestSyntaxRoundTrip is a helper with zero @Test methods: it wraps a Syntax.toXML() tree in a <syntax> root and reads it back with SyntaxSpecLoader, so the round-trip tests are two lines each.

Production fixes found by those tests

Fix	Change
VerbSyntax.prepare	Passed no label to the MuSequence it builds, so the mu-syntax was unlabelled and a MuBackReference could not resolve it. Now new MuSequence(label, new MuSymbol(symbol), new MuPreset(arg.getLabel(), "true")).
VerbSyntax.toXML	Emitted argName; SyntaxSpecLoader reads argLabel for <verb>, so every serialize/re-parse round trip threw. Now emits argLabel, and the loader additionally falls back to argName so pre-existing descriptors still load.
SyntaxSpecLoader.getFlag	eager="false" returned true. Every plugin descriptor with an explicit eager="false" was parsed as eager.
LongArgument.doComplete	A leftover System.err.println("Testing completion '" + candidate + "'") inside the candidate loop emitted up to 100 stderr lines per TAB press (COMPLETION_THRESHOLD = 100).
DeviceArgument.state()	Dereferenced apiClass unconditionally, so Argument.toString() threw NPE for the two- and three-argument constructors. argumentKind() in the same class already guarded null.
URLArgument.doComplete	Called complete() instead of doComplete() on the helper FileArgument it constructs, which is never bound to a bundle — so completion of any file: URL always failed.
Pipeline.read / Pipeline.write	Used len as an end index instead of off + len in both loops, so read(b,2,7) moved 5 bytes.
Pipeline.skip	Never advanced its local off, so it drained the whole buffer and always returned -1.
FanoutWriter.removeStream	Built the shrunk Writer[] and never assigned it back, so a removed target kept receiving every subsequent write and flush.

Gotchas & Non-Obvious Behavior

• Backtracking modifies argument state — MuParser records which arguments were modified and calls undoLastValue() when backtracking; do not perform side effects in doAccept()
• Step limit prevents infinite loops — grammars with complex alternations can exhaust the DEFAULT_STEP_LIMIT; adjust via the stepLimit parameter if needed
• EXISTING and NONEXISTENT are not logical negations — they check different validation properties; a file can be non-existent for creation but inaccessible for reading
• EXISTING and NONEXISTENT affect completion only — neither performs runtime validation. MkdirCommand dropped NONEXISTENT from its FileArgument purely so tab-completion would still suggest an already-existing directory (see Shell-Commands)
• Labels must be unique within an ArgumentBundle; the parser matches labels across syntax tree nodes to argument instances
• SharedStack avoids copying — SharedStack is used instead of LinkedList for the syntax stack to reduce allocation overhead during backtracking
• Command sequencing is not part of this framework — &&, ||, ;, |, < and > are handled earlier by the interpreters, in Shell-Interpreter
• GroupSyntax.childLabels() holds labels of direct children only — Syntax.childLabels() returns Collections.emptySet(); GroupSyntax's constructor (shell/src/shell/org/jnode/shell/syntax/GroupSyntax.java) adds child.getLabel() for each labelled child and unions the child sets, so nesting is one level deep, not transitive. Unlabelled leaves such as new ArgumentSyntax("fileArg") contribute nothing (that constructor sets label = null)
• A null child is tolerated only for XML serialization — the constructor skips null children and basicElement() substitutes a shared EmptySyntax; prepare() / format() in AlternativesSyntax, OptionalSyntax etc. still dereference every child, so null children are only safe in the XML path
• toXML() must emit the attribute names the loader reads. VerbSyntax wrote argName while SyntaxSpecLoader read argLabel, so a round trip through the syntax command threw. The loader now also accepts argName as a fallback, but the emission side is the contract
• Never write a test that asserts a known defect. Three times in the 2026-10 test batch a real bug was found, a follow-up issue filed, and the test written to lock in the broken behaviour (testChildLabelsDoesNotIncludeLeafLabels, testToXMLRoundTripFailsOnArgNameAttribute, testEagerFalseFlagCurrentlyBehavesLikeTrue). Each shipped green and forced the eventual fix to rewrite the test. The rule is now written into .opencode/skills/jnode-issue-resolver/SKILL.md as anti-pattern S8 and into .github/AGENTS.md as a blocking (🔴) review finding. See Agent-Issue-Pipeline
• A test class no suite references never runs. shell/build-tests.xml's <batchtest> names exactly one file, org/jnode/test/shell/AllTests.java; everything else in src/test compiles and is silently ignored. See Core-Test-Suite and Testing
• SystemArgumentTypesTest needs a JNDI namespace installed. ShellPropertyNameArgument.doComplete reaches InitialNaming, which NPEs on the host JDK, so the class carries a @BeforeClass that mirrors HostArgumentTypesTest.setUpDeviceManager; without it the class only passes as part of the full suite. See InitialNaming-Service-Registry

Related Pages

• Syntax — Deep-dive documentation of the full syntax framework, Argument types, and parsing internals
• Shell-Commands — Parent hub for the command framework
• Shell-Interpreter — Command sequences, pipelines, redirection, and the shared tokenizer
• Plugin-System — How commands and syntaxes are registered via extensions
• Testing / Core-Test-Suite — how the shell suites are wired and run on the host
• Agent-Issue-Pipeline — the S8 anti-pattern: never lock a known defect into a test
• CLI — Command-line interface concepts