diff --git a/.github/workflows/java.yaml b/.github/workflows/java.yaml index 61f1ab706..64c92b1ef 100644 --- a/.github/workflows/java.yaml +++ b/.github/workflows/java.yaml @@ -60,14 +60,14 @@ jobs: - name: Compile, check formatting, and lint maintenance tools run: make lint/java - - name: Test Go maintenance tools + - name: Test maintenance and release tools run: make test/java/tools - name: Verify canonical migration mirror run: make verify/java-migrations - - name: Check publishable archives - run: make check/java/package + - name: Check and stage complete GitHub release archives + run: make check/java/github-release java_versions: name: Test (Java ${{ matrix.java-version }}, SQLite) diff --git a/Makefile b/Makefile index 12e92428e..7955d7781 100644 --- a/Makefile +++ b/Makefile @@ -193,8 +193,9 @@ test/java/sqlite: generate/fixtures verify/java-migrations $(java_mvn) -f java/pom.xml verify -Driver.test.database=sqlite -DexcludedGroups=postgres .PHONY: test/java/tools -test/java/tools: ## Test Java maintenance tools and Maven warning enforcement +test/java/tools: ## Test Java maintenance tools, release tooling, and Maven warning enforcement bash java/bin/check-maven-warnings_test.sh + python3 -m unittest discover -s java/bin -p '*_test.py' go test ./java/bin/check-packages/*.go go test ./java/bin/sync-migrations/*.go @@ -278,6 +279,10 @@ check/java/package: ## Build and verify publishable Maven archives without publi check/java/package: build/java go run ./java/bin/check-packages/main.go +.PHONY: check/java/github-release +check/java/github-release: ## Build, test, and stage all Java GitHub release assets without publishing + python3 java/bin/github-release.py prepare + .PHONY: check/java/release check/java/release: ## Build a signed Maven Central bundle without uploading (requires an OpenPGP key) check/java/release: generate/fixtures verify/java-migrations diff --git a/java/Makefile b/java/Makefile index c103031c3..ab08f0ca0 100644 --- a/java/Makefile +++ b/java/Makefile @@ -1,4 +1,4 @@ -.PHONY: test test/conformance test/postgres test/sqlite test/tools lint lint/tools format package package/release +.PHONY: test test/conformance test/postgres test/sqlite test/tools lint lint/tools format package package/github-release package/release test: $(MAKE) -C .. test/java test/conformance: @@ -17,5 +17,7 @@ format: mvn spotless:apply package: $(MAKE) -C .. check/java/package +package/github-release: + $(MAKE) -C .. check/java/github-release package/release: $(MAKE) -C .. check/java/release diff --git a/java/README.md b/java/README.md index f31742519..583b1f310 100644 --- a/java/README.md +++ b/java/README.md @@ -9,6 +9,15 @@ retry, or work them using the same database. Java 21 or newer is required. Add `com.riverqueue:river:0.1.0-alpha.1` and a JDBC driver to your application's dependencies. Jackson 3 is included transitively. ```xml + + + river + https://riverqueue.com/maven + fail + false + + + com.riverqueue river @@ -21,6 +30,8 @@ Java 21 or newer is required. Add `com.riverqueue:river:0.1.0-alpha.1` and a JDB ``` +The River repository is public and requires no credentials. Maven Central supplies third-party dependencies. + For SQLite, replace the Postgres JDBC dependency with `org.xerial:sqlite-jdbc:3.53.4.0`. To install a local copy of River from source, see the [development guide](docs/development.md#building-from-source). ## Quick start @@ -78,24 +89,30 @@ for each operation. River never closes an application-owned data source. An ## Migration CLI -The `river-cli` executable JAR includes Postgres and SQLite JDBC drivers and -runs with Java 21 or newer. It needs no application classpath. -For a local build, see the [development guide](docs/development.md#building-from-source). Set `RIVER_CLI` to the executable JAR, then run: +The CLI runs with Java 21 or newer. [JBang](https://www.jbang.dev/documentation/jbang/latest/installation.html) +resolves the thin CLI JAR and its dependencies and installs a reusable command: ```sh -RIVER_CLI=/path/to/river-cli-0.1.0-alpha.1-all.jar +jbang app install --name river-java --java 21 \ + --java-options=--enable-native-access=ALL-UNNAMED \ + --repos https://riverqueue.com/maven --repos central \ + --main com.riverqueue.cli.MigrationCli \ + com.riverqueue:river-cli:0.1.0-alpha.1 +export PATH="$HOME/.jbang/bin:$PATH" export DATABASE_URL=postgres://localhost/myapp # For SQLite: export DATABASE_URL=jdbc:sqlite:/absolute/path/myapp.db -java -jar "$RIVER_CLI" --help -java -jar "$RIVER_CLI" migrate-list -java -jar "$RIVER_CLI" migrate-up --dry-run -java -jar "$RIVER_CLI" migrate-up +river-java --help +river-java migrate-list +river-java migrate-up --dry-run +river-java migrate-up ``` -The Maven executable artifact is `com.riverqueue:river-cli` with version -`0.1.0-alpha.1` and classifier `all`. Copy that single JAR into your deployment image to run -migrations before starting workers. `--database-url URL` overrides `DATABASE_URL`. +Dependencies, including both JDBC drivers, are downloaded once and cached. No +`all` classifier is needed. For containers or installations with Maven and a JDK, +use the [Maven-only instructions](#running-with-maven-and-java) below, or +[build from source](docs/development.md#building-from-source). +`--database-url URL` overrides `DATABASE_URL`. `--schema background_jobs` selects a Postgres schema; an actual migration up creates it if needed. Listing and dry runs do not create schemas or migration tables. @@ -113,10 +130,10 @@ version `0` explicitly removes the entire migration line. A zero `--max-steps` uses the command's default: all pending up migrations or one down migration. ```sh -java -jar "$RIVER_CLI" migrate-down --max-steps 2 --dry-run --show-sql -java -jar "$RIVER_CLI" migrate-up --target-version 8 -java -jar "$RIVER_CLI" migrate-get --version 3 --up > river_3.up.sql -java -jar "$RIVER_CLI" migrate-get --driver sqlite --all --up > river_sqlite.up.sql +river-java migrate-down --max-steps 2 --dry-run --show-sql +river-java migrate-up --target-version 8 +river-java migrate-get --version 3 --up > river_3.up.sql +river-java migrate-get --driver sqlite --all --up > river_sqlite.up.sql ``` SQL export defaults to Postgres without a database URL. Use `--driver sqlite` @@ -136,6 +153,31 @@ var preview = new Migrator(database).migrate(Migrator.Direction.DOWN, Migrator.Options.defaults().maxSteps(2).dryRun(true)); ``` +### Running with Maven and Java + +In an empty directory, create this consumer POM and download the runtime JARs: + +```sh +cat > pom.xml <<'EOF' + + 4.0.0 + localriver-cli-install1 + riverhttps://riverqueue.com/maven + com.riverqueueriver-cli0.1.0-alpha.1 + +EOF +mvn --batch-mode --no-transfer-progress \ + org.apache.maven.plugins:maven-dependency-plugin:3.11.0:copy-dependencies \ + -DincludeScope=runtime +java --enable-native-access=ALL-UNNAMED -cp 'target/dependency/*' \ + com.riverqueue.cli.MigrationCli --help +``` + +Replace `--help` with any migration command above. Copy the entire +`target/dependency` directory into a deployment image and adjust the classpath to +its location. Running it needs only Java; Maven and network access are needed +when downloading the dependencies. + ## Transactional enqueueing [Enqueue in the application's transaction](https://riverqueue.com/docs/transactional-enqueueing) diff --git a/java/bin/check-packages/main.go b/java/bin/check-packages/main.go index ee1ed072e..0c6ed6f73 100644 --- a/java/bin/check-packages/main.go +++ b/java/bin/check-packages/main.go @@ -98,7 +98,6 @@ func run(root string) error { {"river/target/river-" + project.Version + "-sources.jar", false}, {"cli/target/river-cli.jar", false}, {"cli/target/river-cli-sources.jar", false}, - {"cli/target/river-cli-" + project.Version + "-all.jar", true}, } { var resources string if artifact.runtime { diff --git a/java/bin/github-release.py b/java/bin/github-release.py new file mode 100644 index 000000000..3364e5108 --- /dev/null +++ b/java/bin/github-release.py @@ -0,0 +1,470 @@ +#!/usr/bin/env python3 +"""Stage Java release artifacts, upload a GitHub draft, or verify a Maven repository.""" + +import argparse +import hashlib +import json +import os +from pathlib import Path +import re +import shutil +import subprocess +import sys +import tempfile +import urllib.parse +import urllib.request +import xml.etree.ElementTree as ET +import zipfile + +ROOT = Path(__file__).resolve().parents[2] +DEFAULT_DIRECTORY = ROOT / "java/target/github-release" +GROUP = "com.riverqueue" +NS = {"m": "http://maven.apache.org/POM/4.0.0"} + + +def artifact_files(version): + result = [] + for artifact in ("river-java", "river", "river-cli"): + suffixes = [".pom"] + if artifact != "river-java": + suffixes += [".jar", "-sources.jar", "-javadoc.jar"] + result.extend( + (artifact, f"{artifact}-{version}{suffix}") for suffix in suffixes + ) + return result + + +def checked_version(version): + number = r"(?:0|[1-9][0-9]*)" + prerelease = r"(?:0|[1-9][0-9]*|[0-9]*[A-Za-z-][0-9A-Za-z-]*)" + if ( + not re.fullmatch( + rf"{number}\.{number}\.{number}(?:-{prerelease}(?:\.{prerelease})*)?", + version, + ) + or "snapshot" in version.lower() + ): + raise ValueError( + "Expected a canonical release version, without v, SNAPSHOT, or build metadata." + ) + return version + + +def command(args, **kwargs): + return subprocess.run(args, check=True, **kwargs) + + +def gh_json(args): + return json.loads(command(["gh", *args], capture_output=True, text=True).stdout) + + +def download(url): + request = urllib.request.Request( + url, headers={"User-Agent": "river-java-release/1.0"} + ) + with urllib.request.urlopen(request, timeout=60) as response: + return response.read() + + +def git(*args): + return command( + ["git", *args], cwd=ROOT, capture_output=True, text=True + ).stdout.strip() + + +def load_manifest(directory): + manifest = json.loads((directory / "manifest.json").read_text()) + if manifest.get("schema_version") != 1: + raise ValueError("Unrecognized release manifest.") + version = checked_version(manifest["version"]) + expected = {name for _, name in artifact_files(version)} + if set(manifest["files"]) != expected: + raise ValueError( + "Release manifest must contain the complete public artifact set." + ) + for name, info in manifest["files"].items(): + path = directory / name + if path.is_symlink() or not path.is_file(): + raise ValueError(f"Expected a regular release file: {name}") + if ( + path.stat().st_size != info["size"] + or hashlib.sha256(path.read_bytes()).hexdigest() != info["sha256"] + ): + raise ValueError(f"Staged artifact changed: {name}") + return manifest + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + actions = parser.add_subparsers(dest="action", required=True) + prepare_parser = actions.add_parser( + "prepare", help="Build, test, and stage all public POMs/JARs locally" + ) + prepare_parser.add_argument("--directory", type=Path, default=DEFAULT_DIRECTORY) + upload_parser = actions.add_parser( + "upload", help="Create or resume a draft and verify every uploaded asset" + ) + upload_parser.add_argument("--directory", type=Path, default=DEFAULT_DIRECTORY) + upload_parser.add_argument("--repo", default="riverqueue/river") + upload_parser.add_argument("--notes-file", required=True, type=Path) + verify_parser = actions.add_parser( + "verify", help="Resolve and run a release through a public Maven repository" + ) + verify_parser.add_argument("--version", required=True, type=checked_version) + verify_parser.add_argument("--repository", default="https://riverqueue.com/maven") + verify_parser.add_argument( + "--directory", + type=Path, + help="Also compare downloads with this staged manifest", + ) + args = parser.parse_args() + if args.action == "prepare": + prepare(args.directory.resolve()) + elif args.action == "upload": + upload(args.directory.resolve(), args.repo, args.notes_file.resolve()) + else: + verify(args.repository, args.version, args.directory) + + +def prepare(directory): + version = checked_version( + ET.parse(ROOT / "java/pom.xml").findtext("m:version", namespaces=NS) + ) + commit = git("rev-parse", "HEAD") + clean = not git("status", "--porcelain") + if directory.exists() and directory != DEFAULT_DIRECTORY: + raise ValueError( + "Choose an empty staging directory; custom directories are never overwritten." + ) + command(["make", "generate/fixtures", "verify/java-migrations"], cwd=ROOT) + with tempfile.TemporaryDirectory(prefix="river-java-stage-") as temporary: + repository = Path(temporary) / "repository" + destination = "github-staging::" + repository.as_uri() + # File deployment uses Maven's canonical artifact names/POMs without + # installing release artifacts locally or activating Central signing. + command( + [ + str(ROOT / "java/bin/check-maven-warnings.sh"), + "mvn", + "--batch-mode", + "--no-transfer-progress", + "-Dstyle.color=never", + "-f", + "java/pom.xml", + "-P!release", + "clean", + "verify", + "org.apache.maven.plugins:maven-deploy-plugin:3.2.0:deploy", + "-Driver.test.database=sqlite", + "-DexcludedGroups=postgres", + "-DaltDeploymentRepository=" + destination, + "-DaltReleaseDeploymentRepository=" + destination, + ], + cwd=ROOT, + ) + command(["go", "run", "./java/bin/check-packages/main.go"], cwd=ROOT) + stage = Path(temporary) / "assets" + stage.mkdir() + files = {} + for artifact, filename in artifact_files(version): + source = repository / "com/riverqueue" / artifact / version / filename + validate_artifact(source.read_bytes(), artifact, filename, version) + shutil.copyfile(source, stage / filename) + files[filename] = { + "sha256": hashlib.sha256(source.read_bytes()).hexdigest(), + "size": source.stat().st_size, + } + if (repository / "com/riverqueue/river-conformance").exists(): + raise ValueError("The conformance adapter must not be published.") + if commit != git("rev-parse", "HEAD"): + raise ValueError("Checkout changed during the build.") + manifest = { + "schema_version": 1, + "version": version, + "commit": commit, + "clean": clean and not git("status", "--porcelain"), + "files": files, + } + (stage / "manifest.json").write_text(json.dumps(manifest, indent=2) + "\n") + directory.parent.mkdir(parents=True, exist_ok=True) + shutil.move(str(stage), directory) + print(f"Staged {len(files)} artifacts for {version} in {directory}") + if not manifest["clean"]: + print( + "This was a development build; upload requires artifacts prepared from a clean release commit." + ) + + +def release_details(repo, tag): + result = subprocess.run( + ["gh", "release", "view", tag, "--repo", repo, "--json", "databaseId"], + capture_output=True, + text=True, + ) + if result.returncode: + if "release not found" in result.stderr.lower(): + return None + raise RuntimeError(result.stderr.strip()) + release_id = json.loads(result.stdout)["databaseId"] + return gh_json(["api", f"repos/{repo}/releases/{release_id}"]) + + +def resolved_from_proxy(provenance, filename): + # Newer Maven Resolver versions qualify repository IDs with a URL hash. + return ( + re.search( + rf"^{re.escape(filename)}>river-release-check(?:-[0-9a-f]{{40}})?=$", + provenance, + re.M, + ) + is not None + ) + + +def upload(directory, repo, notes_file): + if not re.fullmatch(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+", repo): + raise ValueError("Expected a GitHub owner/repository.") + manifest = load_manifest(directory) + if ( + not manifest["clean"] + or git("status", "--porcelain") + or git("rev-parse", "HEAD") != manifest["commit"] + ): + raise ValueError( + "Upload requires a clean checkout at the commit used to prepare the artifacts." + ) + if not notes_file.is_file(): + raise ValueError("Release notes file does not exist.") + tag = "java/v" + manifest["version"] + if git("rev-parse", tag + "^{commit}") != manifest["commit"]: + raise ValueError("Local release tag does not match the prepared commit.") + remote = gh_json(["api", f"repos/{repo}/git/ref/tags/{tag}"])["object"] + for _ in range(4): + if remote["type"] != "tag": + break + remote = gh_json(["api", f"repos/{repo}/git/tags/{remote['sha']}"])["object"] + if remote["type"] != "commit" or remote["sha"] != manifest["commit"]: + raise ValueError("Push the matching release tag before uploading assets.") + release = release_details(repo, tag) + if release is None: + args = [ + "gh", + "release", + "create", + tag, + "--repo", + repo, + "--verify-tag", + "--draft", + "--latest=false", + "--title", + tag, + "--notes-file", + str(notes_file), + ] + if "-" in manifest["version"]: + args += ["--prerelease"] + command(args) + release = release_details(repo, tag) + if not release["draft"]: + raise ValueError( + "Only draft releases may be uploaded by this script; published versions are immutable." + ) + pending = verify_assets(release["assets"], manifest, allow_missing=True) + if pending: + command( + [ + "gh", + "release", + "upload", + tag, + "--repo", + repo, + *[str(directory / name) for name in pending], + ] + ) + verify_assets(release_details(repo, tag)["assets"], manifest) + print(f"All {len(manifest['files'])} assets verified on draft {tag}.") + print( + f"Publish with: gh release edit {tag} --repo {repo} --draft=false --latest=false" + ) + + +def validate_artifact(data, artifact, filename, version): + if filename.endswith(".pom"): + if b"\n") + repository_xml = ET.Element("repositories") + entry = ET.SubElement(repository_xml, "repository") + ET.SubElement(entry, "id").text = "river-release-check" + ET.SubElement(entry, "url").text = repository + ET.SubElement(ET.SubElement(entry, "releases"), "checksumPolicy").text = "fail" + ET.SubElement(ET.SubElement(entry, "snapshots"), "enabled").text = "false" + (work / "pom.xml").write_text( + f"4.0.0checkcheck1{ET.tostring(repository_xml, encoding='unicode')}{GROUP}river-cli{version}" + ) + environment = os.environ.copy() + environment.update(MAVEN_ARGS="", MAVEN_OPTS="") + for name in list(environment): + if name.startswith(("MAVEN_GPG_", "CENTRAL_")): + environment.pop(name) + args = [ + "mvn", + "--batch-mode", + "--no-transfer-progress", + "--strict-checksums", + "-s", + str(settings), + "-gs", + str(settings), + "-Dmaven.repo.local=" + str(work / "m2"), + ] + plugin = "org.apache.maven.plugins:maven-dependency-plugin:3.11.0" + command( + args + [plugin + ":build-classpath", "-Dmdep.outputFile=classpath.txt"], + cwd=work, + env=environment, + ) + for artifact, filename in artifact_files(version): + if "-sources.jar" in filename or "-javadoc.jar" in filename: + continue + path = work / "m2/com/riverqueue" / artifact / version / filename + if hashlib.sha256(path.read_bytes()).hexdigest() != downloaded[filename]: + raise ValueError(f"Maven downloaded different bytes: {filename}") + provenance = (path.parent / "_remote.repositories").read_text() + if not resolved_from_proxy(provenance, filename): + raise ValueError(f"Maven resolved {filename} from another repository.") + classpath = (work / "classpath.txt").read_text().strip() + (work / "Smoke.java").write_text( + 'public class Smoke { public static void main(String[] args) { if (!com.riverqueue.Client.class.getName().equals("com.riverqueue.Client")) throw new AssertionError(); if (!tools.jackson.databind.json.JsonMapper.builder().build().writeValueAsString("river").equals("\\"river\\"")) throw new AssertionError(); } }' + ) + command( + ["javac", "--release", "21", "-cp", classpath, "Smoke.java"], + cwd=work, + env=environment, + ) + command( + ["java", "-cp", "." + os.pathsep + classpath, "Smoke"], + cwd=work, + env=environment, + ) + cli = [ + "java", + "--enable-native-access=ALL-UNNAMED", + "-cp", + classpath, + "com.riverqueue.cli.MigrationCli", + ] + output = command( + cli + ["--version"], + cwd=work, + env=environment, + capture_output=True, + text=True, + ).stdout.strip() + if output != "river " + version: + raise ValueError(f"CLI version differs: {output}") + environment["DATABASE_URL"] = "jdbc:sqlite:" + str(work / "river.db") + for operation in ( + ["migrate-up"], + ["migrate-list"], + ["migrate-down", "--target-version", "0"], + ): + command(cli + operation, cwd=work, env=environment) + print( + f"Verified all {len(downloaded)} artifacts, fresh Maven resolution, consumer compilation, and {output} with SQLite migrations." + ) + + +if __name__ == "__main__": + try: + main() + except ( + OSError, + ValueError, + RuntimeError, + subprocess.CalledProcessError, + ET.ParseError, + zipfile.BadZipFile, + ) as error: + print(f"Java release: {error}", file=sys.stderr) + sys.exit(1) diff --git a/java/bin/github_release_test.py b/java/bin/github_release_test.py new file mode 100644 index 000000000..c7f7bb2d1 --- /dev/null +++ b/java/bin/github_release_test.py @@ -0,0 +1,238 @@ +import contextlib +import hashlib +import importlib.util +import io +import json +from pathlib import Path +import tempfile +import unittest +from unittest.mock import patch + +SPEC = importlib.util.spec_from_file_location( + "github_release", Path(__file__).with_name("github-release.py") +) +release = importlib.util.module_from_spec(SPEC) +SPEC.loader.exec_module(release) + + +class GitHubReleaseTest(unittest.TestCase): + def setUp(self): + self.temporary = tempfile.TemporaryDirectory() + self.addCleanup(self.temporary.cleanup) + self.root = Path(self.temporary.name) + self.prepare_version("99.88.77-alpha.4") + self.notes = self.root / "notes.md" + self.notes.write_text("Release notes") + + def prepare_version(self, version): + self.version = version + self.manifest = { + "schema_version": 1, + "version": self.version, + "commit": "a" * 40, + "clean": True, + "files": {}, + } + for _, name in release.artifact_files(self.version): + data = name.encode() + (self.root / name).write_bytes(data) + self.manifest["files"][name] = { + "sha256": hashlib.sha256(data).hexdigest(), + "size": len(data), + } + self.write_manifest() + self.assets = [ + { + "name": name, + "state": "uploaded", + "size": data["size"], + "digest": "sha256:" + data["sha256"], + } + for name, data in self.manifest["files"].items() + ] + + def write_manifest(self): + (self.root / "manifest.json").write_text(json.dumps(self.manifest)) + + def mock_git(self, *args): + return "" if args[0] == "status" else self.manifest["commit"] + + def test_artifacts_include_parent_without_fat_jar_or_conformance(self): + names = {name for _, name in release.artifact_files(self.version)} + self.assertEqual(len(names), 9) + self.assertIn("river-java-" + self.version + ".pom", names) + self.assertIn("river-cli-" + self.version + ".jar", names) + self.assertNotIn("river-cli-" + self.version + "-all.jar", names) + self.assertFalse(any("conformance" in name for name in names)) + + def test_manifest_rejects_missing_extra_changed_and_symlinked_files(self): + release.load_manifest(self.root) + name = next(iter(self.manifest["files"])) + (self.root / name).write_text("changed") + with self.assertRaisesRegex(ValueError, "changed"): + release.load_manifest(self.root) + (self.root / name).unlink() + (self.root / name).symlink_to(self.notes) + with self.assertRaisesRegex(ValueError, "regular"): + release.load_manifest(self.root) + del self.manifest["files"][name] + self.write_manifest() + with self.assertRaisesRegex(ValueError, "complete"): + release.load_manifest(self.root) + self.manifest["files"]["../unexpected.jar"] = {} + self.write_manifest() + with self.assertRaisesRegex(ValueError, "complete"): + release.load_manifest(self.root) + + def test_pom_coordinates_parent_inheritance_and_packaging(self): + parent = f"com.riverqueueriver-java{self.version}" + xml = f'{parent}river' + release.validate_artifact(xml.encode(), "river", "river.pom", self.version) + with self.assertRaisesRegex(ValueError, "coordinates"): + release.validate_artifact( + xml.encode(), "river-cli", "river-cli.pom", self.version + ) + xml = xml.replace( + "river", "river-java" + ) + with self.assertRaisesRegex(ValueError, "packaging"): + release.validate_artifact( + xml.encode(), "river-java", "river-java.pom", self.version + ) + release.validate_artifact( + xml.replace("", "pom").encode(), + "river-java", + "river-java.pom", + self.version, + ) + + def test_repository_tracking_accepts_old_and_url_qualified_ids(self): + name = "river-" + self.version + ".jar" + for repository in ("river-release-check", "river-release-check-" + "a" * 40): + self.assertTrue( + release.resolved_from_proxy(f"{name}>{repository}=\n", name) + ) + for repository in ( + "", + "central", + "central-" + "a" * 40, + "river-release-check-other", + ): + self.assertFalse( + release.resolved_from_proxy(f"{name}>{repository}=\n", name) + ) + self.assertFalse( + release.resolved_from_proxy(f"different.jar>river-release-check=\n", name) + ) + + def test_upload_creates_drafts_without_replacing_latest(self): + for version, prerelease in (("99.88.77-alpha.4", True), ("99.88.77", False)): + with self.subTest(version=version): + self.prepare_version(version) + before = {"draft": True, "assets": []} + after = {"draft": True, "assets": self.assets} + remote = [ + {"object": {"type": "tag", "sha": "b" * 40}}, + {"object": {"type": "commit", "sha": self.manifest["commit"]}}, + ] + output = io.StringIO() + with ( + patch.object(release, "git", side_effect=self.mock_git), + patch.object(release, "gh_json", side_effect=remote), + patch.object( + release, "release_details", side_effect=[None, before, after] + ), + patch.object(release, "command") as command, + contextlib.redirect_stdout(output), + ): + release.upload(self.root, "riverqueue/river", self.notes) + create, upload = [call.args[0] for call in command.call_args_list] + self.assertEqual( + create[:4], ["gh", "release", "create", "java/v" + version] + ) + for flag in ("--draft", "--verify-tag", "--latest=false"): + self.assertIn(flag, create) + self.assertEqual("--prerelease" in create, prerelease) + self.assertEqual( + create[create.index("--notes-file") + 1], str(self.notes) + ) + self.assertEqual(upload[:3], ["gh", "release", "upload"]) + self.assertEqual(len(upload[6:]), 9) + self.assertIn( + f"gh release edit java/v{version} --repo riverqueue/river --draft=false --latest=false", + output.getvalue(), + ) + + def test_upload_resumes_matching_draft_and_never_clobbers(self): + before = {"draft": True, "assets": self.assets[:2]} + after = {"draft": True, "assets": self.assets} + remote = {"object": {"type": "commit", "sha": self.manifest["commit"]}} + with ( + patch.object(release, "git", side_effect=self.mock_git), + patch.object(release, "gh_json", return_value=remote), + patch.object(release, "release_details", side_effect=[before, after]), + patch.object(release, "command") as command, + contextlib.redirect_stdout(io.StringIO()), + ): + release.upload(self.root, "riverqueue/river", self.notes) + args = command.call_args.args[0] + self.assertEqual(args[:3], ["gh", "release", "upload"]) + self.assertNotIn("--clobber", args) + self.assertEqual(len(args[6:]), 7) + + def test_upload_refuses_dirty_build_wrong_tag_and_published_release(self): + self.manifest["clean"] = False + self.write_manifest() + with self.assertRaisesRegex(ValueError, "clean"): + release.upload(self.root, "riverqueue/river", self.notes) + self.manifest["clean"] = True + self.write_manifest() + remote = {"object": {"type": "commit", "sha": "b" * 40}} + with ( + patch.object(release, "git", side_effect=self.mock_git), + patch.object(release, "gh_json", return_value=remote), + ): + with self.assertRaisesRegex(ValueError, "matching release tag"): + release.upload(self.root, "riverqueue/river", self.notes) + remote["object"]["sha"] = self.manifest["commit"] + with ( + patch.object(release, "git", side_effect=self.mock_git), + patch.object(release, "gh_json", return_value=remote), + patch.object(release, "release_details", return_value={"draft": False}), + ): + with self.assertRaisesRegex(ValueError, "Only draft"): + release.upload(self.root, "riverqueue/river", self.notes) + + def test_verify_assets_refuses_mismatches_before_uploading_anything(self): + self.assets[-1]["digest"] = "sha256:wrong" + with self.assertRaisesRegex(ValueError, "refusing to overwrite"): + release.verify_assets(self.assets[2:], self.manifest, allow_missing=True) + with self.assertRaisesRegex(ValueError, "Missing"): + release.verify_assets([], self.manifest) + + def test_verify_refuses_credentials_and_non_https_urls(self): + for url in ( + "https://user:secret@example.com/maven", + "http://example.com/maven", + "https://example.com/?token=secret", + ): + with self.subTest(url=url), self.assertRaises(ValueError): + release.verify(url, self.version) + + def test_versions_reject_snapshots_aliases_and_paths(self): + for value in ( + "1.0", + "v1.0.0", + "01.0.0", + "1.0.0-SNAPSHOT", + "1.0.0+build", + "../1.0.0", + "1.0.0-01", + ): + with self.subTest(version=value), self.assertRaises(ValueError): + release.checked_version(value) + self.assertEqual(release.checked_version(self.version), self.version) + + +if __name__ == "__main__": + unittest.main() diff --git a/java/cli/pom.xml b/java/cli/pom.xml index 57fdf6b21..52fe00ab7 100644 --- a/java/cli/pom.xml +++ b/java/cli/pom.xml @@ -14,22 +14,21 @@ river-cli - org.apache.maven.pluginsmaven-shade-plugin3.6.1 - packageshade - false - trueall - - - META-INF/LICENSE - META-INF/NOTICE - - com.riverqueue.cli.MigrationCli - true${project.version}ALL-UNNAMED - - - - - *:*META-INF/MANIFEST.MFMETA-INF/*.SFMETA-INF/*.DSAMETA-INF/*.RSAmodule-info.classMETA-INF/versions/*/module-info.class + org.apache.maven.pluginsmaven-jar-plugin + + + true + dependency/ + com.riverqueue.cli.MigrationCli + + ${project.version} + + + + org.apache.maven.pluginsmaven-dependency-plugin3.11.0 + prepare-packagecopy-dependencies + runtime + ${project.build.directory}/dependency org.apache.maven.pluginsmaven-failsafe-plugin3.5.4 diff --git a/java/cli/src/test/java/com/riverqueue/cli/ExecutableIT.java b/java/cli/src/test/java/com/riverqueue/cli/ExecutableIT.java index 27465e02f..fe6b57714 100644 --- a/java/cli/src/test/java/com/riverqueue/cli/ExecutableIT.java +++ b/java/cli/src/test/java/com/riverqueue/cli/ExecutableIT.java @@ -2,26 +2,41 @@ import static org.junit.jupiter.api.Assertions.*; +import java.io.File; import java.nio.file.Path; import java.util.ArrayList; import java.util.List; import java.util.concurrent.TimeUnit; -import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; class ExecutableIT { @TempDir Path directory; - private String command(String... args) throws Exception { + private String command(boolean executableJar, String... args) throws Exception { var command = new ArrayList<>( List.of( Path.of(System.getProperty("java.home"), "bin", "java").toString(), - "-jar", - "target/river-cli-0.1.0-alpha.1-all.jar")); + "--enable-native-access=ALL-UNNAMED")); + var jar = Path.of("target/river-cli.jar").toAbsolutePath().toString(); + if (executableJar) { + command.addAll(List.of("-jar", jar)); + } else { + command.addAll( + List.of( + "-cp", + jar + File.pathSeparator + Path.of("target/dependency/*").toAbsolutePath(), + "com.riverqueue.cli.MigrationCli")); + } command.addAll(List.of(args)); var output = directory.resolve("process.log").toFile(); - var builder = new ProcessBuilder(command).redirectErrorStream(true).redirectOutput(output); + var builder = + new ProcessBuilder(command) + .directory(directory.toFile()) + .redirectErrorStream(true) + .redirectOutput(output); builder.environment().put("DATABASE_URL", "jdbc:sqlite:" + directory.resolve("river.db")); var process = builder.start(); try { @@ -34,12 +49,14 @@ private String command(String... args) throws Exception { } } - @Test - void bundledJarRunsWithoutExternalClasspath() throws Exception { - assertTrue(command("--version").contains("0.1.0-alpha.1")); - assertTrue(command("migrate-up").contains("008 [up]")); - assertTrue(command("migrate-list").contains("applied")); - assertTrue(command("migrate-down", "--target-version", "0").contains("001 [down]")); - assertFalse(command("migrate-list").contains("applied")); + @ParameterizedTest + @ValueSource(booleans = {false, true}) + void thinJarRunsWithRuntimeDependencies(boolean executableJar) throws Exception { + assertEquals("river 0.1.0-alpha.1", command(executableJar, "--version").strip()); + assertTrue(command(executableJar, "migrate-up").contains("008 [up]")); + assertTrue(command(executableJar, "migrate-list").contains("applied")); + assertTrue( + command(executableJar, "migrate-down", "--target-version", "0").contains("001 [down]")); + assertFalse(command(executableJar, "migrate-list").contains("applied")); } } diff --git a/java/docs/development.md b/java/docs/development.md index 708c96149..2131479e3 100644 --- a/java/docs/development.md +++ b/java/docs/development.md @@ -13,24 +13,145 @@ make generate/fixtures mvn --batch-mode --no-transfer-progress -f java/pom.xml install ``` -The executable migration CLI is built at `java/cli/target/river-cli-0.1.0-alpha.1-all.jar`. For a build without running tests, use `make build/java`. +The CLI is built at `java/cli/target/river-cli.jar`, with its runtime dependencies +in `java/cli/target/dependency/`. Run it with: -## Maven Central artifacts +```sh +java --enable-native-access=ALL-UNNAMED \ + -jar java/cli/target/river-cli.jar --help +``` + +Keep the `dependency/` directory beside the CLI JAR when moving it. For a build +without running tests, use `make build/java`. -Normal builds attach sources and Javadoc JARs for the library and CLI. The -`release` Maven profile additionally signs the artifacts and builds a Central -Publisher Portal bundle with checksums. It includes three coordinates, all under -`com.riverqueue` and using the version in `java/pom.xml`: +## Release artifacts -| Artifact | Contents | +GitHub releases tagged `java/vVERSION` are the publication source for +`https://riverqueue.com/maven`. The proxy imports the complete asset set into R2 +and serves it anonymously to Maven and Gradle. Publishers use `gh`; no Sonatype +account, OpenPGP key, or Cloudflare credential is needed. + +| Artifact | GitHub attachments | | --- | --- | -| `river-java` | Parent POM with shared dependencies and publication metadata | -| `river` | Library, sources, Javadoc, and POM | -| `river-cli` | Thin CLI, executable `all` JAR, sources, Javadoc, and POM | +| `river-java` | Parent POM | +| `river` | POM, library JAR, sources JAR, Javadoc JAR | +| `river-cli` | POM, thin JAR, sources JAR, Javadoc JAR | + +Publish all nine files together. The parent POM is required by this release's +library and CLI; the conformance adapter is excluded. The proxy generates +checksums and preserves imported versions, so publish a new version to change +bytes or attachments. New releases do not build or upload a shaded `-all.jar`. +Use [JBang or Maven](../README.md#migration-cli) to resolve CLI dependencies; +older published standalone JARs remain available. + +## Releasing a new version + +Run from a clean checkout with JDK 21 or 25, Maven 3.9+, Go, `golangci-lint`, +Python 3.9+, and `gh`. A human runs `gh auth login` in their terminal once; a human +or local agent can then use the same authenticated `gh` session. Keep tokens out +of chat, shell arguments, and release files. No additional credential handoff is +needed. Stop if a command fails. Versions omit `v`; tags use `java/vVERSION`. + +1. Start from a clean checkout, fetch changes and tags, and choose the release version: + + ```sh + git checkout master && git pull --rebase + git fetch --tags + export VERSION=0.1.0-alpha.2 + git checkout -b "$USER-java-$VERSION" + ``` + +2. Update the POMs, CLI tests, documentation examples, and adapter version references together: + + ```sh + python3 - <<'PY' + import json, os, pathlib, subprocess, xml.etree.ElementTree as ET + old = ET.parse("java/pom.xml").findtext("{http://maven.apache.org/POM/4.0.0}version") + files = subprocess.check_output([ + "git", "grep", "-Il", "-F", old, "--", "java", + ":!java/CHANGELOG.md", ":!java/VALIDATION.md", + ":!java/bin/github_release_test.py", + ":!java/conformance/src/main/resources/manifest.json", + ]).decode().splitlines() + for name in files: + path = pathlib.Path(name) + path.write_text(path.read_text().replace(old, os.environ["VERSION"])) + path = pathlib.Path("java/conformance/src/main/resources/manifest.json") + manifest = json.loads(path.read_text()) + manifest["implementations"]["java"]["version"] = os.environ["VERSION"] + path.write_text(json.dumps(manifest, indent=2) + "\n") + PY + git diff -- java + ``` + + Move `java/CHANGELOG.md` entries from `Unreleased` into a `[$VERSION] - YYYY-MM-DD` section. For the first release, describe the initial library and CLI; for a stable release, remove "Prerelease" from the README introduction. + +3. Check the release and open a PR. Keep CI enabled and have the PR reviewed and merged after the Java checks pass: + + ```sh + make lint/java test/java/tools test/java/sqlite check/java/package + git add java + git commit -m "[java] Release v$VERSION" + git push -u origin HEAD + gh pr create --fill + ``` + +4. After merge, pull the release commit into a clean checkout, confirm its version, and create the local tag: + + ```sh + git checkout master && git pull --rebase && + test -z "$(git status --porcelain)" && + test "$(python3 -c 'import xml.etree.ElementTree as E; print(E.parse("java/pom.xml").findtext("{http://maven.apache.org/POM/4.0.0}version"))')" = "$VERSION" && + git tag "java/v$VERSION" -m "release java/v$VERSION" + ``` + +5. Build, test, and stage the complete release. This uses a temporary **local file repository**, without signing, uploading, or installing release artifacts into your normal Maven cache: + + ```sh + python3 java/bin/github-release.py prepare + python3 - <<'PYNOTES' + import os, pathlib, re + text = pathlib.Path("java/CHANGELOG.md").read_text() + match = re.search(r"^## \[" + re.escape(os.environ["VERSION"]) + r"\][^\n]*\n(.*?)(?=^## |\Z)", text, re.M | re.S) + if not match: + raise SystemExit("Version missing from Java changelog") + pathlib.Path("java/target/github-release-notes.md").write_text(match.group(1).strip() + "\n") + PYNOTES + ``` + +6. Push the tag and upload all artifacts to a draft. The script checks the local/remote tag against the build commit, resumes matching draft uploads, and refuses changed assets or an already published release: + + ```sh + git push origin "java/v$VERSION" + python3 java/bin/github-release.py upload \ + --notes-file java/target/github-release-notes.md + gh release view "java/v$VERSION" --repo riverqueue/river + ``` + +7. Publish the verified draft. Alpha, beta, and RC versions are marked as prereleases automatically. Keep `--latest=false` for every Java release, including stable versions, so the repository's Latest release remains the Go release: + + ```sh + gh release edit "java/v$VERSION" --repo riverqueue/river --draft=false --latest=false + ``` -The parent POM must be published with its children so consumers can resolve their -dependencies. `river-conformance` is explicitly excluded from Central publication -and skips signing, Javadoc, and local installation. +8. Verify the public proxy, all asset checksums, fresh Maven dependency resolution, a compiled consumer, and thin CLI SQLite migrations. This also checks that Maven obtained River artifacts from the proxy rather than Central. The proxy imports on a schedule; requests do not trigger discovery. If the release is not visible yet, wait a minute and retry: + + ```sh + python3 java/bin/github-release.py verify --version "$VERSION" \ + --directory java/target/github-release + ``` + + To verify an existing release without a local build, omit `--directory`. This + requires a thin CLI JAR with `Implementation-Version` manifest metadata. The + originally published alpha.1 thin JAR reports `development` instead of its + release version and fails the CLI version check. + +## Optional Maven Central publication + +The GitHub/proxy release path above does not use Central publication. The existing +`release` Maven profile remains available for separately publishing to Central; +it signs the same public coordinates and creates a Portal bundle. A version +published to both repositories must use the same artifact bytes. ### Local tooling and signing @@ -109,102 +230,10 @@ Portal. Ordinary builds do not activate signing or Central publishing. See the [Central Maven plugin guide](https://central.sonatype.org/publish/publish-portal-maven/) for the upload and validation lifecycle. -## Releasing a new version - -Run these commands from the repository root, in the same shell, with JDK 21 or 25, -Maven, Go, `golangci-lint`, Python 3, and `gh` available. Complete the signing and -publishing setup above first. Stop if any command fails. Java versions have no -leading `v`; tags use `java/vX.Y.Z`. - -1. Start from a clean checkout, fetch changes and tags, and choose the release version: - - ```sh - git checkout master && git pull --rebase - git fetch --tags - export VERSION=0.1.0-alpha.1 - git checkout -b "$USER-java-$VERSION" - ``` - -2. Update the POMs, CLI tests, documentation examples, and adapter version references together: - - ```sh - python3 - <<'PY' - import json, os, pathlib, subprocess, xml.etree.ElementTree as ET - old = ET.parse("java/pom.xml").findtext("{http://maven.apache.org/POM/4.0.0}version") - files = subprocess.check_output([ - "git", "grep", "-Il", "-F", old, "--", "java", - ":!java/CHANGELOG.md", ":!java/VALIDATION.md", - ":!java/conformance/src/main/resources/manifest.json", - ]).decode().splitlines() - for name in files: - path = pathlib.Path(name) - path.write_text(path.read_text().replace(old, os.environ["VERSION"])) - path = pathlib.Path("java/conformance/src/main/resources/manifest.json") - manifest = json.loads(path.read_text()) - manifest["implementations"]["java"]["version"] = os.environ["VERSION"] - path.write_text(json.dumps(manifest, indent=2) + "\n") - PY - git diff -- java - ``` - - Move `java/CHANGELOG.md` entries from `Unreleased` into a `[$VERSION] - YYYY-MM-DD` section. For the first release, describe the initial library and CLI; for a stable release, remove "Prerelease" from the README introduction. - -3. Check the release and open a PR. Keep CI enabled and have the PR reviewed and merged after the Java checks pass: - - ```sh - make lint/java test/java/sqlite check/java/package - git add java - git commit -m "[java] Release v$VERSION" - git push -u origin HEAD - gh pr create --fill - ``` - -4. After merge, pull the release commit into a clean checkout, confirm its version, and create the local tag: - - ```sh - git checkout master && git pull --rebase && - test -z "$(git status --porcelain)" && - test "$(python3 -c 'import xml.etree.ElementTree as E; print(E.parse("java/pom.xml").findtext("{http://maven.apache.org/POM/4.0.0}version"))')" = "$VERSION" && - git tag "java/v$VERSION" -m "release java/v$VERSION" - ``` - -5. Load your private key, then enter its passphrase, the Central token username, and the Central token password at the three `read` commands. Build, test, sign, and upload the release for validation: - - ```sh - export MAVEN_GPG_KEY="$(cat /absolute/path/to/private-key.asc)" - read -r -s MAVEN_GPG_PASSPHRASE - read -r CENTRAL_USERNAME - read -r -s CENTRAL_PASSWORD - export MAVEN_GPG_PASSPHRASE CENTRAL_USERNAME CENTRAL_PASSWORD - test -z "$(git status --porcelain)" && - test "$(git rev-parse HEAD)" = "$(git rev-parse "java/v$VERSION^{commit}")" && - make generate/fixtures verify/java-migrations && - ./java/bin/check-maven-warnings.sh mvn --batch-mode --no-transfer-progress \ - -Dstyle.color=never -f java/pom.xml -Prelease clean deploy \ - -Driver.test.database=sqlite -DexcludedGroups=postgres \ - -Dmaven.install.skip=true -Dcentral.skipUpload=false - unset MAVEN_GPG_KEY MAVEN_GPG_PASSPHRASE CENTRAL_USERNAME CENTRAL_PASSWORD - ``` - -6. In the [Central Portal deployments page](https://central.sonatype.com/publishing/deployments), open the validated deployment, confirm it contains `river-java`, `river`, and `river-cli` at `$VERSION`, and click **Publish**. Wait for **Published**. If uploading timed out, check this page before retrying. - -7. Verify the published CLI and its transitive library/parent dependencies using a fresh Maven cache. If Central has not propagated the release yet, wait and rerun with a new temporary directory: - - ```sh - export JAVA_RELEASE_CHECK_DIR="$(mktemp -d)" - (cd "$JAVA_RELEASE_CHECK_DIR" && mvn --batch-mode --no-transfer-progress \ - -Dmaven.repo.local="$JAVA_RELEASE_CHECK_DIR/m2" dependency:get \ - -Dartifact="com.riverqueue:river-cli:$VERSION:jar:all") && - java -jar "$JAVA_RELEASE_CHECK_DIR/m2/com/riverqueue/river-cli/$VERSION/river-cli-$VERSION-all.jar" --version && - git push origin "java/v$VERSION" - ``` - -8. Create a [GitHub release](https://github.com/riverqueue/river/releases/new) for `java/v$VERSION`, copy that version's Java changelog notes into the body, and attach the downloaded `river-cli-$VERSION-all.jar`. Mark alpha, beta, and RC versions as prereleases. - ## Tests and checks From the repository root, with Go (the version in `go.work`), JDK 21 or 25, -and Maven installed: +Maven, and Python 3.9+ installed: ```sh make test/java/conformance @@ -251,7 +280,7 @@ canonical Go drivers. `verify/java-migrations` detects changed, missing, or extr migrations. `check/java/package` checks the library and CLI archives, including source JARs, for development content and verifies bundled runtime resources. The Go maintenance tools live in `java/bin/`. `make test/java` and `make lint/java` -include their tests and lint checks; `make test/java/tools` and +include their tests and lint checks; the tool tests also cover GitHub release staging and upload guards; `make test/java/tools` and `make lint/java/tools` run only the tool checks. From `java/`, use `make test/tools` and `make lint/tools`. These checks run in Java CI, separately from the root Go-only `make test` and `make lint` targets. @@ -287,11 +316,13 @@ the existing API documentation does not annotate every parameter and return valu Automatic external documentation links are disabled so Javadoc generation does not depend on another documentation site being reachable. -Executable JARs merge dependency `META-INF/LICENSE` and `META-INF/NOTICE` text, -retain service registrations, and generate their own executable manifests. -Dependency module descriptors and signatures are excluded from these combined -archives. Other overlapping classes or resources still produce warnings and fail -the checked Make targets. Library and source JARs keep their normal packaging. +The development-only conformance adapter JAR merges dependency +`META-INF/LICENSE` and `META-INF/NOTICE` text, retains service registrations, and +generates its own executable manifest. +Dependency module descriptors and signatures are excluded from this combined +archive. Other overlapping classes or resources still produce warnings and fail +the checked Make targets. Library, CLI, and source JARs keep their normal +packaging; CLI dependencies stay in separate JARs. ## Legacy cross-process harness