Skip to content

fix(server): keep Server crash diagnostics after a container restart - #3258

Open
byteayan wants to merge 24 commits into
apache:masterfrom
byteayan:fix/server-hstore-crash-diagnostics
Open

byteayan wants to merge 24 commits into
apache:masterfrom
byteayan:fix/server-hstore-crash-diagnostics

Conversation

@byteayan

@byteayan byteayan commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Purpose of the PR

When a hugegraph/server (HStore) container exits during startup, kubectl logs --previous shows only Starting HugeGraphServer failed and a pointer to logs/hugegraph-server.log. That file, any hs_err_pid*.log and any heap dump live inside the container, and Kubernetes discards them when it restarts the pod. In #3203 the stack trace was only recovered by mounting a volume at /hugegraph-server/logs before the first boot.

Main Changes

  • Dockerfile-hstore sets STDOUT_MODE=true. fix(docker): enable docker logs for pd/store/server containers #2980 added it to the standalone, PD and Store Dockerfiles and missed this one.
  • conf/log4j2.xml: the org.apache.hadoop, org.apache.zookeeper, com.alipay.sofa, io.netty and org.apache.commons loggers are additivity="false" with only the file appender. Each gets <appender-ref ref="console" level="WARN"/>, so WARN and above reaches stdout and INFO stays in the file. Root and org.apache.hugegraph already log to console. Audit and slow-query logs are unchanged.
  • hugegraph-server.sh:
    • -XX:+HeapDumpOnOutOfMemoryError and -XX:HeapDumpPath used to sit inside the JAVA_OPTIONS default block, so setting JAVA_OPTIONS dropped them, and nothing set -XX:ErrorFile, so hs_err files went to the install root. Now the defaults go at the front of JAVA_TOOL_OPTIONS on every start. The JVM reads the operator's own JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS, the command line (JAVA_OPTIONS, -j) and _JAVA_OPTIONS after that, so any of them overrides a default.
    • Heap dumps go to a per-launch directory, logs/heapdump_<host>_<launch time>/, where each JVM writes java_pid<pid>.hprof; crash logs are logs/hs_err_pid%p_<host>_<launch time>.log. Child JVMs (computer jobs) inherit the environment, and HeapDumpPath only expands %p from JDK 25, so the directory is what gives every JVM its own dump. The host name separates pods on a shared volume, and a counter claimed with a plain mkdir separates restarts and concurrent launches. HotSpot truncates an existing crash log (JDK 17+) and will not write a heap dump over an existing file, which is why the names must be unique.
    • The directory is created after every preflight check, just before exec, so a start that fails earlier leaves nothing. If it cannot be created (a full volume, say), the launcher warns and dumps into logs/ instead of refusing to start. It never deletes dump directories, because a computer-job JVM can keep using one after its Server exits.
    • With telemetry on (-y true), the OpenTelemetry -javaagent option is appended to JAVA_TOOL_OPTIONS instead of replacing it.
    • Launcher errors that stop the Server before Java starts (unwritable logs/ or plugins/, missing riscv64 libatomic.so.1, unsupported JDK, too little memory, unknown GC option, missing security properties, telemetry download or checksum failure) go through report_error to hugegraph-server.log and stderr, so they reach kubectl logs too.
    • -Dhugegraph.bootstrap.error.log is passed in every mode; with STDOUT_MODE the HStore image would otherwise stop copying fatal bootstrap errors into hugegraph-server.log.
    • The paths are built from $LOGS, so they follow LOGS_OVERRIDE if feat(dist): Make conf/logs/plugins/pid paths overridable and fix log4j2 config resolution  #3253 lands. The two PRs touch nearby lines in this script.
  • util.sh: ensure_path_writable (imbajin's commit) and configure_riscv64_libatomic take an optional error handler; other callers are unchanged.
  • start-hugegraph.sh: in STDOUT_MODE the failure message points at "container logs ('docker logs' or 'kubectl logs')".
  • CI: ci-service-utils.sh lists heap dumps instead of printing them, the HStore diagnostics artifact leaves them out, and the riscv64 smoke test's log grep skips them.
  • docker/README.md: a "Server logs and crash files" section (version scope, what reaches stdout, file names, volumes per pod, sizing, cleanup, the opt-out through JAVA_TOOL_OPTIONS in the container environment). helm/hugegraph/README.md notes that the chart mounts no logs volume yet.

Behaviour changes to be aware of:

  • Heap dumps are on whenever the launcher runs, also when JAVA_OPTIONS is set, and they move from logs/java_pid<pid>.hprof into logs/heapdump_*/.
  • Each start leaves one heapdump_* directory, empty unless something ran out of memory, and the JVMs a Server starts now get the dump and crash-log defaults too.
  • The JVM prints Picked up JAVA_TOOL_OPTIONS: ... on stderr at startup.

Not in this PR:

  • immediateFlush="false" on the file appender (mentioned in the issue). WARN and above now also reach the console, which flushes each line.
  • The PD and Store start scripts also lack -XX:ErrorFile, which is left for a separate PR.
  • A logs volume in the Helm chart.

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:

test-java-security-properties.sh (run by server-ci) starts a real JVM with each launcher run's captured JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS, _JAVA_OPTIONS and command line, and checks the values it resolves: the defaults, overrides from each source (quoted, @argfile, -j), name collisions with a fixed clock and a dangling symlink, another host, two real JVMs running out of memory and each leaving a dump, telemetry, a failed preflight leaving no directory, the mkdir-failure fallback, an unwritable logs/ and the riscv64 libatomic error. It passes on JDK 11 locally (with the minimum lowered) under bash 3.2 and 5, leaves nothing in the distribution, and its new checks fail when the matching launcher behaviour is broken. I did not build or run the Docker images.

Does this PR potentially affect the following parts?

Documentation Status

  • Doc - TODO: required documentation is pending; complete it before merging.
  • Doc - Done: documentation is included here or linked below.
  • Doc - No Need: no user-visible documentation is affected.

Documentation files in this PR or paired hugegraph-doc PR:

When a hugegraph/server container exited during startup, kubectl logs
showed only "Starting HugeGraphServer failed". The cause stayed in
logs/ inside the container, and Kubernetes dropped it on restart.

- Set STDOUT_MODE=true in Dockerfile-hstore. apache#2980 did this for the
  standalone Dockerfile and missed this one.
- In conf/log4j2.xml, also send WARN and above from the hadoop,
  zookeeper, sofa, netty and commons loggers to console. They were
  file-only.
- In hugegraph-server.sh, always set -XX:+HeapDumpOnOutOfMemoryError,
  -XX:HeapDumpPath and -XX:ErrorFile under $LOGS. The heap-dump flags
  used to be dropped whenever JAVA_OPTIONS was set, and hs_err files
  went to the install directory. The file names carry the launch time
  because HotSpot will not overwrite an existing crash log or heap
  dump, and a restarted container often reuses the PID. The flags go
  before JAVA_OPTIONS and -j, so values given there still win; a flag
  already set in JAVA_TOOL_OPTIONS or JDK_JAVA_OPTIONS, which the JVM
  reads before the command line, is left out.
- Say "container logs" rather than "docker logs" in the start failure
  message, since the same image runs on Kubernetes.
- Cover the new flags and their overrides in
  test-java-security-properties.sh, and document the log directory
  and the Kubernetes mount in docker/README.md.

This applies to every Server install, bare metal included: an OOM now
writes a heap dump to logs/ even when JAVA_OPTIONS is set.

Fixes apache#3256
@codecov

codecov Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 40.24%. Comparing base (e62c961) to head (41764dd).

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3258      +/-   ##
============================================
- Coverage     41.19%   40.24%   -0.95%     
+ Complexity     6773     6504     -269     
============================================
  Files           766      743      -23     
  Lines         66086    63719    -2367     
  Branches       8773     8481     -292     
============================================
- Hits          27225    25646    -1579     
+ Misses        35818    35162     -656     
+ Partials       3043     2911     -132     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: When telemetry is enabled, crash options supplied through JAVA_TOOL_OPTIONS can be discarded together with the launcher defaults. Evidence: static trace of the crash-option scan and the telemetry environment assignment.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh Outdated
With telemetry on, the launcher replaced JAVA_TOOL_OPTIONS with the
OpenTelemetry -javaagent option. The crash-file defaults leave out any
flag already set in JAVA_TOOL_OPTIONS, so an operator's -XX:ErrorFile
there was dropped twice and HotSpot wrote hs_err to the working
directory. Append the agent to the existing value instead, and cover
the telemetry path in test-java-security-properties.sh with a stubbed
agent checksum.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Holding approval until the two findings below are addressed in the published branch. I prepared fixes and regression checks locally, but cannot push to the PR's fork, so those changes are not part of this review's commit.

The local fix lets the JVM handle diagnostic-option precedence by prepending defaults to JAVA_TOOL_OPTIONS, and removes the temporary telemetry JAR through the test's EXIT cleanup. Focused checks reproduced the quoted heap-dump opt-out failure on this PR head and passed with the local fixes, covering quoted environment options, JDK argument files, telemetry preservation, and cleanup on failure. Shell syntax, whitespace, and editorconfig checks passed. A full Maven compile was attempted but failed in hugegraph-common under Maven's locally selected JDK 26; full-build validation remains outstanding.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh Outdated

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: Heap-dump filenames can collide when a container restarts within one second and reuses its PID, so consecutive failures cannot retain distinct dumps. Evidence: hugegraph-server.sh:138,154; OpenJDK 11 defaults the heap dump writer to overwrite=false.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh Outdated
Review follow-ups on the crash-file defaults:

- Put the defaults at the front of JAVA_TOOL_OPTIONS instead of on the
  command line, and drop the shell scan of JAVA_TOOL_OPTIONS and
  JDK_JAVA_OPTIONS. The JVM reads the operator's JAVA_TOOL_OPTIONS,
  JDK_JAVA_OPTIONS, the command line and _JAVA_OPTIONS after the
  defaults, so any of them overrides a default. Word splitting in the
  shell missed quoted options and @argfiles, and could treat flag-like
  text inside a property value as a flag.
- Add a counter to the launch time in the file names when a heap dump
  or crash log with that name already exists in $LOGS. A restart in the
  same second with the same PID no longer targets an existing file.
- Test the values a real JVM resolves, covering a quoted opt-out, an
  @argfile in JDK_JAVA_OPTIONS, flag text in a property value, operator
  paths in JAVA_OPTIONS, name collisions and telemetry. The EXIT cleanup
  now removes the telemetry jar and log fixtures, so a failing
  step does not leave them in the distribution.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The latest-head checks pass, but one test-fixture cleanup issue remains. Evidence: the new crash-file fixture uses fixed paths under the supplied distribution's logs/ and removes them unconditionally.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 10/10. Summary: At ac9d98b the heap dump and crash log defaults go first in JAVA_TOOL_OPTIONS, so the JVM's own precedence lets JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS, JAVA_OPTIONS, -j and _JAVA_OPTIONS override them, file names get a counter when a name is already taken in logs/, and the telemetry agent is appended instead of replacing JAVA_TOOL_OPTIONS. The four earlier inline threads are addressed in this head and I found no new issues. Evidence: static review of the full diff and of hugegraph-server.sh, start-hugegraph.sh, docker-entrypoint.sh, Dockerfile-hstore and log4j2.xml at this head. CI at this head: 22 of 22 checks pass, and the build-server (rocksdb, 11) job ran the 'Run Java security properties tests' step, which starts a real JVM with the launcher's captured options, with conclusion success.

The collision check created two fixed crash-file names under the
supplied distribution's logs/ and removed them afterwards, which would
delete real files that already had those names. Record and remove only
the files this run creates.
byteayan added a commit to byteayan/hugegraph-doc that referenced this pull request Oct 4, 2026
…iles

Review follow-ups:

- Docker cluster guide: no 1.7.0 image sets STDOUT_MODE, the standalone
  hugegraph/hugegraph:1.7.0 used on the page included. Describe stdout
  logging as current master behaviour for all images, and widen the
  version note to every 1.7.0 and older image, linking #2980 and #3258.
- Server quickstart: the defaults now go first in JAVA_TOOL_OPTIONS
  (apache/hugegraph#3258), so list every source that overrides them,
  _JAVA_OPTIONS included, describe the counter added to a taken file
  name, and mention the "Picked up JAVA_TOOL_OPTIONS" startup line.
- Chinese pages: wrap only at existing spaces, so a soft break no longer
  renders as a space between two Chinese characters.
@byteayan

byteayan commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

The two red checks look unrelated to this change: store failed on the 3 s wait in OrderedKvIteratorTest.testConcurrentInitializeFailsWithoutWaitingForSlowSource (line 309), and build-server-macos-rocksdb (macos-15-intel) timed out downloading maven-antrun-plugin from Maven Central before any test ran. Both passed on the previous commits. Could someone re-run those two jobs?

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: At e7c16d2 the HStore Server image logs to stdout, third-party WARN reaches the console, and the heap dump and crash log defaults go first in JAVA_TOOL_OPTIONS so every operator source still overrides them. The earlier inline threads are addressed, including the fixture cleanup fix in this head, and I found no new issues. Evidence: static review of the full diff plus docker-entrypoint.sh, both Server Dockerfiles, log4j2.xml and server-ci.yml at this head; on JDK 11.0.32, quoted HeapDumpPath and ErrorFile values with a space in the path resolve correctly, and an exec'd JVM wrote its OOM dump to logs/java_pid_.hprof with the launcher's PID. CI at this head: all 22 checks pass, including the rocksdb lane that runs test-java-security-properties.sh.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: Unique crash-dump names preserve each OOM dump across restarts, but a persistent log volume can grow by a heap-sized file on every repeated OOM. Please document recurring-dump capacity and cleanup, or add configurable retention. Evidence: hugegraph-server.sh:148-155 generates a per-launch HeapDumpPath; docker/README.md:237 recommends a PersistentVolumeClaim and sizes it for a dump.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh Outdated
Each launch writes its heap dump to a new file, so a Server that keeps
running out of memory adds a heap-sized dump per restart until a
persistent /hugegraph-server/logs volume is full. Say that nothing
removes old dumps, and how to size the volume, clean up, or turn heap
dumps off.
byteayan added a commit to byteayan/hugegraph-doc that referenced this pull request Oct 4, 2026
Every launch writes its heap dump to a new file
(apache/hugegraph#3258), so a server that keeps running out of memory
adds a heap-sized dump per restart until the volume is full. Say that
nothing removes old dumps, and how to size the volume, clean up, or
turn dumps off, in the Docker guide and the Server quickstart, English
and Chinese.

@bitflicker64 bitflicker64 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Score: 9/10. Summary: Since e7c16d2 this head adds only the docker/README.md section on repeated OOM dumps, which answers the last open thread, plus a master merge that changes none of the PR's files. I found no new issues. Evidence: I read the full diff against merge base 5004a35 and confirmed the java -version probe runs before the JAVA_TOOL_OPTIONS export and that the launcher execs java. On a real JDK 17, a command-line -XX:-HeapDumpOnOutOfMemoryError overrides the launcher's quoted defaults in JAVA_TOOL_OPTIONS, as the README says. After the CI refactor, test-java-security-properties.sh still runs in build-server (rocksdb, 11) and passed at this head. All 28 checks pass.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: Two issues remain: crash-file naming can race when pods share a log PVC, and some pre-JVM startup errors still appear only in the file log. Evidence: exact-head source paths in the inline comments; all 28 checks passed.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh Outdated
Comment thread docker/README.md Outdated
- Crash-file names now carry the host name, the pod name on Kubernetes,
  before the launch time. Pods sharing one log volume each run as PID 1
  and could start in the same second; with only the PID and time they
  could pick the same name.
- Launcher errors that stop the Server before Java starts (unsupported
  JDK, too little free memory, unknown GC option, missing security
  properties, telemetry agent download or checksum failure) now go to
  stderr as well as hugegraph-server.log, through one report_error
  helper. start-hugegraph.sh passes that stderr to the terminal or the
  container log, so kubectl logs --previous shows these causes too.
- Tests cover the host part of the names, another host in the same
  second, unsafe host characters, and a preflight error on stderr and
  in the server log. docker/README.md is updated to match.
byteayan added a commit to byteayan/hugegraph-doc that referenced this pull request Oct 5, 2026
apache/hugegraph#3258 now puts the host name, the pod name on
Kubernetes, into heap dump and crash log names so servers sharing one
log directory do not pick the same name. Update the Server quickstart
in English and Chinese to match.
Resolve the hugegraph-server.sh conflict with the Java 17 baseline (apache#3261): keep the crash-file defaults in JAVA_TOOL_OPTIONS, drop the removed JAVA_VERSION > 9 --add-exports block (now in jvm-module.options), and report the new missing module options error through report_error.
Setting JAVA_OPTS replaces the image default (UseContainerSupport,
MaxRAMPercentage=50, ...). Recommend JAVA_TOOL_OPTIONS for the
-XX:-HeapDumpOnOutOfMemoryError opt-out, which the launcher reads after
its own defaults and which keeps JAVA_OPTS intact, and note the caveat
for JAVA_OPTS.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: One new logging-documentation detail is inaccurate for the current WARN-level logger configuration. Evidence: checked the exact-head Log4j2 logger levels.

Comment thread docker/README.md Outdated
byteayan and others added 4 commits October 6, 2026 09:19
The ZooKeeper and SOFA loggers are set to WARN, so their INFO events are
discarded, not kept in files. Limit the file-only INFO statement to the
Hadoop, Netty and Commons loggers.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: One crash-file collision remains possible because the fixed heap-dump target is exported to child JVMs launched by Server computer jobs. Evidence: static trace of the JAVA_TOOL_OPTIONS export and AbstractComputer's ProcessBuilder.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh Outdated
Child JVMs the Server starts, such as Hadoop computer jobs, inherit
JAVA_TOOL_OPTIONS. HeapDumpPath does not expand %p, so the fixed file
path built from the launcher's PID was shared: if a child dumped first,
HotSpot would not overwrite the file on a later Server OOM.

HeapDumpPath now names a per-launch directory,
logs/heapdump_<host>_<launch time>/, in which HotSpot writes
java_pid<pid>.hprof for each JVM. ErrorFile already expands %p.
Because the directory exists even when nothing is dumped, each start
removes this host's empty heapdump_* directories from earlier
launches; rmdir never removes a directory holding a dump.

The test checks the directory exists, runs two real JVMs out of memory
with the launcher's options and expects one dump each, and covers the
name counter and the empty-directory cleanup.
byteayan added a commit to byteayan/hugegraph-doc that referenced this pull request Oct 6, 2026
apache/hugegraph#3258 now points HeapDumpPath at a directory per
launch, logs/heapdump_<host>_<launch time>/, so the Server and any JVM
it starts (computer jobs inherit JAVA_TOOL_OPTIONS) each write their
own java_pid<pid>.hprof. Update the names and cleanup advice, and note
that startup removes only this host's empty heapdump_* directories, in
the Server quickstart and the Docker guide, English and Chinese.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: Empty heap-dump cleanup can remove a target still used by a live JVM. Evidence: Static trace of the launcher and test cleanup paths.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh Outdated
A live JVM's heap dump directory stays empty until it runs out of
memory, so removing every empty heapdump_* directory at startup, or in
the test's EXIT cleanup, could take the target from a running Server
that shares the logs directory.

Each launch now writes <dir>/.owner with the JVM's PID and its process
start time (the script execs java, so both are the JVM's). Startup
removes a directory only when it holds nothing but .owner and no process
with that PID and start time exists, so a PID reused after a container
restart does not look like the old owner. Directories without a marker,
with a dump, or owned by a running process stay, and nothing is removed
when ps cannot report a start time. The test's cleanup removes only
directories created during the run, and the test covers a dead owner,
a live owner, an unowned directory, another host's directory and one
holding a dump.
byteayan added a commit to byteayan/hugegraph-doc that referenced this pull request Oct 7, 2026
apache/hugegraph#3258 now removes a heapdump_* directory at startup
only when it holds no dump and the JVM recorded as its owner has
exited, so a running Server's empty directory is kept. Update the
Server quickstart and the Docker guide, English and Chinese.
RAT scans the built distribution under hugegraph-server/ and skipped
only **/logs/*.log. Each Server start now leaves a
logs/heapdump_<host>_<launch time>/.owner marker, so any CI step or
local run that starts the dist before a later Maven build failed the
license check with one unapproved file. A heap dump left in logs/
would fail it the same way. Everything under logs/ is runtime output,
and no tracked file lives there.
The previous commit widened the RAT exclusion in pom.xml so the .owner
marker would pass the license check. The CI Maven cache is keyed on a
hash of the pom.xml files, so that change made every job start from an
empty cache, where hg-pd-common fails to resolve hugegraph:${revision}
during compile. Revert it, leaving pom.xml identical to master.

Name the marker pid instead. RAT already excludes files named pid
(**/pid), and the file holds the owning JVM's PID and start time.
@byteayan

byteayan commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

@imbajin all review threads are resolved, and both required checks (check-license and Server memory tests) pass on 09a9812. Could you take another look and approve if it looks right to you?

The four red checks fail before any test runs. server_hbase and pd_store / hstore stop in hg-pd-common with maven-remote-resources-plugin reporting Invalid Artifact Requests: [org.apache.hugegraph:hugegraph:pom:${revision} ...]. affected-module-tests and codecov/project are red because those suites did not run. The same error hits master's own server_hbase at d9abcd4 and #3278, #3279 and #3280, so it looks like a build issue on master rather than something in this PR.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: Cleanup can remove a heap-dump target still used by a child JVM after the Server exits. Evidence: exact-head launcher and ProcessBuilder lifecycle trace; see the inline comment.

Comment thread hugegraph-server/hugegraph-dist/src/assembly/static/bin/hugegraph-server.sh Outdated
Startup cleanup removed a heapdump_* directory once the Server JVM that
created it had exited. Computer-job JVMs inherit the same HeapDumpPath
and can outlive their Server, so the next start could remove a
directory a running child still needs, and that child's OOM dump would
then fail. The launcher has no portable way to see those children.

Drop the cleanup and the pid owner marker. Each start leaves one
directory, empty unless something ran out of memory, and the README
says old ones can be deleted once no HugeGraph JVM from that launch is
running. The test checks that an empty directory from an earlier
launch is kept.
byteayan added a commit to byteayan/hugegraph-doc that referenced this pull request Oct 7, 2026
apache/hugegraph#3258 dropped the startup cleanup of heapdump_*
directories, since a computer-job JVM can keep using one after its
Server exits. Say that each start leaves a directory, empty unless
something ran out of memory, and that old ones can be deleted once no
JVM from that launch is running. English and Chinese.
Launcher (hugegraph-server.sh):
- Create the heap dump directory after every preflight check, just
  before exec, so a launch that fails before Java starts leaves nothing.
- Claim the name with a plain mkdir, so concurrent launches with the
  same host name and second get distinct names; treat a dangling
  symlink as taken.
- When the directory cannot be created (a full volume, for example),
  warn and dump into logs/ instead of refusing to start.
- Cap the host part at 64 characters and sanitize it with LC_ALL=C.
- Skip the defaults with a warning when the logs path contains a double
  quote or %, which JAVA_TOOL_OPTIONS cannot carry.
- Pass -Dhugegraph.bootstrap.error.log in every mode. With STDOUT_MODE
  the HStore image had stopped copying fatal bootstrap errors into
  hugegraph-server.log.
- report_error writes the log file before stderr and uses printf.
- The name check tolerates failglob.
- Correct the comments: HotSpot truncates an existing crash log on
  JDK 17+, HeapDumpPath expands %p from JDK 25, and @argfiles only work
  in JDK_JAVA_OPTIONS or on the command line.

CI: list heap dumps instead of printing them in ci-service-utils.sh,
leave them out of the HStore diagnostics artifact, and keep them out of
the riscv64 log grep.

Tests: check JDK_JAVA_OPTIONS and _JAVA_OPTIONS as the launcher passes
them, -j overriding a default, a dangling symlink, that a failed
preflight leaves no directory, the mkdir-failure fallback and an
unwritable logs/. Clean up only this run's directories, using a
test-only host name, and remove OOM dumps on any exit. Keep the OOM
runs' output for diagnosis, read flag values that contain spaces, and
ignore the runner's JVM option variables.

Docs: docker/README.md gains a version scope, the child-JVM-wide
opt-out, a warning not to remove a running Server's directory, one log
volume per pod, hostNetwork, ephemeral-storage and medium: Memory
notes, and the Helm chart's missing logs volume; the Helm README points
to it.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The crash-diagnostics changes are largely in place, but two Docker documentation details need clarification. Evidence: exact-head launcher and Compose configuration.

Comment thread docker/README.md Outdated
Comment thread docker/README.md Outdated
…OOL_OPTIONS

- configure_riscv64_libatomic takes an optional error handler, like
  ensure_path_writable, and the launcher defines report_error before
  calling it, so a missing libatomic.so.1 also reaches
  hugegraph-server.log. Other callers keep the plain stderr message.
- docker/README.md says JAVA_TOOL_OPTIONS has to be set in the
  container's environment (docker run -e, a Compose environment: entry,
  Kubernetes env); the shipped Compose files do not pass it through.
- The test checks the riscv64 error reaches stderr and the server log,
  with uname and ldconfig mocked.
- Say "all taken" when the 1000 counter values for a stamp are used,
  instead of naming the 1000th directory as uncreatable.
- In the test, feed PrintFlagsFinal output to awk through a here-string
  and filter bootstrap stderr with one awk call; with a pipe, an early
  awk exit could kill the writer with SIGPIPE under pipefail on large
  output.
- docker/README.md: both images declare VOLUME /hugegraph-server, so a
  Compose recreate keeps the files and docker compose down leaves an
  unattached anonymous volume until down -v; recommend a named volume
  or bind mount. Note that the full-volume fallback writes plain
  java_pid<pid>.hprof files into logs/, where a reused PID cannot write
  over an earlier dump.
byteayan added a commit to byteayan/hugegraph-doc that referenced this pull request Oct 7, 2026
Update the Server quickstart and the Docker guide, English and Chinese,
to match apache/hugegraph#3258:
- -j or JAVA_OPTIONS turns heap dumps off for the server JVM only;
  JAVA_TOOL_OPTIONS reaches the JVMs it starts.
- Never move the newest heapdump_* directory of a running server.
- The directory is created even when dumps are off, and the server
  starts anyway when it cannot be created.
- One log volume per pod, ephemeral-storage and sizeLimit, no
  medium: Memory, and the Helm chart's missing logs volume.
- The 1.7.0 note says dumps used to go straight to
  logs/java_pid<pid>.hprof.
- Name the script, give the launch time format, fix the garbled
  sentence about why directories are kept, and polish the Chinese.

@imbajin imbajin left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: no. Summary: The review found three documentation corrections around Kubernetes hostname naming, per-Pod PVC subpaths, and heap-dump fallback when the volume is full. Evidence: Exact-head launcher and Helm template review, plus Kubernetes documentation.

Comment thread docker/README.md Outdated
Comment thread docker/README.md Outdated
Comment thread docker/README.md Outdated
- On Kubernetes the host name is the pod name, or spec.hostname when
  set; drop the hostNetwork exception and note that the counter also
  separates pods sharing a host name.
- Call the fallback to logs/ best effort: a full volume may have no
  room for the dump either.
- subPathExpr: $(POD_NAME) only isolates pods when POD_NAME comes in
  through the Downward API; show the env and volumeMounts snippet and
  say the Helm chart sets neither.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] HStore Server image (hugegraph/server) keeps crash diagnostics only inside the container, so a restart on Kubernetes loses them

3 participants