Skip to content

fix(server): trust PROXY headers on SSH from configured ranges - #7674

Open
otavio wants to merge 4 commits into
masterfrom
fix/ssh-proxy-trusted-ips
Open

otavio wants to merge 4 commits into
masterfrom
fix/ssh-proxy-trusted-ips

Conversation

@otavio

@otavio otavio commented Oct 23, 2026

Copy link
Copy Markdown
Member

Summary

The SSH listener trusted a PROXY protocol header only from loopback, with the ranges hard-coded. Behind a Layer 4 load balancer or an ingress that passes TCP through, v0.27.0 closed every SSH connection and logged nothing.

 NewServer(opts)
+  policy = loopback + opts.ProxyTrustedIPs     # SHELLHUB_SSH_PROXY_TRUSTED_IPS, space-separated
+  malformed range -> refuse to start
 ListenAndServe
-  newProxyListener(lis)                        # loopback only
+  newProxyListener(lis, policy)
 handshake fails
+  ConnectionFailedCallback: header from an untrusted peer -> warn with peer address
  • SHELLHUB_SSH_PROXY_TRUSTED_IPS is empty by default, so a deployment that sets nothing keeps the GHSA-c5cf-fr53-8wq4 protection. It has its own name because the gateway's SHELLHUB_PROXY_TRUSTED_IPS defaults to trusting every peer.
  • Wired through docker-compose.yml and .env. The docs pages that said the gateway variables apply to SSH are corrected.
  • Commits: refactor (the listener takes its policy as an argument), then the rejection warning, then the configurable ranges, then docs.

For cloud.shellhub.io, after release: Envoy passes the header through on port 22 again, the server gets SHELLHUB_SSH_PROXY_TRUSTED_IPS=10.0.128.0/17, and a NetworkPolicy limits port 2222 to the Envoy pods. Trust is by source address, so that NetworkPolicy is part of the fix.

Evidence

server/ssh/server/proxy_test.go runs a real SSH handshake through the production listener. The client sends PROXY TCP4 198.51.100.7 ... from the host's own non-loopback address, since loopback is always trusted and can't show the bug.

untrusted peer, default config  -> handshake fails with ErrSuperfluousProxyHeader, warning names the peer
peer listed as trusted          -> handshake succeeds, RemoteAddr() == 198.51.100.7:40000
  • Before: with the trusted ranges ignored and the warning removed (the v0.27.0 behaviour), both tests fail:
    --- FAIL: TestProxyHeaderFromUntrustedPeerIsRefusedAndLogged   Expected value not to be nil.
    --- FAIL: TestProxyHeaderFromTrustedPeerSetsClientAddress      Received unexpected error
    
    After:
    level=warning msg="closed an ssh connection whose PROXY protocol header came from an untrusted peer" peer="172.16.0.2:47128"
    --- PASS: TestProxyHeaderFromUntrustedPeerIsRefusedAndLogged
    --- PASS: TestProxyHeaderFromTrustedPeerSetsClientAddress
    --- PASS: TestProxyPolicy (loopback, inside a trusted range, outside it, Docker bridge)
    --- PASS: TestProxyPolicyRejectsMalformedRange
    ok  github.com/shellhub-io/shellhub/server/ssh/server
    
  • go test ./ssh/... ./app/... and golangci-lint v2.14.0 pass (0 issues). The docs lint (remark, cspell, vale) passes.

Merge Danger

Door: two-way

A new optional variable with a safe default. Reverting restores the loopback-only listener.

Blast Radius: SSH

Every SSH connection goes through this listener. With the variable unset, the only change is the new warning. If the variable is set too wide, any peer in the range can forge a client address, which affects firewall rules, lockout, access policies and the audit trail. The docs and the commit say so.

Closes shellhub-io/team#362

@otavio
otavio requested review from a team as code owners October 23, 2026 20:57
@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown

Documentation preview: https://091bc40d.shellhub-docs.pages.dev

Comment thread server/ssh/server/server.go
Comment thread server/ssh/server/proxy_test.go Outdated
Comment thread server/ssh/server/proxy_test.go Outdated
@otavio
otavio force-pushed the fix/ssh-proxy-trusted-ips branch from 873b8d2 to b551e8d Compare October 23, 2026 21:15
@otavio

otavio commented Oct 23, 2026

Copy link
Copy Markdown
Member Author

/review

The listener read the loopback-only policy from a package variable. Taking it as an argument lets
the server build the policy from its configuration.
A PROXY header from an untrusted peer failed the SSH handshake inside gliderssh, and nothing
reached the log. On cloud.shellhub.io every SSH login closed before the banner after the upgrade
to v0.27.0, with no server-side trace. The listener now logs a warning naming the peer, which is
the address to trust.

The read error surfaces from gossh.NewServerConn, so the warning hangs off
ConnectionFailedCallback rather than the listener. The test serves a server built by NewServer,
so it fails when that wiring is dropped. It dials the host's own non-loopback address, because
loopback is always trusted and cannot show the rejection; it skips on a host without one.

Refs: shellhub-io/team#362
@github-actions

This comment has been minimized.

@otavio
otavio force-pushed the fix/ssh-proxy-trusted-ips branch from b551e8d to 9a48db5 Compare October 23, 2026 21:18
@otavio

otavio commented Oct 23, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions

Copy link
Copy Markdown

Documentation preview: https://7607693d.shellhub-docs.pages.dev

@github-actions

This comment has been minimized.

Comment thread ui/apps/docs/src/pages/selfhosted/environment-variables.mdx Outdated
Comment thread ui/apps/docs/src/pages/selfhosted/troubleshooting.mdx Outdated
@otavio
otavio force-pushed the fix/ssh-proxy-trusted-ips branch from 9a48db5 to c94a9d6 Compare October 23, 2026 21:26
@otavio

otavio commented Oct 23, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown

Documentation preview: https://37f575c8.shellhub-docs.pages.dev

Comment thread ui/apps/docs/src/pages/manage/team/multi-factor-authentication.mdx
Comment thread ui/apps/docs/src/pages/selfhosted/troubleshooting.mdx Outdated
Comment thread server/ssh/server/server_test.go Outdated
Since GHSA-c5cf-fr53-8wq4 the SSH listener reads a PROXY header only from loopback, and the
ranges were hard-coded. A Layer 4 load balancer or an ingress that passes TCP through sends the
header from another address, so v0.27.0 closed every such SSH connection. SHELLHUB_PROXY and
SHELLHUB_PROXY_TRUSTED_IPS did not help, because only the gateway reads them.

SHELLHUB_SSH_PROXY_TRUSTED_IPS lists the extra ranges, space-separated like the gateway's
variable. It is a separate name because the two defaults differ: the gateway trusts every peer,
SSH trusts loopback only. Empty keeps the advisory's protection. A header from a peer outside the
list is still rejected, and a malformed range stops the server at startup instead of panicking
in MustPolicyFromRanges.

Trust is by source address, so any peer in a listed range can declare a client address. A
deployment that lists a pod network has to restrict who reaches port 2222, for example with a
NetworkPolicy that admits only the ingress pods. The documentation warns Compose users off the
Docker bridge range, which the userland proxy makes every external client appear to come from.

Fixes: shellhub-io/team#362
@otavio
otavio force-pushed the fix/ssh-proxy-trusted-ips branch from c94a9d6 to 249b577 Compare October 23, 2026 21:33
@otavio

otavio commented Oct 23, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown

Documentation preview: https://7441cede.shellhub-docs.pages.dev

Comment thread ui/apps/docs/src/pages/selfhosted/troubleshooting.mdx Outdated
Comment thread ui/apps/docs/src/pages/about/ports-and-connectivity.mdx Outdated
The load balancer, troubleshooting, firewall rule, access policy and session pages said the
client address of SSH connections comes from SHELLHUB_PROXY_TRUSTED_IPS, which only the
gateway reads. They now name SHELLHUB_SSH_PROXY_TRUSTED_IPS for SSH clients. The gateway variable
still decides the address of web terminal sessions: the gateway sets X-Real-IP, and the web
terminal hands that address to the SSH session it opens on loopback.

Troubleshooting also covers the symptom seen on cloud.shellhub.io: SSH closing before the banner,
with the server's rejection warning.

Refs: shellhub-io/team#362
@otavio
otavio force-pushed the fix/ssh-proxy-trusted-ips branch from 249b577 to 1dbb378 Compare October 23, 2026 21:40
@otavio

otavio commented Oct 23, 2026

Copy link
Copy Markdown
Member Author

/review

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Claude finished @otavio's task in 3m 4s —— View job


Code Review Complete

Reviewed 13 files on all three axes: Standards (repo rules, language conventions, tests, code smells), Spec (does it implement the issue?) and Correctness (security and runtime defects). No new issues found. The code looks good as-is.

@github-actions

Copy link
Copy Markdown

Documentation preview: https://76be20d3.shellhub-docs.pages.dev

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant