refactor(ingress): one CAA flow for every dns-01 setup

custom-domain/dstack-ingress/DNS_PROVIDERS.md

@@ -21,7 +21,7 @@ This guide explains how to configure dstack-ingress to work with different DNS p
 
 - `DNS_PROVIDER` - DNS provider to use (`cloudflare`, `linode`, `namecheap`, `route53`). Detected from the provider's credential variable when unset; set it explicitly for `route53`
 - `ACME_EMAIL` - Contact address for the ACME account (`CERTBOT_EMAIL` still works). Published in the evidence, so leave it unset unless you want one
-- `SET_CAA` - Enable CAA record setup: `true` or `false` (default). Except on Cloudflare, this makes a non-wildcard domain's pointer an A record; see the README
+- `SET_CAA` - Enable CAA record setup: `true` or `false` (default `false`; `true` with `DELEGATION_ZONE`). Except on Cloudflare, this makes a non-wildcard domain's pointer an A record; see the README
 - `PORT` - HTTPS port (default: 443)
 - `TXT_PREFIX` - Prefix for TXT records (default: "_dstack-app-address")
 
@@ -112,7 +112,7 @@ PolicyDocument:
 **Important Notes for Route53:**
 - The certbot plugin uses the format `certbot-dns-route53` package
 - CAA values are merged into the existing record set at the name: only values with the same tag are replaced, so `iodef` and other tags are kept
-- Where a CAA record shares the name (`DELEGATION_ZONE`, `SET_CAA=true` on a non-wildcard domain, or any CAA already there), the gateway pointer is an A record to the resolved gateway IP instead of a CNAME, replacing any CNAME or AAAA in the same change. It is refreshed every `RENEW_INTERVAL`
+- Where a CAA record shares the name (`SET_CAA=true` under `DELEGATION_ZONE`, where it is the default, or on a non-wildcard domain, or any CAA already there), the gateway pointer is an A record to the resolved gateway IP instead of a CNAME, replacing any CNAME or AAAA in the same change. It is refreshed every `RENEW_INTERVAL`
 - Wildcard record names are listed by Route53 as `\052.<domain>`; they are matched as `*.<domain>`
 - It is essential that the AWS service account used can only assume the limited role. See cloudformation example.
 

custom-domain/dstack-ingress/README.md

@@ -175,7 +175,7 @@ otherwise arbitrary — the gateway answers every name under its domain.
 |----------|---------|-------------|
 | `PORT` | `443` | HAProxy listen port |
 | `ROUTING_MAP` | | Multi-domain routing: `domain=host:port` per line. A wildcard entry (`*.example.com=host:port`) matches one label, and exact entries take precedence over it. Unmatched names go to the first entry. It routes only: every name still needs to be in `DOMAIN`/`DOMAINS` to get a certificate |
-| `SET_CAA` | `false` | Enable CAA DNS record (dns-01 only; tls-alpn-01 cannot write DNS). Replaces existing values with the same tag; for a wildcard whose base is a zone apex this restricts the whole zone (see [Wildcard Domain Support](#wildcard-domain-support)). Except on Cloudflare, a non-wildcard domain is then published as an A record to the gateway's current address instead of a CNAME, because a CNAME would hide the CAA beside it; the address is refreshed every `RENEW_INTERVAL`. The same happens whenever a CAA record already exists at the name, whoever wrote it, so turning `SET_CAA` off later keeps the A record until you delete the CAA. Linode always uses an A record. With `DELEGATION_ZONE` the CAA is always published, whatever this says |
+| `SET_CAA` | `false` | Enable CAA DNS record (dns-01 only; tls-alpn-01 cannot write DNS). Replaces existing values with the same tag; for a wildcard whose base is a zone apex this restricts the whole zone (see [Wildcard Domain Support](#wildcard-domain-support)). Except on Cloudflare, a non-wildcard domain is then published as an A record to the gateway's current address instead of a CNAME, because a CNAME would hide the CAA beside it; the address is refreshed every `RENEW_INTERVAL`. The same happens whenever a CAA record already exists at the name, whoever wrote it, so turning `SET_CAA` off later keeps the A record until you delete the CAA. Linode always uses an A record. Defaults to `true` with `DELEGATION_ZONE`; turning it off there lets anyone who can write the delegation zone get certificates for the domain. Either way, the CAA in force is checked before every issuance and a domain it refuses is not sent to the CA |
 | `TXT_PREFIX` | `_dstack-app-address` | DNS TXT record prefix |
 | `ACME_EMAIL` | | ACME contact address, in either mode. `CERTBOT_EMAIL` is the historical name and still works. See [below](#the-acme-contact-address-is-optional-and-public) — it is optional, and published |
 | `ACME_STAGING` | `false` | Use Let's Encrypt staging, in either mode. `CERTBOT_STAGING` is the historical name and still works |
@@ -188,7 +188,7 @@ otherwise arbitrary — the gateway answers every name under its domain.
 | `DNS_WEBHOOK_URL` | | `DNS_SETUP_MODE=webhook`: endpoint to notify |
 | `DNS_WEBHOOK_TOKEN` | | Shared secret; the payload is HMAC-SHA256 signed with it |
 | `DNS_WEBHOOK_TIMEOUT` | `15` | Seconds per webhook POST attempt |
-| `DOH_RESOLVERS` | Google + Cloudflare | Comma-separated DoH endpoints used to verify records. `DOH_RESOLVER` is the historical name and still works |
+| `DOH_RESOLVERS` | Google + Cloudflare | Comma-separated DoH endpoints used to verify records, and to check CAA before issuing. `DOH_RESOLVER` is the historical name and still works |
 | `TLSALPN_ADDRESS` | `127.0.0.1` | Address lego's ACME responder binds to, and haproxy forwards `acme-tls/1` connections to |
 | `TLSALPN_PORT` | `9443` | Loopback port lego's ACME responder binds to |
 | `TLS_TERMINATE_PORT` | `9444` | Loopback port the TLS frontend moves to in tls-alpn-01 mode |
@@ -243,12 +243,12 @@ dstack-ingress publishes what they point at, in the delegation zone:
 
 ```
 svc.example.com.deleg.example.net                      A      <address of GATEWAY_DOMAIN>
-svc.example.com.deleg.example.net                      CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…"
+svc.example.com.deleg.example.net                      CAA    0 issue "letsencrypt.org;validationmethods=dns-01;accounturi=…"   (SET_CAA, on by default here)
 _dstack-app-address.svc.example.com.deleg.example.net  TXT    "<app-id>:<port>"
 _acme-challenge.svc.example.com.deleg.example.net      TXT    <challenge>   (transient)
 ```
 
-The gateway pointer and the CAA share a name. Cloudflare answers a CAA query
+With `SET_CAA` the gateway pointer and the CAA share a name. Cloudflare answers a CAA query
 with the CAA beside a CNAME, so the pointer stays a CNAME there. Elsewhere it is
 an A record: by the DNS rules resolvers follow a CNAME without looking beside it,
 and Route53 and Linode refuse the pair. The address

custom-domain/dstack-ingress/TESTING.md

@@ -284,8 +284,9 @@ These fail fast and are cheap, so run them on every change:
 | Scenario | Expected |
 |---|---|
 | tls-alpn-01 + a wildcard domain | Refused before any ACME call, citing RFC 8737 |
-| Delegation, CAA in the delegated zone changed to forbid the CA | Blocked before any ACME attempt — proves the published CAA is reachable through the CNAME |
-| Delegation with no CAA record at all | Proceeds, reporting `no CAA record set; issuance is unrestricted` — see below |
+| Delegation, `SET_CAA=false`, CAA in the delegated zone changed to forbid the CA | Blocked before any ACME attempt — proves a CAA there is reachable through the CNAME. With `SET_CAA` on, the next pass replaces it instead |
+| dns-01, `SET_CAA=false`, a CAA naming another CA or pinned to another account | Blocked before any ACME attempt with `CAA forbids issuing <domain> with this account` and the reason quoted back |
+| Delegation, `SET_CAA=false`, no CAA record at all | Proceeds, reporting `no CAA record set; issuance is unrestricted` — see below |
 | Delegation, later pass, gateway CNAME broken | Renewal proceeds — that record is for serving, and dns-01 does not use it |
 | A CAA record with `validationmethods=dns-01`, mode tls-alpn-01 | Blocked with the restriction quoted back |
 | TXT holding the wrong value | Reported as `TXT present but wrong value`, not "missing" |
@@ -298,7 +299,7 @@ it is observable rather than merely argued.
 had to create the CAA record, its absence was the state that mattered: nothing
 then stopped anyone else who could satisfy the delegated challenge, and the
 container held no token to create it. Once the container began publishing the
-delegated CAA itself, absence stopped meaning that — it means "not published
+delegated CAA itself (`SET_CAA`, the default under delegation), absence stopped meaning that — it means "not published
 yet", and the next pass fixes it — so the gate was removed along with
 `ALLOW_MISSING_CAA` and dnsguide's `--caa-required`. `check_caa` now follows
 RFC 8659 for delegation as it always did elsewhere: an absent CAA record set
@@ -310,19 +311,20 @@ attempt, which is what proves the published record is reachable through the
 operator's CNAME. Expect the reason quoted back, e.g. `issue allows
 'example.com', not letsencrypt.org`.
 
-Note also that the first pass on a fresh deployment necessarily runs before any
-CAA exists: the record names the ACME account, so the account has to exist
-first. `process_domain` issues, writes the CAA, and issues again only if the
-first attempt failed and had to create the account.
+Note also that the CAA record names the ACME account, so `process_domain`
+registers the account on its own first. Every dns-01 pass runs the same steps
+in the same order — register, publish the records (CAA with `SET_CAA`), check
+that the CAA in force lets this account issue, issue once — so even a fresh
+deployment's first certificate is issued under the pin.
 
 ## Traps
 
-**Start from a clean zone.** Records left over from an earlier run change which
-branch executes. A stale CAA record naming a previous ACME account makes dns-01's
-first issuance attempt fail and the second succeed — which happens to be the path
-where the code was correct, hiding a bug in the ordinary path where the first
-attempt succeeds. Delete test records between runs and verify against the
-authoritative nameserver, not a cached resolver.
+**Start from a clean zone.** Records left over from an earlier run change what a
+pass does. A stale CAA record naming a previous ACME account is rewritten with
+`SET_CAA=true`, but resolvers may keep the old one for its TTL, so the CAA check
+fails a pass or two before issuance; with `SET_CAA=false` it blocks for good.
+Delete test records between runs and verify against the authoritative
+nameserver, not a cached resolver.
 
 **One resolver is not enough to trust either answer.** Public resolvers cache
 negative answers for the zone's SOA minimum, so a resolver queried before a

custom-domain/dstack-ingress/scripts/cert-lib.sh

@@ -90,10 +90,9 @@ run_pass() {
         # "nothing to do", and under `set -e` a bare non-zero would kill the
         # whole script on every quiet renewal pass.
         if ( process_domain "$domain" ); then status=0; else status=$?; fi
-        # A step after issuance can still fail (a CAA write, a timeout that
-        # kills the client after it saved the certificate). The new certificate
-        # is on disk by then and the next pass will find it current, so apply
-        # it now or never.
+        # Issuance can fail after saving the certificate (a timeout that kills
+        # the client). The new certificate is on disk by then and the next pass
+        # will find it current, so apply it now or never.
         [ "$(cert_checksum "$domain")" = "$before" ] || changed=1
         case $status in
             0) changed=1 ;;

custom-domain/dstack-ingress/scripts/certman.py

@@ -706,6 +706,51 @@
             return
         print(f"Renewal window for {domain} set to {days} days before expiry")
 
+    def register_account(self, email: str) -> bool:
+        """Create the ACME account without issuing, so the CAA can name it
+        before the first order."""
+        cmd = self._get_certbot_command() + [
+            "register", "--non-interactive", "--agree-tos", "--no-eff-email",
+            "--server", acme_server()]
+        # Contactless unless an address is given; see _build_certbot_command.
+        cmd += ["--email", email] if email else ["--register-unsafely-without-email"]
+        try:
+            result = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
+        except subprocess.TimeoutExpired:
+            print("ACME account registration timed out", file=sys.stderr)
+            return False
+        if result.returncode != 0 or not self.acme_account_exists():
+            print(f"ACME account registration failed: {result.stderr.strip()}", file=sys.stderr)
+            return False
+        print(f"✓ ACME account registered with {acme_server()}")
+        return True
+
+    def _lineage_account_current(self, domain: str) -> bool:
+        """Whether `certbot renew` would find the account the lineage names.
+
+        certbot renews with the account recorded in the lineage and fails once
+        that one is gone, and register_account runs before every pass, so
+        "an account exists" no longer tells. Deterministic certificates are
+        issued outside any lineage, with whatever account is current. A
+        lineage that records no account is left to certbot rather than
+        replaced on every pass.
+        """
+        current = account_file()
+        if current is None:
+            return False
+        if deterministic_key.enabled():
+            return True
+        recorded = None
+        try:
+            with open(self._renewal_conf_path(domain), encoding="utf-8") as fh:
+                for line in fh:
+                    match = re.match(r"\s*account\s*=\s*(\S+)", line)
+                    if match:
+                        recorded = match.group(1)
+        except OSError:
+            pass
+        return recorded is None or recorded == os.path.basename(os.path.dirname(current))
+
     def acme_account_exists(self) -> bool:
         """Check if an ACME account exists for the current server."""
         return account_file() is not None
@@ -731,14 +776,14 @@
                 # certbot always records the server; a lineage without one has
                 # a broken renewal config, which re-issuing would not repair.
                 server = acme_server()
-            if server == acme_server() and self.acme_account_exists():
+            if server == acme_server() and self._lineage_account_current(domain):
                 success, renewed = self.renew_certificate(domain, email)
                 return success, renewed  # Only generate evidence if actually renewed
             if server != acme_server():
                 print(f"Certificate for {domain} came from {server or 'an unknown ACME server'}, "
                       f"replacing it from {acme_server()}")
             else:
-                print(f"Certificate exists for {domain} but the ACME account is missing, "
+                print(f"Certificate for {domain} belongs to an ACME account that is gone, "
                       f"re-obtaining")
             success = self.obtain_certificate(domain, email, replace=True)
             return success, success
@@ -758,7 +803,7 @@
     )
     parser.add_argument(
         "action", choices=["obtain", "renew", "auto", "setup", "cert-dir",
-                            "account-file"],
+                            "account-file", "register"],
         help="Action to perform"
     )
     parser.add_argument("--domain", help="Domain name")
@@ -775,6 +820,8 @@
             sys.exit(1)
         print(path)
         return
+    if args.action == "register" and account_file():
+        return  # every pass asks; skip loading the provider SDKs
     if args.action == "cert-dir":
         if not args.domain:
             parser.error("--domain is required")
@@ -793,6 +840,15 @@
             print(f"Setup completed for {manager.provider_type} provider")
             return
 
+        # The contact address is optional; see _build_certbot_command.
+        if not args.email:
+            args.email = os.environ.get("ACME_EMAIL") or os.environ.get(
+                "CERTBOT_EMAIL", ""
+            )
+
+        if args.action == "register":
+            sys.exit(0 if manager.register_account(args.email) else 1)
+
         # Domain is required for certificate operations
         if not args.domain:
             print(
@@ -801,12 +857,6 @@
             )
             sys.exit(1)
 
-        # The contact address is optional; see _build_certbot_command.
-        if not args.email:
-            args.email = os.environ.get("ACME_EMAIL") or os.environ.get(
-                "CERTBOT_EMAIL", ""
-            )
-
         success, needs_evidence = manager.run_action(
             args.domain, args.email, args.action
         )