From 3c0f57edac22e89f18c0f4e4c5c6ad70ee912822 Mon Sep 17 00:00:00 2001 From: Ramkumar Ethiraj <64294490+rmkr-dev@users.noreply.github.com> Date: Tue, 3 Nov 2026 05:12:17 +0000 Subject: [PATCH 1/2] Add HTTP Idempotency-Key guardrails skill for Spring Boot APIs --- skillbook/skills.json | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/skillbook/skills.json b/skillbook/skills.json index 97c2a96..a7fcf2e 100644 --- a/skillbook/skills.json +++ b/skillbook/skills.json @@ -940,6 +940,26 @@ ], "formatNotes": "All three. Copilot: path-scoped `.github/instructions/jvm-container.instructions.md` with applyTo for `**/Dockerfile*`, `**/k8s/**`, `**/kustomization.yaml`, `**/deployment*.yaml`, and `**/application*.yml`, or `.github/skills/jvm-container-memory-guardrails/SKILL.md`; one always-on line in `.github/copilot-instructions.md` (\"heap is a percentage of the container limit, never -Xmx equal to it; heap plus non-heap plus headroom must fit\"). Claude Code: `.claude/skills/jvm-container-memory-guardrails/SKILL.md`, plus a CLAUDE.md line naming the JDK, collector, and where JVM flags are set. Codex: a \"JVM in containers\" section in root `AGENTS.md` (or the same SKILL.md under `.agents/skills/`). Pair with aks-hpa-requests-limits-guardrails for requests, limits, and scaling, k8s-probes-graceful-shutdown-guardrails for startup probes, container-image-hardening for the base image, java-virtual-threads-spring-guardrails for thread counts, and spring-boot-micrometer-prometheus-guardrails for JVM metrics. This skill owns how the JVM fits inside its container.", "body": "# Skill: JVM Container Memory and GC Guardrails for AKS\n\n## Goal\nEvery Java service knows its memory budget: heap plus everything that is not heap fits under the container limit with room to spare, the JVM sees the CPU count you intended, the garbage collector is chosen on purpose, and a Java `OutOfMemoryError` ends the process cleanly with evidence instead of a silent `OOMKilled` (exit code 137) with nothing in the logs.\n\n## When to apply\n- Writing or reviewing the Dockerfile, entrypoint, `JAVA_TOOL_OPTIONS` / `JDK_JAVA_OPTIONS`, or `resources` block for a JVM workload\n- Pods restart with `OOMKilled` while heap graphs look fine\n- Slow startup, failing startup probes, or long GC pauses after a CPU limit change\n- Moving to a new JDK, a new base image, cgroup v2 node pools, virtual threads, or Netty-based clients\n- Someone proposes `-Xmx` equal to the memory limit, or \"just double the limit\"\n\n## Know what the limit covers\nThe kernel enforces the memory limit on the whole process (resident set plus page cache charged to the cgroup), not on the heap. The JVM only sizes the heap from the limit. Budget the rest explicitly:\n\n```text\ncontainer limit >= heap max\n + metaspace (classes; Spring apps often 100-250 MiB)\n + code cache (JIT; up to ReservedCodeCacheSize, 240 MiB by default)\n + thread stacks (platform threads x committed stack)\n + direct / NIO buffers (Netty, gRPC, HTTP clients; defaults to heap max)\n + GC structures (G1/ZGC bookkeeping, a few % of heap)\n + native libs, malloc arenas, mapped files\n + headroom (10-20%)\n```\n\nMeasure, then decide. Typical results for a Spring Boot service land the heap at 50-70% of the limit, not 75-90%.\n\n## Baseline rules\n1. **Container-aware JDK only.** Use a current LTS JDK whose cgroup v2 support is in place (AKS Ubuntu 22.04+ and Azure Linux node pools are cgroup v2). Confirm with `java -Xlog:os+container=info -version` inside the image, and `cat /sys/fs/cgroup/memory.max` in the pod.\n2. **Size heap as a share of the limit, not a hard-coded number.** Set `-XX:MaxRAMPercentage` (start near 60-70 and adjust from measurement) so a limit change resizes the heap. Never set `-Xmx` equal to the limit. If you must pin `-Xmx`, also pin the limit next to it and review them together.\n3. **Memory request equals memory limit.** Memory is not compressible; a request below the limit invites node pressure evictions. Change both together. A fixed heap (`-XX:InitialRAMPercentage` equal to the max) avoids resize churn for latency-sensitive services; the memory is reserved anyway.\n4. **Cap what can grow unbounded.** Set `-XX:MaxDirectMemorySize` when Netty, gRPC, or NIO buffers are in play; set `-XX:MaxMetaspaceSize` only after measuring (a cap below real need turns into `OutOfMemoryError: Metaspace`). Bound thread pools; with platform threads, each thread costs stack memory. Consider `MALLOC_ARENA_MAX=2` for glibc images with many threads after measuring the gain.\n5. **CPU the JVM can see.** Current JDKs derive the processor count from the CPU limit (quota), not from requests or shares. With no CPU limit, the JVM sees every core on the node and sizes GC and common pool threads for it. Either set a CPU limit, or set `-XX:ActiveProcessorCount` to match the CPU you actually give the pod. Below 2 processors or about 1792 MiB of memory the JVM silently picks SerialGC.\n6. **Choose the collector on purpose.** G1 for most request/response services; generational ZGC for large heaps or strict tail-latency goals (it costs more CPU and memory); Parallel for batch throughput; Serial only for tiny single-core tools. Write the choice and the reason in the Dockerfile or manifest comment.\n7. **Fail fast and keep evidence.** `-XX:+ExitOnOutOfMemoryError` so a broken JVM restarts instead of limping. `-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps` on a mounted volume sized for the heap (an `emptyDir` with a `sizeLimit` and a collection step, or a persistent volume). Heap dumps contain secrets, tokens, and personal data: restrict access, never upload them to tickets or public storage, and delete them after analysis.\n8. **One place for JVM flags.** Put flags in one environment variable (`JDK_JAVA_OPTIONS` for the `java` launcher, or `JAVA_TOOL_OPTIONS` when a buildpack or tool expects it), set in the manifest or image, not split across Dockerfile `CMD`, entrypoint scripts, and Helm values. If you use Paketo buildpacks, its memory calculator sizes the heap from the limit and `BPL_JVM_THREAD_COUNT` / `BPL_JVM_HEAD_ROOM`; tune those instead of adding a competing `-Xmx`.\n9. **Startup needs CPU.** JIT and class loading burn CPU in the first minute. A tight CPU limit throttles startup and fails probes. Use a `startupProbe` sized for the measured startup time, and consider Class Data Sharing or the JDK AOT cache (Spring Boot supports building a CDS archive) before raising limits.\n10. **Measure under load, not at idle.** Enable `-XX:NativeMemoryTracking=summary` in a load test (it costs a few percent, keep it off by default in production), take `jcmd 1 VM.native_memory baseline`, drive peak traffic, then `jcmd 1 VM.native_memory summary.diff scale=MB`. Compare NMT's committed total to the pod working set; the gap is native memory NMT cannot see, and it belongs in the budget.\n11. **Watch the right graphs.** Container: `container_memory_working_set_bytes` against the limit, and the last termination reason (`OOMKilled`). JVM (Micrometer): `jvm_memory_used_bytes` by area and pool, `jvm_buffer_memory_used_bytes`, `jvm_threads_live_threads`, `jvm_gc_pause_seconds`, and GC overhead. Alert on working set above 90% of the limit and on any OOMKilled restart, not only on heap usage.\n12. **Do not scale on JVM memory.** A JVM keeps heap it has grown into, so memory-based autoscaling rarely scales down. Scale on CPU, request rate, or queue depth.\n\n## Copy-ready baseline (Deployment snippet)\n```yaml\ncontainers:\n - name: app\n image: myregistry.azurecr.io/orders-api:1.4.2\n # ActiveProcessorCount matches the CPU limit; keep it if the CPU limit is removed\n env:\n - name: JDK_JAVA_OPTIONS\n value: >-\n -XX:MaxRAMPercentage=65\n -XX:InitialRAMPercentage=65\n -XX:MaxDirectMemorySize=192m\n -XX:+UseG1GC\n -XX:ActiveProcessorCount=2\n -XX:+ExitOnOutOfMemoryError\n -XX:+HeapDumpOnOutOfMemoryError\n -XX:HeapDumpPath=/dumps\n -Xlog:gc*:stdout:time,uptime,level,tags\n resources:\n requests: { cpu: \"1\", memory: 1536Mi }\n limits: { cpu: \"2\", memory: 1536Mi } # memory request == limit\n volumeMounts:\n - { name: dumps, mountPath: /dumps }\nvolumes:\n - name: dumps\n emptyDir: { sizeLimit: 2Gi }\n```\n\nCheck what actually took effect:\n```bash\nkubectl exec deploy/orders-api -- java -XX:+PrintFlagsFinal -version | grep -E 'MaxHeapSize|ActiveProcessorCount|Use(G1|Z|Serial|Parallel)GC '\nkubectl exec deploy/orders-api -- jcmd 1 VM.flags\nkubectl exec deploy/orders-api -- jcmd 1 GC.heap_info\n```\n`jcmd 1` assumes the JVM is PID 1 and the image ships `jcmd`. JRE-only or distroless images need `kubectl debug` with a JDK ephemeral container that shares the process namespace.\n\n## Review checklist\n- [ ] Memory budget written down: heap, metaspace, code cache, threads, direct, native, headroom\n- [ ] Heap set as a percentage of the limit (or pinned next to the limit with a comment)\n- [ ] Memory request equals memory limit\n- [ ] Processor count the JVM sees matches the CPU given to the pod; not silently on SerialGC\n- [ ] Collector chosen with a written reason\n- [ ] Direct memory capped where Netty or NIO is used; thread pools bounded\n- [ ] ExitOnOutOfMemoryError on; heap dumps go to a sized, access-controlled volume\n- [ ] JVM flags in one place; buildpack calculator not fighting a manual `-Xmx`\n- [ ] Startup probe sized from measured startup under the CPU limit\n- [ ] NMT measurement from a load test recorded; working-set and OOMKilled alerts exist\n\n## Anti-patterns\n- `-Xmx` equal to (or above) the container memory limit\n- Raising the limit after every OOMKilled without measuring what grew\n- No CPU limit and no `ActiveProcessorCount`, so a 2-core pod sizes GC threads for a 64-core node\n- Memory request far below the limit on a JVM that never gives heap back\n- Heap dumps written to the container's writable layer, or copied to a shared drive\n- Autoscaling a JVM on memory utilization\n- Flags set in three places that override each other\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: jvm-container-memory-guardrails\ndescription: Use when writing or reviewing JVM settings, Dockerfiles, or Kubernetes resources for Java services on AKS: memory budget of heap plus metaspace, code cache, threads, direct buffers and native memory under the container limit; MaxRAMPercentage instead of -Xmx equal to the limit; memory request equal to limit; CPU count the JVM sees and ActiveProcessorCount; collector choice; ExitOnOutOfMemoryError and access-controlled heap dumps; one place for JVM flags; startup CPU; NMT measurement under load; working-set and OOMKilled alerts.\n---\n\nFollow JVM Container Memory and GC Guardrails: write a memory budget where heap plus metaspace, code cache, thread stacks, direct buffers, native memory, and 10-20% headroom fit under the container limit. Size heap with MaxRAMPercentage (start near 60-70, adjust from measurement), never -Xmx equal to the limit. Set memory request equal to the limit. Make the processor count the JVM sees match the pod's CPU (CPU limit or ActiveProcessorCount) and avoid accidental SerialGC. Choose G1, generational ZGC, or Parallel with a written reason. Cap direct memory where Netty or NIO is used. Turn on ExitOnOutOfMemoryError and write heap dumps to a sized, access-controlled volume. Keep JVM flags in one environment variable. Size the startup probe for startup under the CPU limit. Measure with NMT under load, compare with the pod working set, and alert on working set above 90% of the limit and on OOMKilled restarts. Do not autoscale on JVM memory.\n```\n" + }, + { + "id": "api-idempotency-key-guardrails", + "title": "HTTP Idempotency-Key Guardrails for Spring Boot APIs", + "whenToUse": "Adding or reviewing POST and PATCH endpoints with side effects (orders, payments, messages, paid provider calls) in Spring Boot APIs, or the client retries that hit them — a per-endpoint Idempotency-Key contract (required or optional, retention), key validation and scoping to client + method + route template, a request fingerprint with 422 on reuse with a different body, an atomic claim with 409 for a duplicate still in flight, the record committed in the same transaction as the business write, outbound side effects that carry the key, which outcomes are stored or released, leases on in-progress records, faithful replay, retention cleanup, stored responses treated as sensitive, filter order, metrics, and OpenAPI docs.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "spring-boot", + "java", + "api", + "idempotency", + "reliability", + "http" + ], + "formatNotes": "All three. Copilot: path-scoped `.github/instructions/api-idempotency.instructions.md` with applyTo for `**/*Controller.java`, `**/*Filter.java`, `**/web/**`, `**/db/migration/**`, and `**/openapi*.yaml`, or `.github/skills/api-idempotency-key-guardrails/SKILL.md`; one always-on line in `.github/copilot-instructions.md` (\"POST/PATCH with side effects take an Idempotency-Key: atomic claim, fingerprint check, record committed with the business write\"). Claude Code: `.claude/skills/api-idempotency-key-guardrails/SKILL.md`, plus a CLAUDE.md line naming the idempotency store, retention, and which endpoints require the key. Codex: an \"Idempotency\" section in root `AGENTS.md` (or the same SKILL.md under `.agents/skills/`). Pair with api-rate-limiting-guardrails for filter order and 429s, spring-boot-http-client-guardrails and spring-boot-resilience4j for client retries that reuse the key, spring-kafka-consumer-reliability-guardrails for consumer-side dedup, flyway-migration-guardrails for the table, and spring-boot-openapi-contract-guardrails for documenting the header and errors. This skill owns making unsafe HTTP methods safe to retry.", + "body": "# Skill: HTTP Idempotency-Key Guardrails for Spring Boot APIs\n\n## Goal\nA client that times out, loses its connection, or gets a 502 can send the same `POST` or `PATCH` again and the server does the work once: one order, one payment, one ticket. The retry gets the first result back, a reused key with a different body is refused, and two copies racing each other cannot both run.\n\n## When to apply\n- Adding or reviewing a `POST` / `PATCH` endpoint that creates something, moves money, sends a message, or calls a paid or rate-limited provider\n- Callers retry: mobile apps, API Management or ingress retries, Resilience4j or Spring Retry on the client, queue-driven workers, other services in the mesh\n- Duplicate rows, double charges, or \"we sent it twice\" incidents\n- Someone proposes \"just make the client stop retrying\" or a unique constraint on a business field as the only guard\n- Exposing an API to partners who expect the `Idempotency-Key` header (IETF httpapi draft, Stripe-style APIs)\n\n## Decide the contract first\nWrite these down per endpoint before any code:\n\n| Question | Typical answer |\n|---|---|\n| Which operations take a key? | Non-idempotent `POST` and `PATCH` with side effects. Not `GET`, `HEAD`, `OPTIONS`. `PUT` and `DELETE` only when they trigger side effects beyond the state change |\n| Required or optional? | Required for money, orders, and outbound messages (missing key is `400`); optional elsewhere |\n| Key scope | Authenticated client (or tenant) + method + route template + key |\n| Retention | At least the longest client retry window; 24 hours is a common default |\n| Replayed outcomes | Success and final business errors are replayed; validation errors and conflicts are not stored |\n| Store down | Fail closed (`503`) for money-moving routes; fail open only where a duplicate is harmless |\n\n## Baseline rules\n1. **The client owns the key.** It creates one random key per logical operation (UUIDv4 or at least 128 bits of randomness) *before* the first attempt and sends the same key on every retry. Never build keys from emails, account numbers, or other personal data. Validate on the server: allowed characters, a maximum length (for example 255), reject malformed keys with `400`. The draft defines the header as a structured-field string (quoted); accept the quoted form and a bare token, and normalize before storing.\n2. **Scope the key to the caller and the operation.** Store records under client identity from the validated token (`azp` / `appid`, or tenant) + HTTP method + route template + key. A global key space lets one client collide with, or read back, another client's response.\n3. **Fingerprint the request.** Hash a canonical form of what makes the request unique (method, route, path variables, relevant query parameters, and the body bytes or a canonical JSON form) with SHA-256. Same key with a different fingerprint is a client bug: answer `422` with `application/problem+json` and a link to your docs. Leave out volatile headers such as tracing ids and timestamps.\n4. **Claim atomically, before doing the work.** Insert an `IN_PROGRESS` row under a unique constraint (`INSERT ... ON CONFLICT DO NOTHING`, or Redis `SET key value NX PX`) and only proceed if you won the insert. Never read-then-insert. A second request that finds the key in progress gets `409 Conflict` with a short `Retry-After`; it does not wait on a lock and it does not run the handler.\n5. **Commit the record with the business change.** Keep the idempotency table in the same database as the data the endpoint writes, and mark the record `COMPLETED` with the stored response in the same transaction. Then \"work done\" and \"record says done\" cannot disagree. Work that leaves the database (payment provider, email, Kafka) gets the key passed along: the provider's own idempotency key, or an outbox row whose dedup id is derived from the key.\n6. **Choose what gets stored, on purpose.** Store and replay `2xx` and final business errors (for example `402` insufficient funds, `409` domain conflict). Do not store errors that happen before execution starts (malformed body, failed validation, auth): release the claim so a fixed request can reuse the key. For `5xx` and timeouts, release the claim only if the transaction rolled back and nothing left the process; if a side effect may have happened, keep the record and reconcile instead of running it again.\n7. **Lease, do not lock forever.** An `IN_PROGRESS` record carries `locked_until`, set a little longer than the request timeout. After the lease expires a retry may take over with a conditional update, but only on endpoints whose work is transactional; otherwise it gets `409` until someone reconciles. Alert on expired leases: each one is a crash or a hung request.\n8. **Replay faithfully and say so.** Return the stored status, body, and the headers that matter (`Content-Type`, `Location`, `ETag`), and add a marker header such as `Idempotent-Replayed: true`. A replay must not re-run validation against current state, emit events, or count as a new business action.\n9. **Retention is a contract.** Expire records after the documented window with a scheduled cleanup on an indexed `expires_at` (or a Redis TTL). After expiry the same key starts a new request; say so in the API docs. Never expire a record that is still `IN_PROGRESS`.\n10. **Treat stored responses as sensitive data.** Bodies can hold personal data and tokens. Cap stored size, consider storing the created resource id and re-rendering instead of the raw body, encrypt at rest if the API carries regulated data, and never log keys next to bodies (log a hash of the key).\n11. **Place it correctly in Spring.** A servlet filter or a `HandlerInterceptor` plus a body-caching wrapper, ordered after authentication (it needs the client identity) and the rate limiter, and before the transaction and the controller. Read and cache the body once for the fingerprint; capture the response with `ContentCachingResponseWrapper` and call `copyBodyToResponse()`. An annotation such as `@Idempotent(required = true)` on handler methods keeps the per-endpoint contract next to the code. Skip streaming and SSE responses, or store only a resource reference for them.\n12. **Make it observable and documented.** Count claims, replays, `409` in-flight, `422` mismatches, lease takeovers, store errors, and store latency by route (no raw keys as tags). Document the header in OpenAPI per operation: required or optional, format, retention, and the `400` / `409` / `422` problem types. Client SDKs and outbound clients reuse the key on every retry and only retry `POST` when a key is set.\n\n## Copy-ready baseline\nFlyway migration (PostgreSQL):\n```sql\ncreate table idempotency_record (\n scope_key varchar(200) not null, -- client id + method + route template\n idem_key varchar(255) not null,\n fingerprint char(64) not null, -- sha-256 hex of the canonical request\n status varchar(16) not null, -- IN_PROGRESS | COMPLETED\n locked_until timestamptz not null,\n response_status int,\n response_headers jsonb,\n response_body bytea,\n created_at timestamptz not null default now(),\n expires_at timestamptz not null,\n primary key (scope_key, idem_key)\n);\ncreate index idx_idempotency_expires on idempotency_record (expires_at);\n```\n\nClaim, then decide:\n```sql\n-- 1 row inserted: you own the key, run the handler\ninsert into idempotency_record (scope_key, idem_key, fingerprint, status, locked_until, expires_at)\nvalues (:scope, :key, :fp, 'IN_PROGRESS', now() + interval '30 seconds', now() + interval '24 hours')\non conflict (scope_key, idem_key) do nothing;\n\n-- 0 rows: load the existing record and branch\n-- fingerprint differs -> 422 problem+json\n-- COMPLETED -> replay stored response + Idempotent-Replayed: true\n-- IN_PROGRESS and locked_until > now() -> 409 problem+json + Retry-After: 1\n-- IN_PROGRESS and lease expired -> conditional takeover (transactional endpoints only)\nupdate idempotency_record\n set locked_until = now() + interval '30 seconds'\n where scope_key = :scope and idem_key = :key\n and status = 'IN_PROGRESS' and locked_until < now();\n```\n\nProblem response for a reused key:\n```http\nHTTP/1.1 422 Unprocessable Content\nContent-Type: application/problem+json\n\n{\"type\":\"https://api.example.com/problems/idempotency-key-reused\",\n \"title\":\"Idempotency-Key was already used with a different request\",\n \"status\":422}\n```\n\n## Review checklist\n- [ ] Each `POST` / `PATCH` with side effects says whether the key is required, optional, or not supported\n- [ ] Keys are validated (format, length) and scoped to client identity + method + route template\n- [ ] Request fingerprint compared; mismatch returns `422`\n- [ ] Claim is one atomic insert under a unique constraint; concurrent duplicate returns `409` with `Retry-After`\n- [ ] Record completion commits in the same transaction as the business write; outbound side effects carry the key\n- [ ] Stored vs not-stored outcomes written down, including `5xx` and timeouts\n- [ ] Lease on in-progress records; expired leases alerted on\n- [ ] Replays return the original status, body, and key headers, plus a replay marker\n- [ ] Retention documented and enforced by an indexed cleanup or TTL\n- [ ] Stored bodies size-capped and treated as sensitive; keys hashed in logs\n- [ ] Filter order: authentication, rate limiting, idempotency, transaction, controller\n- [ ] Metrics, OpenAPI header and error docs, and client retry code reuse the same key\n- [ ] Tests: replay, mismatch, two concurrent requests, crash mid-request, expiry\n\n## Anti-patterns\n- Check-then-insert (`findById` then `save`) instead of an atomic claim\n- One global key space shared by every client\n- Ignoring the body, so a reused key with a new amount silently returns the old result\n- Storing the record in Redis while the business write goes to PostgreSQL with no plan for the gap\n- Caching validation failures so a corrected request can never use its key\n- Re-running the handler on retry and hoping a unique constraint catches it\n- Generating a fresh key on every retry attempt in the client\n- Keeping in-progress locks with no lease, so a crashed pod blocks the key until cleanup\n- Logging full stored responses for debugging\n\n## Copy-ready SKILL.md stub\n```markdown\n---\nname: api-idempotency-key-guardrails\ndescription: Use when adding or reviewing POST/PATCH endpoints with side effects in Spring Boot APIs, or client retries against them: Idempotency-Key header contract (required or optional, retention), key validation and scoping to client + method + route, request fingerprint with 422 on mismatch, atomic claim with 409 for in-flight duplicates, record committed with the business write, outbound side effects carrying the key, stored vs released outcomes, leases, faithful replay, retention cleanup, sensitive stored bodies, filter order, metrics and OpenAPI docs.\n---\n\nFollow HTTP Idempotency-Key Guardrails: decide per endpoint whether the key is required, optional, or unsupported, and document retention (24 hours unless the retry window is longer). The client creates one random key per operation and reuses it on every retry. Validate the key and scope it to client identity + method + route template. Fingerprint the canonical request; same key with a different fingerprint returns 422 problem+json. Claim with one atomic insert under a unique constraint before doing work; a duplicate in flight gets 409 with Retry-After. Mark the record completed with the stored response in the same transaction as the business write, and pass the key to providers or the outbox. Store success and final business errors; release the claim for pre-execution failures and for rolled-back 5xx; reconcile when a side effect may have happened. Use a lease on in-progress records. Replay status, body, and key headers with Idempotent-Replayed: true. Clean up expired records on an index. Treat stored bodies as sensitive and hash keys in logs. Order filters: auth, rate limit, idempotency, transaction, controller. Add metrics, OpenAPI docs, and tests for replay, mismatch, concurrency, crash, and expiry.\n```\n" } ] } From 37f26977d28b353ae4ec65769dcfde4ce3ecd6c3 Mon Sep 17 00:00:00 2001 From: Ramkumar Ethiraj <64294490+rmkr-dev@users.noreply.github.com> Date: Tue, 3 Nov 2026 05:12:17 +0000 Subject: [PATCH 2/2] Add prompt for adding Idempotency-Key support to a Spring Boot API --- promptbook/prompts.json | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/promptbook/prompts.json b/promptbook/prompts.json index 3e3d3eb..09cd596 100644 --- a/promptbook/prompts.json +++ b/promptbook/prompts.json @@ -942,6 +942,26 @@ ], "formatNotes": "All three. Point the tool at the Dockerfile or buildpack config, entrypoint scripts, the Deployment, Kustomize or Helm values that set resources and JVM env vars, and `application*.yml`, and paste `kubectl describe pod` last state, working-set graphs, GC logs, or NMT output if you have them. Copilot: the JVM container path instructions plus those files in agent mode (or save it as `.github/prompts/right-size-jvm-container-memory.prompt.md`). Claude Code: the jvm-container-memory-guardrails skill plus the files. Codex: AGENTS.md JVM in containers section, then the diff. The prompt measures before it raises any limit and marks every number as measured or estimated.", "body": "Task: Right-size memory and GC for this Java service on Kubernetes so it stops getting OOMKilled (or stops wasting memory), the heap and non-heap fit the container limit with stated headroom, and the JVM sees the CPU it is given. Finish with a memory budget table, a manifest and image diff, a measurement plan, and alerts.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Service: {{SERVICE}} (e.g. orders-api; Spring Boot version, web stack such as MVC, WebFlux, or Netty clients)\n- JDK and image: {{JDK}} (e.g. Temurin 21 JRE on Ubuntu, distroless Java 21, Paketo buildpack image)\n- Current resources: {{RESOURCES}} (requests and limits for CPU and memory, HPA min and max)\n- Current JVM flags: {{FLAGS}} (Dockerfile CMD/ENTRYPOINT, JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS, Helm or Kustomize values)\n- Threads: {{THREADS}} (platform or virtual threads; known pool sizes such as Tomcat, Kafka consumers, executors)\n- Latency goal: {{SLO}} (e.g. p99 under 300 ms, batch throughput only)\n- Symptoms: {{SYMPTOMS}} (e.g. OOMKilled exit 137 every few days, OutOfMemoryError: Direct buffer memory, slow startup, long GC pauses, CPU throttling)\n- Evidence: {{EVIDENCE}} (pod events, `kubectl describe` last state, working-set graphs, GC logs, NMT output if any)\n\nRequirements:\n\n1. Find every JVM setting\n - List all places flags are set and which one wins (explicit `-Xmx` beats percentages; later flags override earlier ones)\n - Show the effective values: heap max, processor count, selected collector (from `PrintFlagsFinal`, `jcmd VM.flags`, or by reasoning from the limits if you cannot run them, marked as reasoning)\n - Flag accidental SerialGC (under 2 visible processors or about 1792 MiB) and a JVM that sees all node cores because there is no CPU limit\n\n2. Memory budget\n - Table: region | current | measured or estimated | cap flag | notes, for heap, metaspace, code cache, thread stacks, direct buffers, GC overhead, native and malloc, headroom\n - Mark every number as measured or estimated; do not invent measurements\n - Derive the limit as heap plus non-heap plus 10-20% headroom, and `MaxRAMPercentage` as heap divided by limit\n\n3. Diagnose the symptom\n - Kernel OOMKilled (exit 137, no Java stack trace) versus Java `OutOfMemoryError` (heap, Metaspace, Direct buffer memory, unable to create native thread): explain which it is from {{EVIDENCE}} and what that implies\n - If the gap between heap and working set is large, name the likely regions and how to confirm\n\n4. Changes\n - Heap from `MaxRAMPercentage` (and `InitialRAMPercentage` if a fixed heap suits {{SLO}}); remove `-Xmx` equal to or near the limit\n - Memory request equal to the limit\n - `ActiveProcessorCount` or a CPU limit so visible processors match the CPU given; justify the CPU limit choice\n - Collector choice with a reason tied to heap size and {{SLO}}\n - `MaxDirectMemorySize` if Netty, gRPC, or NIO buffers are used; bounded thread pools if {{THREADS}} is platform threads\n - `ExitOnOutOfMemoryError`; heap dumps to a sized `emptyDir` or volume with restricted access and a cleanup step\n - All flags in one environment variable; if the image uses the Paketo memory calculator, tune it instead of adding `-Xmx`\n - `startupProbe` sized from startup time measured under the CPU limit\n - GC logging to stdout at a level the log pipeline can afford\n\n5. Measurement plan\n - Load test with `-XX:NativeMemoryTracking=summary`: baseline after warm-up, peak traffic, `summary.diff`; compare NMT committed with the pod working set\n - Soak test long enough to catch slow native growth\n - Record heap after GC, GC pause p99, throughput, working set at peak, and startup time before and after\n\n6. Observability\n - Alerts: working set above 90% of the limit for 10 minutes; any OOMKilled restart; GC pause p99 above a threshold from {{SLO}}\n - Dashboard row: working set vs limit, heap used after GC, non-heap by pool, direct buffers, live threads, GC pause and overhead, CPU throttling\n - No memory-based autoscaling for this JVM; propose CPU or request-rate scaling if memory is used today\n\nConstraints:\n- Do not invent limits, measurements, or traffic numbers; use placeholders and list what must be measured.\n- Do not \"fix\" OOMKilled by only raising the limit; show what grew.\n- Do not move heap dumps off the cluster or into tickets; they can hold secrets and personal data.\n- Minimal diff; no unrelated refactors.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete files and a checklist.\n\nOutput format:\n1. Effective settings today (where set, which wins, values)\n2. Memory budget table with measured or estimated marked\n3. Diagnosis of the symptom\n4. Patch: Deployment or Kustomize patch, Dockerfile or buildpack env, probe changes\n5. Measurement plan with expected results\n6. Alerts and dashboard definitions\n7. Risks and follow-ups (e.g. leaks to chase, pools to bound, startup work such as CDS)\n\nStart with the effective settings. Measure before you raise any limit.\n" + }, + { + "id": "add-idempotency-keys", + "title": "Add Idempotency-Key Support to a Spring Boot API", + "whenToUse": "Make retried POST and PATCH requests safe in a Spring Boot API: decide per endpoint whether an Idempotency-Key is required, validate and scope keys to the caller and route, fingerprint the request (422 on reuse with a different body), claim keys atomically in a Flyway-managed table (409 for a duplicate still in flight), commit the record with the business write, pass the key to payment providers or the outbox, replay the original response, clean up after the retention window, and prove it with concurrency, crash, and expiry tests before making the header required.", + "tools": [ + "copilot", + "claude-code", + "codex" + ], + "tags": [ + "spring-boot", + "java", + "api", + "idempotency", + "reliability", + "http" + ], + "formatNotes": "All three. Point the tool at the controllers for the POST/PATCH routes, the security and rate-limit filters, the service layer and transaction boundaries, outbound provider clients or the outbox, `db/migration/`, the OpenAPI document, and `application*.yml`, and paste incident notes or logs showing duplicates if you have them. Copilot: the idempotency path instructions plus those files in agent mode (or save it as `.github/prompts/add-idempotency-keys.prompt.md`). Claude Code: the api-idempotency-key-guardrails skill plus the files. Codex: AGENTS.md idempotency section, then the diff. The prompt ships the header as optional first and proves the concurrent-duplicate test before requiring it.", + "body": "Task: Add Idempotency-Key support to this Spring Boot API so a retried POST or PATCH runs its side effects once, the retry gets the original result, a reused key with a different request is refused, and concurrent duplicates cannot both run. Finish with an endpoint contract table, a code and migration diff, tests, and a rollout plan.\n\nContext placeholders (fill from the repo or ask only if missing):\n- Service and endpoints: {{SERVICE}} (e.g. payments-api; list POST/PATCH routes and what each one creates or triggers)\n- Caller identity: {{IDENTITY}} (e.g. JWT azp/appid claim, tenant claim, API key; how the principal is available in a filter)\n- Database: {{DATABASE}} (e.g. Azure Database for PostgreSQL with Flyway; where the business writes go)\n- Other store: {{STORE}} (e.g. Azure Cache for Redis, none)\n- Side effects that leave the process: {{SIDE_EFFECTS}} (e.g. payment provider call, email, Kafka publish via outbox)\n- Retry sources: {{RETRIES}} (e.g. mobile app, API Management retry policy, Resilience4j in calling services, workers)\n- Retention: {{RETENTION}} (e.g. 24 hours; longest client retry window)\n- Current symptoms: {{SYMPTOMS}} (e.g. duplicate orders after gateway timeouts, double charges, none yet)\n\nRequirements:\n\n1. Endpoint contract\n - Table: method + route | side effect | key required/optional/unsupported | fingerprint fields | stored outcomes | store-down behaviour\n - Justify each \"required\" from {{SIDE_EFFECTS}}; mark assumptions\n - Retention from {{RETENTION}}, written into the API docs\n\n2. Key handling\n - Read `Idempotency-Key`; accept the quoted structured-field form and a bare token; validate characters and length; 400 for malformed, and for missing when required\n - Scope: client identity from {{IDENTITY}} + method + route template + key; never a global key space\n - Fingerprint: SHA-256 of a canonical request (method, route, path variables, relevant query, body bytes or canonical JSON); exclude tracing and timestamp headers\n\n3. Store and claim\n - Flyway migration for an `idempotency_record` table in {{DATABASE}} (scope, key, fingerprint, status, locked_until, response status, selected headers, body or resource reference, created_at, expires_at; primary key on scope + key; index on expires_at)\n - Atomic claim with one insert under the unique constraint; no read-then-insert\n - Branches: fingerprint mismatch -> 422 problem+json; completed -> replay with `Idempotent-Replayed: true`; in progress with live lease -> 409 problem+json + `Retry-After`; expired lease -> conditional takeover only for transactional endpoints\n - If {{STORE}} is Redis and the business data is in {{DATABASE}}, explain the consistency gap and prefer the database unless there is a measured reason not to\n\n4. Execution and outcomes\n - Mark the record completed with the response in the same transaction as the business write\n - Pass the key (or a derived id) to each item in {{SIDE_EFFECTS}}: provider idempotency key, outbox dedup id\n - Store 2xx and final business errors; release the claim for validation and auth failures and for rolled-back 5xx; keep and flag the record when a side effect may have happened\n - Replays return stored status, body, `Content-Type`, `Location`, `ETag`; they do not re-run the handler or emit events\n\n5. Spring wiring\n - A filter or `HandlerInterceptor` ordered after authentication and the rate limiter, before the transaction and controller\n - Body cached once for the fingerprint; response captured with `ContentCachingResponseWrapper` and `copyBodyToResponse()`\n - An annotation such as `@Idempotent(required = true)` (or config per route) as the per-endpoint switch; streaming and SSE excluded\n - Scheduled cleanup of expired, completed records; never delete in-progress ones\n - Stored bodies size-capped and treated as sensitive; key hashed in logs\n\n6. Clients and docs\n - OpenAPI: header parameter per operation, required flag, retention, 400/409/422 problem types\n - Callers in {{RETRIES}}: generate the key before the first attempt, reuse it on every retry, only retry POST when a key is set, honor 409 Retry-After\n\n7. Observability\n - Counters by route: claimed, replayed, in-flight 409, mismatch 422, lease takeover, store error; store latency; no raw keys as tags\n - Alerts: lease takeovers or store errors above zero; replay rate jump\n\n8. Tests\n - Unit: fingerprint canonicalization, key validation, branch decisions with a fake clock\n - Integration (Testcontainers PostgreSQL): replay returns identical status and body; mismatch 422; two concurrent requests with one key run the handler once; crash between claim and commit; expiry starts a new request\n - Contract: problem+json shapes and headers match the OpenAPI document\n\n9. Rollout\n - Ship optional first, measure how many callers send keys, then make it required per endpoint with notice\n - Feature switch in config; rollback plan\n\nConstraints:\n- Do not invent retention windows, traffic numbers, or provider idempotency features; use placeholders and list what must be confirmed.\n- Do not rely on a business unique constraint as the only duplicate guard.\n- Do not cache pre-execution validation failures against the key.\n- Minimal diff; no unrelated refactors.\n- Optimize for GitHub Copilot, Claude Code, and Codex: give concrete files and a checklist.\n\nOutput format:\n1. Endpoint contract table with assumptions marked\n2. Patch: migration, record entity or repository, filter or interceptor, annotation, problem responses, cleanup job, application.yml\n3. Outbound key propagation changes\n4. OpenAPI changes and client retry changes\n5. Metrics, alerts, and log fields\n6. Tests\n7. Rollout plan, risks, and follow-ups\n\nStart with the contract table. Prove the concurrent-duplicate test passes before wiring more endpoints.\n" } ] }