[tommy-ng0]

Adds a new config key: api.server.decisions_stream.max_concurrent_requests, next to the page_size from #4733.

What and why

Every in-flight /v1/decisions/stream request holds a page of decisions in memory. With ~1.2k bouncers pulling a large delta at once, that's what put our LAPI into an OOM loop (#4711). #4733 caps the memory per request; this caps how many requests do it at once.

api:
  server:
    decisions_stream:
      page_size: 10000
      max_concurrent_requests: 200

When set, stream requests beyond the limit wait for a free slot, held until the response is fully written. A request that waited reads the cursor after getting its slot, so it still gets an up-to-date delta. A bouncer that disconnects while queued leaves without taking a slot (503, logged at debug), and since nothing was sent its cursor doesn't move. Unset or 0 keeps the current behaviour (no limit); negative warns and also means no limit. Only GET /v1/decisions/stream is limited.

Things to know when sizing it:

• Queued requests wait with no time limit of their own. go-cs-bouncer has no client timeout, but lua-cs-bouncer gives up after STREAM_REQUEST_TIMEOUT (15s by default), so keep queue waits well under the shortest stream timeout in your fleet.
• Peak stream memory is roughly max_concurrent_requests × the memory of one page_size page. This only bounds /v1/decisions/stream; live-mode GET /v1/decisions is unpaged and unlimited, and out of scope here.
• Time spent queued shows up in the existing response time metric. Happy to add a queued/in-flight gauge here or in a follow-up if you want one.

Known limitation - your call on how to handle it. A slot is held until the response is fully written, with no time limit. A bouncer that keeps its connection open but stops reading (a hung process, a suspended VM, a link that silently drops traffic) keeps its slot, and max_concurrent_requests of them would stall delivery to every other bouncer. This only happens when the limit is on.

I tried a per-write deadline (http.ResponseController.SetWriteDeadline, renewed as data goes out). It works over HTTP/1.1, but doing it properly needs two more things. Over HTTP/2, a stalled connection only fails with the server-wide HTTP2Config.WriteByteTimeout. And the deadline has to be paused around the page queries and the cursor update, or a slow query cuts off a healthy bouncer and can advance its cursor on a truncated response. That's a bigger change to the stream code, so I've left it out. Happy to add it here or as a follow-up, whichever you prefer.

Part of #4711

How it was tested

TestLimitStreamConcurrency covers the limiter against a handler that holds its slot until released: requests beyond the limit wait, freeing one slot lets exactly one queued request in, a request canceled while queued gets a 503 without taking a slot, and 0 and -1 don't limit.

TestStreamDecisionMaxConcurrentRequests runs against the real LAPI: the config reaches the route, the slot is given back after a full response, and while one pull is stuck mid-response a second gets a 503 when the limit is 1 and a 200 when unset or negative.

$ go test ./pkg/apiserver/controllers/v1/ -run TestLimitStreamConcurrency -race -count=3 -v
    --- PASS: TestLimitStreamConcurrency/a_request_canceled_while_queued_leaves_without_taking_a_slot (0.10s)
        --- PASS: TestLimitStreamConcurrency/requests_beyond_the_limit_wait_for_a_slot/limit=1 (0.10s)
        --- PASS: TestLimitStreamConcurrency/requests_beyond_the_limit_wait_for_a_slot/limit=3 (0.10s)
        --- PASS: TestLimitStreamConcurrency/zero_or_negative_does_not_limit/limit=0 (0.00s)
        --- PASS: TestLimitStreamConcurrency/zero_or_negative_does_not_limit/limit=-1 (0.00s)
ok  	github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1	1.951s

$ go test ./pkg/apiserver/ -run 'TestStreamDecisionMaxConcurrentRequests|TestStreamDecisionPageSize' -race -count=1
ok  	github.com/crowdsecurity/crowdsec/pkg/apiserver	2.424s

$ TEST_LOCAL_ONLY=1 go test ./pkg/apiserver/... ./pkg/csconfig/... -count=1
ok  	github.com/crowdsecurity/crowdsec/pkg/apiserver	15.383s
ok  	github.com/crowdsecurity/crowdsec/pkg/apiserver/controllers/v1	0.318s
ok  	github.com/crowdsecurity/crowdsec/pkg/csconfig	0.014s

To check the tests aren't tautological, I broke the slot release, the cancel handling, the config wiring and the route wiring, one at a time. A test failed each time.

golangci-lint 2.13.2, run like make lint for linux, windows and freebsd: no issues in the changed files.

Not yet done: a local run against a real instance. Docs: crowdsecurity/crowdsec-docs#1180.

Checklist

• One concern only.
• No break to LAPI/CAPI payloads, database schema, config keys, or cscli -o json|raw output.
• A human has reviewed this diff line by line.
• A human has tested this change, not only an agent.

AI assistance used: mostly

/kind enhancement
/area local-api
/area configuration

[codecov]

Codecov Report

❌ Patch coverage is 97.56098% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 71.34%. Comparing base (192480d) to head (ead48f3).

Files with missing lines	Patch %	Lines
pkg/apiserver/controllers/v1/decisions.go	95.83%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4747      +/-   ##
==========================================
+ Coverage   71.31%   71.34%   +0.03%     
==========================================
  Files         542      542              
  Lines       36798    36838      +40     
==========================================
+ Hits        26241    26281      +40     
  Misses      10556    10556              
  Partials        1        1              

Flag	Coverage Δ
bats	45.51% <36.58%> (-0.02%)	⬇️
unit-linux	47.27% <97.56%> (+0.06%)	⬆️
unit-windows	35.61% <92.68%> (+0.13%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[blotus]

LGTM overall, I'd just like to have a new config option to control how long a request can be blocked waiting.
It should default to something reasonable like 10s.

[blotus]

We should a timer here (whose duration is controlled by a api.server.decisions_stream.max_queue_wait config option) that will return a 503 if the wait duration has been exceeded. This would allow for bouncer (and crowdsec) to log a warning if a request is blocked waiting.

[tommy-ng0]

Added api.server.decisions_stream.max_queue_wait, default 10s. That's under lua-cs-bouncer's 15s STREAM_REQUEST_TIMEOUT, so it gets the 503 before giving up on its own. Past the wait LAPI returns a 503 and logs a warning with the bouncer name. 0 means the default, negative warns and uses the default, same as page_size.

While testing I found that when a slot is free and the timer has already fired, select picks between them at random, so a tiny max_queue_wait could 503 a request even with a slot free. Fixed by trying for a slot before starting the timer, with a test for it.