Repository navigation
ateapi: add --experimental-openfga-cache-ttl - #3009
Shruti Nair (SHRUTI6991) wants to merge 2 commits into
Conversation
Enable OpenFGA's Check query cache and iterator cache, each with a 1s TTL, when --experimental-openfga-cache is set. Policy writes don't invalidate them, so a grant or revoke can take up to 2s to apply.
| postgresSchema = pflag.String("postgres-schema", "substrate", "PostgreSQL schema for Substrate tables. This overrides a search_path connection parameter.") | ||
| postgresPoolMaxConns = pflag.Int32("postgres-pool-max-conns", 0, "Maximum connections in the shared Substrate and OpenFGA read/write PostgreSQL pool. Does not affect the owner, watch, or readiness probe pools. The DSN or pgx default is used when unset.") | ||
| experimentalEnableAuthz = pflag.Bool("experimental-enable-authz", false, "Enforce OpenFGA authorization checks on all registered RPCs (experimental). AccessPolicy RPCs are always checked.") | ||
| experimentalOpenFGACache = pflag.Bool("experimental-openfga-cache", false, "Cache OpenFGA Check results and tuple reads in memory for 1s each (experimental). A grant or revoke can take up to 2s to apply.") |
There was a problem hiding this comment.
Should we expose a knob for setting the TTL?
There was a problem hiding this comment.
+1. A TTL would let you control both the amount of staleness you are willing to tolerate and whether the feature is enabled or not (TTL=0).
There was a problem hiding this comment.
Done. Replaced the bool with --experimental-openfga-cache-ttl. 0 disables the cache.
haiyanmeng
added
kind/feature
| // also miss writes: changelog rows are stamped with the transaction | ||
| // start time, so a long transaction can commit a row older than one | ||
| // the controller has already seen. | ||
| opts = append(opts, |
There was a problem hiding this comment.
Is there any otel metric to monitor the cache? (hit ratio, size, etc).
There was a problem hiding this comment.
Not as OTel metrics. OpenFGA registers Prometheus counters for both caches: openfga_check_cache_hit_count and openfga_check_cache_total_count for the query cache, openfga_tuples_cache_hit_count and openfga_tuples_cache_total_count for the iterator cache, and openfga_cache_item_count for size. ate-api-server serves them on :9090/metrics, but doesn't push them over OTLP.
I have an overall issue for metric tracking here: #2990.
Replace --experimental-openfga-cache with --experimental-openfga-cache-ttl, where 0 disables the cache. Each cached sub-Check gets a fresh TTL, so staleness grows by up to one TTL per level of the relation chain. Describe the bound as up to about five times the TTL, and add a test for the cache.
Shruti Nair (SHRUTI6991)
changed the title
3 participants
Every OpenFGA Check reads PostgreSQL. atenet-router calls ResumeActor on every request it routes, so a busy actor gets the same
can_resumeCheck many times a second. This adds an experimental flag to cache Checks in ate-api-server so we can measure the effect under suspend/resume load.--experimental-openfga-cache-ttlturns on OpenFGA's Check query cache and iterator cache with the given TTL. The default, 0, disables both.Policy writes don't invalidate either cache. Each cached sub-Check gets a fresh TTL, so a grant or revoke can take up to one TTL per level of the relation chain to apply. The longest chains in
model.fgahave five levels, so the bound is up to about five times the TTL.docs/authorization.mdlists this under known limitations.OpenFGA's cache counters (
openfga_check_cache_*,openfga_tuples_cache_*,openfga_cache_item_*) are served on ate-api-server's/metrics, but not pushed over OTLP. Bridging them is a follow-up.Benchmark numbers will follow.
AI use: AI-assisted (Jetski). I reviewed the diff.
TestNewOpenFGAServer_CheckCacheneeds Docker; not run locally yet)