Backport/tentacle 20.2.4 cobaltcore storage/keystone iam policy

.github/workflows/test-ceph-custom.yml

@@ -4,12 +4,12 @@ on:
   workflow_dispatch:
     inputs:
       base_branch:
-        description: 'Base branch to test (e.g. squid-19.2.4-cobaltcore-storage)'
+        description: 'Base branch to test (e.g. tentacle-20.2.4-cobaltcore-storage)'
         required: true
         type: string
   pull_request:
     branches:
-      - 'squid-*-cobaltcore-storage'
+      - 'tentacle-*-cobaltcore-storage'
 
 jobs:
   test-ceph:
@@ -49,6 +49,13 @@ jobs:
           git submodule foreach --recursive git clean -fdx
           git submodule update --init --recursive
 
+      # A job that dies mid-build (e.g. lost runner) leaves its container
+      # behind, and build-with-container.py always names it ceph_build.
+      - name: Remove leftover build container
+        run: |
+          podman ps -a --filter name=ceph_build
+          podman rm -f ceph_build
+
       # --- 2. COMPUTE PARALLELISM ---
       - name: Compute build + test parallelism
         run: |

PendingReleaseNotes

@@ -879,6 +879,13 @@ CephFS: Disallow delegating preallocated inode ranges to clients. Config
 * RGW: The serialized format of notification and topics has changed, so that 
   new/updated topics will be unreadable by old RGWs. We recommend completing 
   the RGW upgrades before creating or modifying any notification topics.
+* RGW: New rate limiting options have been added to control the number of
+  bucket listing requests (`radosgw-admin ratelimit set ... --max-list-ops=`)
+  and delete operations (`radosgw-admin ratelimit set ... --max-delete-ops=`)
+  per accumulation interval, along with the ability to configure the
+  accumulation interval duration (`rgw_ratelimit_interval`).
+  Their default value is `0`, which keeps backward compatibility to the
+  previous behavior of READ and WRITE operations rate limiting.
 * RBD: Trailing newline in passphrase files (`<passphrase-file>` argument in
   `rbd encryption format` command and `--encryption-passphrase-file` option
   in other commands) is no longer stripped.

doc/radosgw/admin.rst

@@ -629,9 +629,21 @@ Rate Limit Management
 =====================
 
 Quotas can be set for The Ceph Object Gateway on users and buckets. The "rate
-limit" includes the maximum number of read operations (read ops) and write
-operations (write ops) per minute as well as the number of bytes per minute
-that can be written or read per user or per bucket.
+limit" includes the maximum number of read operations and write operations
+per accumulation interval as well as the number of bytes per accumulation interval
+that can be written or read per user or per bucket. It also includes the maximum
+number of list requests and delete operations per accumulation interval.
+The accumulation interval is configured by the :confval:`rgw_ratelimit_interval` option.
+The default value is 60 seconds.
+(Note: S3 Multi-Object Delete operation are currently not supported by rate limiting)
+
+The configured limits should be divided by the number of active object gateways. For example,
+if "user A" is to be be limited to 10 ops per minute and there are two object gateways in the cluster,
+then the limit on "user A" should be 5 (10 ops per minute / 2 RGWs).
+If the requests are not balanced between RGWs, the rate limit might be underutilized.
+For example: if the ops limit is 5 and there are two RGWs,
+but the Load Balancer sends load to only one of those RGWs,
+the effective limit is 5 ops, because this limit is enforced per RGW.
 
 Read Requests and Write Requests
 --------------------------------
@@ -643,8 +655,8 @@ How Metrics Work
 Each object gateway tracks per-user metrics separately from bucket metrics.
 These metrics are not shared with other gateways. The configured limits should
 be divided by the number of active object gateways. For example, if "user A" is
-to be be limited to 10 ops per minute and there are two object gateways in the
-cluster, then the limit on "user A" should be ``5`` (10 ops per minute / 2
+to be be limited to 10 ops per accumulation interval and there are two object gateways in the
+cluster, then the limit on "user A" should be ``5`` (10 ops per accumulation interval / 2
 RGWs). If the requests are **not** balanced between RGWs, the rate limit might
 be underutilized. For example: if the ops limit is ``5`` and there are two
 RGWs, **but** the Load Balancer sends load to only one of those RGWs, the
@@ -661,12 +673,18 @@ rate limit is reached during the execution of the request. The RGW keeps track
 of a "debt" consisting of bytes used in excess of the configured value; users
 or buckets that incur this kind of debt are prevented from sending more
 requests until the "debt" has been repaid. The maximum size of the "debt" is
-twice the max-read/write-bytes per minute. If "user A" is subject to a 1-byte
-read limit per minute and they attempt to ``GET`` an object that is 1 GB in size,
+twice the max-read/write-bytes per accumulation interval. If "user A" is subject to a 1-byte
+read limit per accumulation interval and they attempt to ``GET`` an object that is 1 GB in size,
 then the ``GET`` action will fail. After "user A" has completed this 1 GB
-operation, RGW blocks the user's requests for up to two minutes. After this
+operation, RGW blocks the user's requests for up to two accumulation intervals. After this
 time has elapsed, "user A" will be able to send ``GET`` requests again.
 
+Rate-limited requests receive an HTTP ``503`` response with the S3 error code
+``SlowDown``, matching the AWS S3 behavior. The response includes a
+``Retry-After`` header with the estimated number of seconds until the rate-limit
+window allows the next request through, computed from the current token-bucket
+state. Load balancers or reverse proxies can be configured to map the ``503``
+with ``Retry-After`` to ``429 Too Many Requests`` if clients require it.
 
 - **Bucket:** The ``--bucket`` option allows you to specify a rate limit for a
   bucket.
@@ -675,19 +693,27 @@ time has elapsed, "user A" will be able to send ``GET`` requests again.
   user.
 
 - **Maximum Read Ops:** The ``--max-read-ops`` setting allows you to limit read
-  bytes per minute per RGW instance. A ``0`` value disables throttling. 
+  bytes per accumulation interval per RGW instance. A ``0`` value disables throttling.
 
 - **Maximum Read Bytes:** The ``--max-read-bytes`` setting allows you to limit
-  read bytes per minute per RGW instance. A ``0`` value disables throttling. 
+  read bytes per accumulation interval per RGW instance. A ``0`` value disables throttling.
 
 - **Maximum Write Ops:** The ``--max-write-ops`` setting allows you to specify
-  the maximum number of write ops per minute per RGW instance. A ``0`` value
+  the maximum number of write ops per accumulation interval per RGW instance. A ``0`` value
   disables throttling.
 
 - **Maximum Write Bytes:** The ``--max-write-bytes`` setting allows you to
-  specify the maximum number of write bytes per minute per RGW instance. A
+  specify the maximum number of write bytes per accumulation interval per RGW instance. A
   ``0`` value disables throttling.
-
+
+- **Maximum List Ops:** The ``--max-list-ops`` setting allows you to
+  specify the maximum number of bucket listing requests per accumulation interval per RGW instance.
+  A ``0`` value disables throttling.
+
+- **Maximum Delete Ops:** The ``--max-delete-ops`` setting allows you to
+  specify the maximum number of delete operations per accumulation interval per RGW instance.
+  A ``0`` value disables throttling.
+
 - **Rate Limit Scope:** The ``--ratelimit-scope`` option sets the scope for the
   rate limit.  The options are ``bucket`` , ``user`` and ``anonymous``. Bucket
   rate limit apply to buckets.  The user rate limit applies to a user.  The
@@ -706,7 +732,8 @@ parameters:
 
    radosgw-admin ratelimit set --ratelimit-scope=user --uid=<uid> \
                                  <[--max-read-ops=<num ops>] [--max-read-bytes=<num bytes>] \
-                                 [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>]>
+                                 [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>] \
+                                 [--max-list-ops=<num ops>] [--max-delete-ops=<num ops>]>
 
 An example of using ``radosgw-admin ratelimit set`` to set a rate limit might
 look like this: 
@@ -771,7 +798,8 @@ The following is the general form of commands that set rate limit parameters:
 
    radosgw-admin ratelimit set --ratelimit-scope=bucket --bucket=<bucket> \
                                  <[--max-read-ops=<num ops>] [--max-read-bytes=<num bytes>] \
-                                 [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>]>
+                                 [--max-write-ops=<num ops>] [--max-write-bytes=<num bytes>] \
+                                 [--max-list-ops=<num ops>] [--max-delete-ops=<num ops>]>
 
 An example of using ``radosgw-admin ratelimit set`` to set a rate limit for a
 bucket might look like this: 

doc/radosgw/adminops.rst

@@ -2788,8 +2788,9 @@ permission. ::
 Rate Limit
 ==========
 
-The Admin Operations API enables you to set and get ratelimit configurations on users and on bucket and global rate limit configurations. See `Rate Limit Management`_ for additional details. 
-Rate Limit includes the maximum number of operations and/or bytes per minute, separated by read and/or write, to a bucket and/or by a user and the maximum storage size in megabytes.
+The Admin Operations API enables you to set and get ratelimit configurations on users and on bucket and global rate limit configurations. See `Rate Limit Management`_ for additional details.
+Rate Limit includes the maximum number of operations and/or bytes per accumulation interval, separated by read and/or write (Additionally list and get operations),
+to a bucket and/or by a user and the maximum storage size in megabytes.
 
 To view rate limit, the user must have a ``ratelimit=read`` capability. To set,
 modify or disable a ratelimit, the user must have ``ratelimit=write`` capability.
@@ -2803,16 +2804,22 @@ Valid parameters for quotas include:
 - **User:** The ``uid`` option allows you to specify a rate limit for a user.
 
 - **Maximum Read Bytes:** The ``max-read-bytes`` setting allows you to specify
-  the maximum number of read bytes per minute. A 0 value disables this setting.
+  the maximum number of read bytes per accumulation interval. A 0 value disables this setting.
 
 - **Maximum Write Bytes:** The ``max-write-bytes`` setting allows you to specify
-  the maximum number of write bytes per minute. A 0 value disables this setting.
+  the maximum number of write bytes per accumulation interval. A 0 value disables this setting.
 
 - **Maximum Read Ops:** The ``max-read-ops`` setting allows you to specify
-  the maximum number of read ops per minute. A 0 value disables this setting.
+  the maximum number of read ops per accumulation interval. A 0 value disables this setting.
 
 - **Maximum Write Ops:** The ``max-write-ops`` setting allows you to specify
-  the maximum number of write ops per minute. A 0 value disables this setting.
+  the maximum number of write ops per accumulation interval. A 0 value disables this setting.
+
+- **Maximum List Ops:** The ``max-list-ops`` setting allows you to specify
+  the maximum number of bucket listing requests per accumulation interval. A 0 value disables this setting.
+
+- **Maximum Delete Ops:** The ``max-delete-ops`` setting allows you to specify
+  the maximum number of delete operations per accumulation interval. A 0 value disables throttling.
 
 - **Global:** The ``global`` option allows you to specify a global rate limit.
   The value should be either 'True' or 'False'.
@@ -2839,7 +2846,7 @@ Set User Rate Limit
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?ratelimit-scope=user&uid=<uid><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][enabled=<True|False>]>
+	POST /{admin}/ratelimit?ratelimit-scope=user&uid=<uid><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 
@@ -2859,7 +2866,7 @@ Set Rate Limit for an Individual Bucket
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?bucket=<bucket-name>&ratelimit-scope=bucket<[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>]>
+	POST /{admin}/ratelimit?bucket=<bucket-name>&ratelimit-scope=bucket<[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 
@@ -2879,7 +2886,7 @@ Set Global User Rate Limit
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?ratelimit-scope=user&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][enabled=<True|False>]>
+	POST /{admin}/ratelimit?ratelimit-scope=user&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 
@@ -2889,7 +2896,7 @@ Set Global Rate Limit Bucket
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?ratelimit-scope=bucket&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>]>
+	POST /{admin}/ratelimit?ratelimit-scope=bucket&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 
@@ -2899,7 +2906,7 @@ Set Global Anonymous User Rate Limit
 To set a rate limit, the user must have ``ratelimit`` capability set with ``write``
 permission. ::
 
-	POST /{admin}/ratelimit?ratelimit-scope=anon&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][enabled=<True|False>]>
+	POST /{admin}/ratelimit?ratelimit-scope=anon&global=<True|False><[&max-read-bytes=<bytes>][&max-write-bytes=<bytes>][&max-read-ops=<ops>][&max-write-ops=<ops>][&max-list-ops=<ops>][&max-delete-ops=<ops>][&enabled=<True|False>]>
 
 
 

doc/radosgw/bucketpolicy.rst

@@ -122,6 +122,11 @@ For all requests, condition keys we support are:
 - aws:UserAgent
 - aws:username
 
+Request that authenticate with Keystone also include:
+
+- keystone:role
+- keystone:userid
+
 We support certain s3 condition keys for bucket and object requests.
 
 *Support for the following bucket-related operations was added in the Mimic

doc/radosgw/keystone.rst

@@ -159,6 +159,19 @@ S3 API (with AWS-like access and secret keys), if the ``rgw s3 auth
 use keystone`` option is set. For details, see
 :doc:`s3/authentication`.
 
+Requests authenticated via Keystone (Swift tokens or S3 with Keystone-managed
+credentials) expose the Keystone idenitity in the IAM policy environment
+as the condition keys: ``keystone:role`` (role names) and
+``keystone:userid`` (user UUID). These conditions can be used in bucket
+policies and idenitity policies with ``StringEquals``, ``StringNotEquals`` etc.
+
+- **keystone:role** - Allow or deny by *role* (e.g only users with role
+  ``reader`` get read access). Use for RBAC.
+- **keystone:userid** - Restrict to specific *user*. Use when policy
+  depends on a specific user, not just their role.
+
+See :doc:`bucketpolicy` for list of supported condition keys.
+
 Service token support
 ---------------------
 

qa/suites/rados/upgrade/cephx/X/ms_bind/.qa

@@ -0,0 +1 @@
+../.qa/

qa/suites/rados/upgrade/cephx/X/tasks/07-client-auth-keys/.qa

@@ -0,0 +1 @@
+../.qa/

qa/suites/rados/upgrade/cephx/X/tasks/08-allowed-ciphers/.qa

@@ -0,0 +1 @@
+../.qa/

qa/suites/rados/upgrade/cephx/release-X/ms_bind/.qa

@@ -0,0 +1 @@
+../.qa/

qa/suites/rados/upgrade/cephx/release-X/tasks/07-client-auth-keys/.qa

@@ -0,0 +1 @@
+../.qa/

qa/suites/rados/upgrade/cephx/release-X/tasks/08-allowed-ciphers/.qa

@@ -0,0 +1 @@
+../.qa/

qa/tasks/ceph.py

@@ -1725,8 +1725,6 @@ def key_prune(ctx, config):
     elif isinstance(config, list):
         config = {'keys': config}
 
-    testdir = teuthology.get_testdir(ctx)
-
     cluster_name = config.setdefault('cluster', 'ceph')
     manager = ctx.managers[cluster_name]
 
@@ -1789,7 +1787,6 @@ def key_rotate(ctx, config):
     ]
 
     daemons = ctx.daemons.resolve_role_list(config.get('daemons', None), CEPH_ROLE_TYPES, True)
-    clusters = set()
 
     new_mon_key = None
     for role in daemons:
@@ -1831,7 +1828,6 @@ def key_rotate(ctx, config):
         daemon.restart()
 
     clients = config.get('clients', [])
-    firstmon = teuthology.get_first_mon(ctx, config, cluster_name)
     keyring = ctx.ceph[cluster_name].keyring
     for client in clients:
         if client == 'all':
@@ -2272,7 +2268,6 @@ def task(ctx, config):
 
             # stop logging health to clog during shutdown, or else we generate
             # a bunch of scary messages unrelated to our actual run.
-            firstmon = teuthology.get_first_mon(ctx, config, cluster_name)
             ctx.ceph[cluster_name].admin.run(
                 args=[
                     'sudo',

qa/tasks/radosbench.py

@@ -110,7 +110,7 @@ def task(ctx, config):
             extra_args.append(f'--auth_exit_on_failure={auth_exit_on_failure}')
 
         bench_args = [
-            f'bench',
+            'bench',
             f'--concurrent-ios={concurrency}',
         ]
 
@@ -163,7 +163,7 @@ def task(ctx, config):
         log.info('joining radosbench (timing out after %ss)', timeout)
         try:
             run.wait(radosbench.values(), timeout=timeout)
-        except CommandFailedError as e:
+        except CommandFailedError:
             for p in radosbench.values():
                 if p.exitstatus == expected_rc:
                     pass