GCP c7n-org with resource gcp.project and iam-policy filter runs against all projects in org when single project is specified

[fdeswardt]

Describe the bug

When running the following policy with c7n-org for GCP

policies:
  - name: gcp-project-test
    resource: gcp.project

Using a config file that only has a single project in it, that returns a result that have all the projects of the GCP Organzition.

What did you expect to happen?

Only the information about the project for the single project should have been returned, not all the projects in the GCP Organization.

Cloud Provider

Google Cloud (GCP)

Cloud Custodian version and dependency information

Custodian:   0.9.14
Python:      3.8.10 (default, Nov 26 2021, 20:14:08)
             [GCC 9.3.0]
Platform:    posix.uname_result(sysname='Linux', nodename='asr-custodian-main', release='5.11.0-1028-gcp', version='#32~20.04.1-Ubuntu SMP Wed Jan 12 20:08:27 UTC 2022', machine='x86_64')
Using venv:  True
Docker: False
Installed:

argcomplete==1.12.3
attrs==21.2.0
boto3==1.19.12
botocore==1.22.12
cachetools==4.2.4
certifi==2021.10.8
charset-normalizer==2.0.7
docutils==0.17.1
google-api-core==2.2.2
google-api-python-client==2.29.0
google-auth==2.3.3
google-auth-httplib2==0.1.0
google-cloud-appengine-logging==1.1.0
google-cloud-audit-log==0.2.0
google-cloud-core==2.1.0
google-cloud-logging==2.7.0
google-cloud-monitoring==2.6.0
google-cloud-storage==1.42.3
google-crc32c==1.3.0
google-resumable-media==2.1.0
googleapis-common-protos==1.53.0
grpc-google-iam-v1==0.12.3
grpcio==1.41.1
httplib2==0.20.2
idna==3.3
importlib-metadata==4.8.1
jmespath==0.10.0
jsonschema==3.2.0
packaging==21.2
proto-plus==1.19.7
protobuf==3.19.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
pyrsistent==0.18.0
python-dateutil==2.8.2
pytz==2021.3
pyyaml==5.4.1
ratelimiter==1.2.0.post0
requests==2.26.0
retrying==1.3.3
rsa==4.7.2
s3transfer==0.5.0
setuptools==44.0.0
six==1.16.0
tabulate==0.8.9
typing-extensions==3.10.0.2
uritemplate==4.1.1
urllib3==1.26.7
zipp==3.6.0

Policy

policies:
  - name: gcp-project-test
    resource: gcp.project

Relevant log/traceback output

No response

Extra information or context

Need to get the IAM policies of each Project using this policy

policies:
  - name: gcp-iam-policy-value
    resource: gcp.project
    filters:
      - type: iam-policy
        doc:
          key: "bindings[*].members[]"
          value: ["allUsers", "allAuthenticatedUsers"]

Since the gcp.project resource returns all projects, the above policy times out due to the insane number of GCP projects in the organization.

This is blocking us from doing any IAM policies on GCP Projects and a critical must have capability.

[kapilt]

The purpose of the gcp project resource is to enumerate projects in the organization.

Its unclear what you mean by "Using a config file that only has a single project in it".. what config file? the policy example you have have, should return all projects in the organization.

you can supply additional server side / api filters in a policy using query attribute of a policy and the 'filter' parameter from gcp.

[fdeswardt]

The config file is the one used by c7n-org in the following command:

c7n-org run -c ./single-project.yaml -s outputs/single-project_test_iam-policy/ -u ./test_iam-policy.yaml

where the contents of the config file single-project.yaml is:

projects:
  - name: my-test-project
    project_id: my-test-project
    tags:
    - label:l1:test

and the contents of the policy file test_iam-policy.yaml is:

policies:
  - name: gcp-iam-policy-value
    resource: gcp.project
    filters:
      - type: iam-policy
        doc:
          key: "bindings[*].members[]"
          value: ["allUsers", "allAuthenticatedUsers"]

Since the intended design for the gcp.project resource is to return all projects in the organization, then which resource should one use to get the iam-policy at the project level?

If this is the correct resource, how will one use the query attribute to scope the policy down to only run the policy at the project scope when running with c7n-org?

The goal is to get the same output from the following gcloud command (docs here):

gcloud projects get-iam-policy my-test-project

[fdeswardt]

Hey @kapilt
So I tested with query attribute having a folder ID as parent.id that only have 8 projects under different levels of folder nesting using the following policy:

policies:
  - name: gcp-iam-policy-owner-editor
    resource: gcp.project
    query:
      - filter: "parent.id:954902212336 parent.type:folder"
    filters:
      - type: iam-policy
        doc:
          key: "bindings[?contains(['roles/owner', 'roles/editor'], role)].members[]"
          value: not-null

and ran it with the following command:

c7n-org run -c ./projects.yaml -s outputs/gcp-iam-policy_owner-editor/ -u policies/gcp-iam-policy-owner-editor.yaml --cache-period 0

Where projects.yaml have all the projects of the organization.

The result is that it again runs the iam-policy filter against all the projects in the projects.yaml, instead of just the projects under the folder ID.

I can see the following block in metadata.json for one of the projects that is not under this parent folder as follows:

  "policy": {
    "name": "gcp-iam-policy-owner-editor",
    "resource": "gcp.project",
    "query": [
      {
        "filter": "parent.id:954902212336 parent.type:folder"
      }
    ],
    "filters": [
      {
        "type": "iam-policy",
        "doc": {
          "key": "bindings[?contains(['roles/owner', 'roles/editor'], role)].members[]",
          "value": "not-null"
        }
      }
    ]
  }

with as expected empty resources.json.

So how does one then run a policy for the gcp.project resource against a subset of projects with c7n-org?

I have tested with the latest version of Custodian having version 0.9.15