tag policies on ec2 on runinstances events w/ trail lambda execution

[mickmorseMB]

Hi

We have Cloud Custodian running compliance on our tagging. EC2 resources part of EMR clusters and ASG get terminated before tags are even allocated to it. I've tried putting this filter in on our policy: "State.Name": Running but the tags aren't even being allocated at pending state.

Was seeing if there was a way to add the system.status as a filter. Basically meaning if the instance is fully provisioned (all checks passed) but still without the appropriate tags, then terminate.

[kapilt]

You'll have to share a sample policy to clarify which execution mode your using, and which event your triggering on, and which region.

in general for ec2 policies its best to ignore emr and asg instances and handle those instances via policies on emr and asg, which is typically done via by ignoring them on their 'aws:' prefiex tags. if your getting to the instance before those exist, thats suggest needing to change the event that your triggering on.. there's multiple events available that come to mind, ec2-instance-state exec mode with state running, config-rule on ec2, and doing poll mode. note ec2 now supports tags as part of the run instances call but not all sw that uses ec2 has switched over to tag on create.

[mickmorseMB]

this is my policy:

    resource: ec2
    description: "EC2 resources created without proper tagging will be terminated"
    mode:
      type: cloudtrail
      role: *lambda_role_arn
      events:
        - RunInstances
    filters:
      - "State.Name" Running
      - or: *tag-compliance-filters
    actions:
      - type: terminate
        force: true
      - type: notify
        template: default.html
        priority_header: 1
        subject: 'AWS Tagging Compliance - Resource Termination (EC2)'
        violation_desc: *violation_desc
        action_desc: "The resource has been terminated."
        to: *email_to
        transport: *notify_to

tag compliance filter is a check to see if certain tags are absent.

The tags don't get allocated until later in the Instance State: Running part but by then its too late?

what do you mean by the ec2-instance-state exec mode

P.S - we also have policies for ASG....

The common theme here is that instances are being spun up with ASG and/or EMR clustering and terminated before they can even get tags allocated to them (even the aws: prefix tags)

[kapilt]

So fundamentally doing trail based policies for tag compliance, where there are multiple api calls needed to properly tag a resource is inherently racy. If you want that to be reliable you should restructure is to not be racy.

For doing this with a lambda policy the right option is probably as a config-rule execution mode, config will automatically delay and batch the resource creation to account for this multi-api call creation before invoking the rule. albeit that behavior is subject to change based on aws config's service evaluation semantics as they try to decrease latency and preserve consistency.

The other option (which we've used historically) is for resources that typically are provisioned over multiple api calls is to use a periodic policy that evaluates resources older then n, younger than y against tag compliance every z minutes. the younger then condition is to track against net new.

[mickmorseMB]

Brilliant. Do you have an example of the periodic policy you are referring to?