How to validate policies within a directory?

[princeanire]

I have a lot of policies contained in a folder called checks I'd want to validate them to check for errors and bugs using custodian validate or any commands,

It is tedious to try validating those one-by one

Is there any command that lets me validate policies just by specifying the directory path?

[benipeled]

Not sure about custodian option but you can loop it with
for file in $(ls /path/to/folder); do custodian validate /path/to/folder/$file; done

[ajkerrigan]

Yeah @benipeled 's answer is a good one. You can also use something like xargs in helpful ways. To validate all YAML files in a given policy directory for instance:

find /path/to/my/policies -iname "*.y*ml" | xargs custodian validate

Or maybe if you're in a git repository of policies and want to validate only policies that differ from what's on main:

git diff --name-only main.. | xargs custodian validate

That said, thanks for asking this question! I occasionally still use this simpler command:

custodian validate /path/to/my/policies/*.y*ml

Which works fine if all of your policies are valid, but it can report errors multiple times in a way that is misleading. But that feels like a bug we should fix, so... #8565

[princeanire]

Thank you for giving me an idea on using xargs with custodian validate!

What worked for me is
cd /path/to/my/policies
ls | xargs custodian validate

currently I'm using custodian validate on a GitLab CI pipeline to check policies made by my teammates and ensure that they are valid

I tried the command both on windows 11 and Ubuntu WSL:
find /path/to/my/policies -iname "*.y*ml" | xargs custodian validate
it works on Ubuntu WSL
.
on Windows 11 it raises:
raise ValueError("Invalid path for config %r" % config_file)

ValueError: Invalid path for config '.\checks\.yml'

ValueError: Invalid path for config 'C:\Users\Prince Andrew Anire\source\repos\cloud-pitstop\checks\.yml'

I may be missing something or doing something wrong on Windows 11

and you are right, custodian raises multiple errors on a same policy, and it won't stop till it tries to validate all the checks,
although once the error(s) happens all configurations after the first error becomes invalid,

logs
2023-05-16 06:52:25,957: custodian.commands:INFO Configuration valid: ./cloudtrail-check-status.yaml
2023-05-16 06:52:25,990: custodian.commands:ERROR Configuration invalid: ./codebuild-not-exist.yaml
2023-05-16 06:52:25,990: custodian.commands:ERROR Only one policy with a given name allowed, duplicates: cloudformation-not-exist
2023-05-16 06:52:26,009: custodian.commands:ERROR Configuration invalid: ./codecommit-has-recent.yaml
2023-05-16 06:52:26,009: custodian.commands:ERROR Only one policy with a given name allowed, duplicates: cloudformation-not-exist
....

[ajkerrigan]

That makes sense - I was leaning on Linuxish conventions and ignoring Windows there 🙈 . A similar PowerShell command might be something like:

ls -r *.y*ml | % { custodian validate  }

# or its more verbose equivalent...
Get-ChildItem -Recurse *.y*ml | ForEach-Object { custodian validate }

And if/when #8565 lands, it would allow the somewhat more concise:

custodian validate $(ls -r *.y*ml)