[Bug]: Unable to generate pydantic-core schema for <class 'prowler.lib.check.compliance_models.CIS_Requirement_Attribute_Profile'>.

[oliviermtl]

Steps to Reproduce

./prowler.py --output-formats {csv,json-asff,json-ocsf,html} -c cis_2.0_aws -w arn:aws:dynamodb:ap-southeast-1:xxxxxxxxxxx:table/prowler_allow_list

Expected behavior

run correctly

Actual Result with Screenshots or Logs

[Container] 2024/05/31 07:03:52.594491 Running command ./prowler.py --output-formats {csv,json-asff,json-ocsf,html} -c cis_2.0_aws -w arn:aws:dynamodb:ap-southeast-1:xxxxxxxxxxx:table/prowler_allow_list
Traceback (most recent call last):
File "/codebuild/output/src513582655/src/prowler/./prowler.py", line 5, in
from prowler.main import prowler
File "/codebuild/output/src513582655/src/prowler/prowler/main.py", line 10, in
from prowler.lib.check.check import (
File "/codebuild/output/src513582655/src/prowler/prowler/lib/check/check.py", line 18, in
from prowler.lib.check.compliance_models import load_compliance_framework
File "/codebuild/output/src513582655/src/prowler/prowler/lib/check/compliance_models.py", line 80, in
class CIS_Requirement_Attribute(BaseModel):
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 202, in new
complete_model_class(
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_model_construction.py", line 539, in complete_model_class
schema = cls.get_pydantic_core_schema(cls, handler)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/main.py", line 626, in get_pydantic_core_schema
return handler(source)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in call
schema = self._handler(source_type)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 502, in generate_schema
schema = self._generate_schema_inner(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 753, in _generate_schema_inner
return self._model_schema(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 580, in _model_schema
{k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 580, in
{k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 916, in _generate_md_field_schema
common_field = self._common_field_schema(name, field_info, decorators)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 1081, in _common_field_schema
schema = self._apply_annotations(
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 1820, in _apply_annotations
schema = get_inner_schema(source_type)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_schema_generation_shared.py", line 82, in call
schema = self._handler(source_type)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 1801, in inner_handler
schema = self._generate_schema_inner(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 758, in _generate_schema_inner
return self.match_type(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 844, in match_type
return self._unknown_type_schema(obj)
File "/root/.pyenv/versions/3.9.16/lib/python3.9/site-packages/pydantic/_internal/_generate_schema.py", line 405, in _unknown_type_schema
raise PydanticSchemaGenerationError(
pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'prowler.lib.check.compliance_models.CIS_Requirement_Attribute_Profile'>. Set arbitrary_types_allowed=True in the model_config to ignore this error or implement __get_pydantic_core_schema__ on your type to fully support it.

If you got this error by calling handler() within __get_pydantic_core_schema__ then you likely need to call handler.generate_schema(<some type>) since we do not call __get_pydantic_core_schema__ on <some type> otherwise to avoid infinite recursion.

How did you install Prowler?

Cloning the repository from github.com (git clone)

Environment Resource

Cloudformation template

OS used

Amazon Linux 2

Prowler version

4.2.1

Pip version

21.1.2

Context

I am using the same cloudformation template that I was using with prowler v2, except for python 3.6->3.9

[jfagoagas]

Hi @oliviermtl could you please share the dependencies you have installed in your environment?

[sergargar]

Hi @oliviermtl, it could also be because the compliance flag is --compliance and not -c. Let us know if that is the issue, thanks for using Prowler! 🚀

[oliviermtl]

Hi @oliviermtl, it could also be because the compliance flag is --compliance and not -c. Let us know if that is the issue, thanks for using Prowler! 🚀

it works fine locally with -c

[oliviermtl]

Hi @oliviermtl could you please share the dependencies you have installed in your environment?

Successfully installed Flask-3.0.3 Jinja2-3.1.4 MarkupSafe-2.1.5 PyJWT-2.8.0 Werkzeug-3.0.3 XlsxWriter-3.2.0 about-time-4.2.1 aiohttp-3.9.5 aiosignal-1.3.1 alive-progress-3.1.5 anyio-4.4.0 attrs-23.2.0 awsipranges-0.3.3 azure-common-1.1.28 azure-core-1.30.1 azure-identity-1.16.0 azure-keyvault-keys-4.9.0 azure-mgmt-applicationinsights-4.0.0 azure-mgmt-authorization-4.0.0 azure-mgmt-compute-31.0.0 azure-mgmt-containerservice-30.0.0 azure-mgmt-core-1.4.0 azure-mgmt-cosmosdb-9.5.0 azure-mgmt-keyvault-10.3.0 azure-mgmt-monitor-6.0.2 azure-mgmt-network-25.4.0 azure-mgmt-rdbms-10.1.0 azure-mgmt-resource-23.1.1 azure-mgmt-security-7.0.0 azure-mgmt-sql-3.0.1 azure-mgmt-storage-21.1.0 azure-mgmt-subscription-3.1.1 azure-mgmt-web-7.2.0 azure-storage-blob-12.20.0 blinker-1.8.2 boto3-1.34.109 botocore-1.34.113 cachetools-5.3.3 cffi-1.16.0 charset-normalizer-3.3.2 click-8.1.7 click-plugins-1.1.1 colorama-0.4.6 cryptography-42.0.7 dash-2.17.0 dash-bootstrap-components-1.6.0 dash-core-components-2.0.0 dash-html-components-2.0.0 dash-table-5.0.0 deprecated-1.2.14 detect-secrets-1.5.0 dnspython-2.6.1 email-validator-2.1.1 frozenlist-1.4.1 google-api-core-2.19.0 google-api-python-client-2.130.0 google-auth-2.29.0 google-auth-httplib2-0.2.0 googleapis-common-protos-1.63.0 grapheme-0.6.0 h11-0.14.0 h2-4.1.0 hpack-4.0.0 httpcore-1.0.5 httplib2-0.22.0 httpx-0.27.0 hyperframe-6.0.1 idna-3.7 importlib-metadata-7.1.0 isodate-0.6.1 itsdangerous-2.2.0 jsonschema-4.22.0 jsonschema-specifications-2023.12.1 kubernetes-29.0.0 microsoft-kiota-abstractions-1.3.3 microsoft-kiota-authentication-azure-1.0.0 microsoft-kiota-http-1.3.1 microsoft-kiota-serialization-form-0.1.0 microsoft-kiota-serialization-json-1.2.0 microsoft-kiota-serialization-multipart-0.1.0 microsoft-kiota-serialization-text-1.0.0 msal-1.28.0 msal-extensions-1.1.0 msgraph-core-1.0.0 msgraph-sdk-1.4.0 msrest-0.7.1 multidict-6.0.5 nest-asyncio-1.6.0 numpy-1.26.4 oauthlib-3.2.2 opentelemetry-api-1.25.0 opentelemetry-sdk-1.25.0 opentelemetry-semantic-conventions-0.46b0 packaging-24.0 pandas-2.2.2 pendulum-3.0.0 plotly-5.22.0 portalocker-2.8.2 proto-plus-1.23.0 protobuf-4.25.3 prowler-4.2.1 py-ocsf-models-0.1.1 pyasn1-0.6.0 pyasn1-modules-0.4.0 pycparser-2.22 pydantic-1.10.15 pyparsing-3.1.2 pytz-2024.1 referencing-0.35.1 requests-2.32.3 requests-file-2.1.0 requests-oauthlib-2.0.0 retrying-1.3.4 rpds-py-0.18.1 rsa-4.9 schema-0.7.7 shodan-1.31.0 slack-sdk-3.27.2 sniffio-1.3.1 std-uritemplate-0.0.57 tabulate-0.9.0 tenacity-8.3.0 time-machine-2.14.1 tldextract-5.1.2 typing-extensions-4.12.0 tzdata-2024.1 tzlocal-5.2 uritemplate-4.1.1 websocket-client-1.8.0 wrapt-1.16.0 yarl-1.9.4 zipp-3.19.0

[oliviermtl]

Solved by using python 3.11 on Amazon-Linux-2023.

This is the BuildSpec from my cloudformation template :

BuildSpec: |
          version: 0.2
          phases:
            install:
              runtime-versions:
                python: 3.11
              commands:
                - echo "Installing Prowler and dependencies..."
                - pip3 install detect-secrets alive_progress colorama jsonschema tabulate pydantic==1.10.15 azure-identity azure.mgmt==4.0.0 schema slack_sdk pytz tzlocal msgraph-core==1.0.0 msgraph-sdk==1.4.0 py_ocsf_models
                - pip3 install --upgrade google-api-python-client
                - yum -y install jq
                - curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
                - unzip awscliv2.zip
                - ./aws/install --update
                - git clone --depth 1 --branch 4.2.1 https://github.com/prowler-cloud/prowler.git

[jfagoagas]

That's great news @oliviermtl 👏

Where did you grab the cloudformation templace? Is it something built by you or present in this repository? If it is the latter it'd be great to include that code into the repository. We can do that or help you to do that.

Thanks!

[oliviermtl]

The template was already here when I took over the position so I cannot confirm where it comes from.
Happy to push a PR with the template.

[jfagoagas]

Awesome! You can include it under contrib/aws. Thanks!

[oliviermtl]

~/Code/prowler cloudformation-template >  git push --set-upstream origin cloudformation-template
remote: Permission to prowler-cloud/prowler.git denied to oliviermtl.
fatal: unable to access 'https://github.com/prowler-cloud/prowler/': The requested URL returned error: 403

What am I missing here ? I git clone, followed all the dev guide, created a branch

[jfagoagas]

You cannot contribute directly to the repository, you have to create a fork, push the code to a branch in your forked repository and then open the pull request from your forked branch to Prowler master.

[oliviermtl]

ok, you might to want to include this information at the top of the developer guide.

[jfagoagas]

ok, you might to want to include this information at the top of the developer guide.

Sure, good catch!

[jfagoagas]

Hi @oliviermtl, we've just included that in our documentation, thanks!

#4238

[jfagoagas]

Hi @oliviermtl are you finally planning to open up a PR with the CodeBuild template? Thanks!

[jfagoagas]

Hi @oliviermtl I'm closing this issue. Please open the PR you mention once is ready, we are looking forward to have that!

Thanks for using Prowler 🚀