We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
I'm receiving errors for projects not listed in project_ids regarding API not being enabled. Logs show the error below.
project_ids
The logs show 2 different project id.
redacted__project_id_configured
redacted__project_id_not_configured
the sql_instances would be loaded to the sync as expected with resources found in configured project_ids
kind: source spec: name: "gcp_sql" path: "cloudquery/gcp" registry: "cloudquery" version: "v14.1.0" tables: ["gcp_sql_instances"] destinations: ["postgresql"] skip_dependent_tables: true # GCP Spec # Learn more about the configuration options at https://cql.ink/gcp_source spec: backoff_retries: 1 enabled_services_only: true folder_recursion_depth: 0 project_ids: ["project_id_redacted"]
from local, run this
cloudquery --log-console sync --log-level trace ./gcp/gcp_sql.yml ./postgresql.yml --log-format json
{ "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "args": [ "./gcp/gcp_sql.yml", "./postgresql.yml" ], "time": "2024-06-17T18:56:38Z", "message": "Loading spec(s)" } { "level": "debug", "module": "cli", "invocation-id": "redacted__invocation_id", "ops": { "AuthToken": "redacted__cloudquery_auth", "TeamName": "redacted_team_name", "LocalPath": ".cq/plugins/source/cloudquery/gcp/v14.1.0/plugin", "PluginTeam": "cloudquery", "PluginKind": "source", "PluginName": "gcp", "PluginVersion": "v14.1.0" }, "time": "2024-06-17T18:56:38Z", "message": "Requesting GET https://api.cloudquery.io/plugins/cloudquery/source/gcp/versions/v14.1.0" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "address": "/var/folders/x5/bggkpsvn09lf8_w6p714p12r0000gq/T/cq-ngcVpMKsVxxhvstf.sock", "message": "Plugin server listening", "time": "2024-06-17T18:56:39Z" } { "level": "debug", "module": "cli", "invocation-id": "redacted__invocation_id", "ops": { "AuthToken": "redacted__cloudquery_auth", "TeamName": "redacted_team_name", "LocalPath": ".cq/plugins/destination/cloudquery/postgresql/v8.1.1/plugin", "PluginTeam": "cloudquery", "PluginKind": "destination", "PluginName": "postgresql", "PluginVersion": "v8.1.1" }, "time": "2024-06-17T18:56:40Z", "message": "Requesting GET https://api.cloudquery.io/plugins/cloudquery/destination/postgresql/versions/v8.1.1" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "address": "/var/folders/x5/bggkpsvn09lf8_w6p714p12r0000gq/T/cq-HHVGnkvdcDaNcMKj.sock", "message": "Plugin server listening", "time": "2024-06-17T18:56:40Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.component": "server", "grpc.method": "GetVersions", "grpc.method_type": "unary", "grpc.service": "cloudquery.discovery.v1.Discovery", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "0.142", "message": "started call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.code": "OK", "grpc.component": "server", "grpc.method": "GetVersions", "grpc.method_type": "unary", "grpc.service": "cloudquery.discovery.v1.Discovery", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "0.185", "message": "finished call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.component": "server", "grpc.method": "GetVersions", "grpc.method_type": "unary", "grpc.service": "cloudquery.discovery.v1.Discovery", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "0.159", "message": "started call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.code": "OK", "grpc.component": "server", "grpc.method": "GetVersions", "grpc.method_type": "unary", "grpc.service": "cloudquery.discovery.v1.Discovery", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "0.306", "message": "finished call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "source": "gcp_sql (cloudquery/[email protected])", "destinations": [ "postgresql (cloudquery/[email protected])" ], "sync_time": "2024-06-17T18:56:41Z", "time": "2024-06-17T18:56:41Z", "message": "Start sync" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.component": "server", "grpc.method": "Init", "grpc.method_type": "unary", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "0.007", "message": "started call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "invocation_id": "redacted__invocation_id", "message": "Initializing postgresql destination", "module": "pg-dest", "pgx_log_level": "invalid level 0", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.code": "OK", "grpc.component": "server", "grpc.method": "Init", "grpc.method_type": "unary", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "113.472", "message": "finished call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.component": "server", "grpc.method": "Init", "grpc.method_type": "unary", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "0.006", "message": "started call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "invocation_id": "redacted__invocation_id", "message": "No organization_ids or organization_filter specified - assuming all organizations", "module": "gcp-source", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "invocation_id": "redacted__invocation_id", "message": "Listing organizations...", "module": "gcp-source", "time": "2024-06-17T18:56:41Z" } { "level": "debug", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.component": "client", "grpc.method": "SearchOrganizations", "grpc.method_type": "unary", "grpc.request.deadline": "2024-06-17T13:57:41-05:00", "grpc.service": "google.cloud.resourcemanager.v3.Organizations", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "0.008", "invocation_id": "redacted__invocation_id", "message": "started call", "module": "gcp-source", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.code": "PermissionDenied", "grpc.component": "client", "grpc.error": "rpc error: code = PermissionDenied desc = Cloud Resource Manager API has not been used in project redacted__project_id_not_configured before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=redacted__project_id_not_configured then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.", "grpc.method": "SearchOrganizations", "grpc.method_type": "unary", "grpc.request.deadline": "2024-06-17T13:57:41-05:00", "grpc.service": "google.cloud.resourcemanager.v3.Organizations", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "489.543", "invocation_id": "redacted__invocation_id", "message": "finished call", "module": "gcp-source", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "error", "module": "cli", "invocation-id": "redacted__invocation_id", "error": "rpc error: code = PermissionDenied desc = Cloud Resource Manager API has not been used in project redacted__project_id_not_configured before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=redacted__project_id_not_configured then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.\nerror details: name = ErrorInfo reason = SERVICE_DISABLED domain = googleapis.com metadata = map[consumer:projects/redacted__project_id_not_configured service:cloudresourcemanager.googleapis.com]\nerror details: name = Help desc = Google developers console API activation url = https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=redacted__project_id_not_configured", "invocation_id": "redacted__invocation_id", "message": "failed to get organizations", "module": "gcp-source", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "invocation_id": "redacted__invocation_id", "message": "Found 1 projects in folders", "module": "gcp-source", "projects": [ "redacted__project_id_configured" ], "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "invocation_id": "redacted__invocation_id", "message": "Found 0 organizations in folders", "module": "gcp-source", "organizations": [], "time": "2024-06-17T18:56:41Z" } { "level": "error", "module": "cli", "invocation-id": "redacted__invocation_id", "error": "rpc error: code = PermissionDenied desc = Cloud Resource Manager API has not been used in project redacted__project_id_not_configured before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=redacted__project_id_not_configured then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.\nerror details: name = ErrorInfo reason = SERVICE_DISABLED domain = googleapis.com metadata = map[consumer:projects/redacted__project_id_not_configured service:cloudresourcemanager.googleapis.com]\nerror details: name = Help desc = Google developers console API activation url = https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=redacted__project_id_not_configured", "invocation_id": "redacted__invocation_id", "message": "failed to get organizations", "module": "gcp-source", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "invocation_id": "redacted__invocation_id", "message": "Retrieved organizations", "module": "gcp-source", "orgs": null, "time": "2024-06-17T18:56:41Z" } { "level": "debug", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.component": "client", "grpc.method": "ListServices", "grpc.method_type": "unary", "grpc.request.deadline": "2024-06-17T13:57:41-05:00", "grpc.service": "google.api.serviceusage.v1.ServiceUsage", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "0.014", "invocation_id": "redacted__invocation_id", "message": "started call", "module": "gcp-source", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:41Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.code": "PermissionDenied", "grpc.component": "client", "grpc.error": "rpc error: code = PermissionDenied desc = Service Usage API has not been used in project redacted__project_id_not_configured before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=redacted__project_id_not_configured then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.", "grpc.method": "ListServices", "grpc.method_type": "unary", "grpc.request.deadline": "2024-06-17T13:57:41-05:00", "grpc.service": "google.api.serviceusage.v1.ServiceUsage", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "384.25", "invocation_id": "redacted__invocation_id", "message": "finished call", "module": "gcp-source", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:42Z" } { "level": "warn", "module": "cli", "invocation-id": "redacted__invocation_id", "error": "rpc error: code = PermissionDenied desc = Service Usage API has not been used in project redacted__project_id_not_configured before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=redacted__project_id_not_configured then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.\nerror details: name = ErrorInfo reason = SERVICE_DISABLED domain = googleapis.com metadata = map[consumer:projects/redacted__project_id_not_configured service:serviceusage.googleapis.com]\nerror details: name = Help desc = Google developers console API activation url = https://console.developers.google.com/apis/api/serviceusage.googleapis.com/overview?project=redacted__project_id_not_configured", "invocation_id": "redacted__invocation_id", "message": "failed to list enabled services because of insufficient permissions. Sync will continue without filtering out disabled services for this project: redacted__project_id_configured", "module": "gcp-source", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.code": "OK", "grpc.component": "server", "grpc.method": "Init", "grpc.method_type": "unary", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:41-05:00", "grpc.time_ms": "909.136", "message": "finished call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "source": "gcp_sql (cloudquery/[email protected])", "destinations": [ "postgresql (cloudquery/[email protected])" ], "time": "2024-06-17T18:56:42Z", "message": "Start fetching resources" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.component": "server", "grpc.method": "Sync", "grpc.method_type": "server_stream", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:42-05:00", "grpc.time_ms": "0.075", "message": "started call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "client": "project:redacted__project_id_configured", "invocation_id": "redacted__invocation_id", "message": "top level table resolver started", "module": "gcp-source", "table": "gcp_sql_instances", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.component": "server", "grpc.method": "Write", "grpc.method_type": "client_stream", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:42-05:00", "grpc.time_ms": "7.022", "message": "started call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:42Z" } { "level": "error", "module": "cli", "invocation-id": "redacted__invocation_id", "client": "project:redacted__project_id_configured", "error": "googleapi: Error 403: Cloud SQL Admin API has not been used in project redacted__project_id_not_configured before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=redacted__project_id_not_configured then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.\nDetails:\n[\n {\n \"@type\": \"type.googleapis.com/google.rpc.Help\",\n \"links\": [\n {\n \"description\": \"Google developers console API activation\",\n \"url\": \"https://console.developers.google.com/apis/api/sqladmin.googleapis.com/overview?project=redacted__project_id_not_configured\"\n }\n ]\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"consumer\": \"projects/redacted__project_id_not_configured\",\n \"service\": \"sqladmin.googleapis.com\"\n },\n \"reason\": \"SERVICE_DISABLED\"\n }\n]\n, accessNotConfigured", "invocation_id": "redacted__invocation_id", "message": "table resolver finished with error", "module": "gcp-source", "table": "gcp_sql_instances", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "client": "project:redacted__project_id_configured", "errors": 1, "invocation_id": "redacted__invocation_id", "message": "table sync finished", "module": "gcp-source", "resources": 0, "table": "gcp_sql_instances", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.code": "OK", "grpc.component": "server", "grpc.method": "Sync", "grpc.method_type": "server_stream", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:42-05:00", "grpc.time_ms": "415.855", "message": "finished call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "summary": { "cli_version": "5.20.5", "destination_errors": 0, "destination_name": "postgresql", "destination_path": "cloudquery/postgresql", "destination_version": "v8.1.1", "destination_warnings": 0, "resources": 0, "source_errors": 3, "source_name": "gcp_sql", "source_path": "cloudquery/gcp", "source_version": "v14.1.0", "source_warnings": 1, "sync_id": "redacted__invocation_id", "sync_time": "2024-06-17T18:56:41.233878Z" }, "time": "2024-06-17T18:56:42Z", "message": "Sync summary" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "invocation_id": "redacted__invocation_id", "message": "Migrating table", "module": "pg-dest", "table": "gcp_sql_instances", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "invocation_id": "redacted__invocation_id", "message": "Table exists, auto-migrating", "module": "pg-dest", "table": "gcp_sql_instances", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.code": "OK", "grpc.component": "server", "grpc.method": "Write", "grpc.method_type": "client_stream", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:42-05:00", "grpc.time_ms": "513.963", "message": "finished call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.component": "server", "grpc.method": "Close", "grpc.method_type": "unary", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:42-05:00", "grpc.time_ms": "0.006", "message": "started call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "grpc.code": "OK", "grpc.component": "server", "grpc.method": "Close", "grpc.method_type": "unary", "grpc.service": "cloudquery.plugin.v3.Plugin", "grpc.start_time": "2024-06-17T13:56:42-05:00", "grpc.time_ms": "0.523", "message": "finished call", "peer.address": "", "protocol": "grpc", "time": "2024-06-17T18:56:42Z" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "resources": 0, "errors": 3, "warnings": 1, "duration": "1s", "result": "Sync completed with errors, see logs for details", "time": "2024-06-17T18:56:42Z", "message": "Sync summary" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "source": "gcp_sql (cloudquery/[email protected])", "destinations": [ "postgresql (cloudquery/[email protected])" ], "sync_time": "2024-06-17T18:56:41Z", "time": "2024-06-17T18:56:42Z", "message": "End sync" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "time": "2024-06-17T18:56:42Z", "message": "Sending sync summary to analyticsv1.cloudquery.io:443" } { "level": "debug", "module": "cli", "invocation-id": "redacted__invocation_id", "time": "2024-06-17T18:56:42Z", "message": "sending interrupt signal to source plugin" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "time": "2024-06-17T18:56:42Z", "message": "waiting for source plugin to terminate" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "address": "/var/folders/x5/bggkpsvn09lf8_w6p714p12r0000gq/T/cq-HHVGnkvdcDaNcMKj.sock", "message": "Got stop signal. Plugin server shutting down", "signal": "interrupt", "time": "2024-06-17T18:56:42Z" } { "level": "debug", "module": "cli", "invocation-id": "redacted__invocation_id", "time": "2024-06-17T18:56:42Z", "message": "sending interrupt signal to source plugin" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "time": "2024-06-17T18:56:42Z", "message": "waiting for source plugin to terminate" } { "level": "info", "module": "cli", "invocation-id": "redacted__invocation_id", "address": "/var/folders/x5/bggkpsvn09lf8_w6p714p12r0000gq/T/cq-ngcVpMKsVxxhvstf.sock", "message": "Got stop signal. Plugin server shutting down", "signal": "interrupt", "time": "2024-06-17T18:56:42Z" }
5.20.5
No response
The content you are editing has changed. Please copy your edits and refresh the page.
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Is there an existing issue for this?
Current Behavior
I'm receiving errors for projects not listed in
project_ids
regarding API not being enabled. Logs show the error below.The logs show 2 different project id.
redacted__project_id_configured
exists in yaml configredacted__project_id_not_configured
is not referencedExpected Behavior
the sql_instances would be loaded to the sync as expected with resources found in configured
project_ids
CloudQuery (redacted) config
Steps To Reproduce
from local, run this
CloudQuery (redacted) logs
CloudQuery version
5.20.5
Additional Context
No response
Pull request (optional)
Tasks
Tasks
The text was updated successfully, but these errors were encountered: