Inconsistent behaviour when creating tag and business metadata bindings on schemas

[kaat-nnit]

When testing adding tag and business metadata bindings to schemas with Confluent Terraform provider version 1.66, we encountered strange errors.

In the following simple example, a single schema was created. In a subsequent run, 1 tag and 1 business metadata binding should be added to the schema, as correctly output by plan:

  # module.kafka-resources.confluent_business_metadata_binding.business_metadata_binding["test_app_pipeline_2:pilot_application-pipeline-example-avro1"] will be created 
  + resource "confluent_business_metadata_binding" "business_metadata_binding" {
      ...
    }

  # module.kafka-resources.confluent_tag_binding.tag_binding["iv_sensitive:pilot_application-pipeline-example-avro1"] will be created 
  + resource "confluent_tag_binding" "tag_binding" {
    ...
    }

Apply step produces the following, unencountered before errors:

Error: error creating Tag Binding 403 Forbidden: User is denied operation POST on resource 
catalog/v1/entity/tags/ 
│ │ with module.kafka-resources.confluent_tag_binding.tag_binding["iv_sensitive:pilot_application-pipeline-example-avro1"], 
│ on module/confluent.tf line 200, in resource "confluent_tag_binding" "tag_binding": 
│ 200: resource "confluent_tag_binding" "tag_binding" {

│ Error: error reading Business Metadata Binding "lsrc-9zwq70/test_app_pipeline_2/lsrc-9zwq70:.:100298/sr_schema": 
404 Not Found: Instance sr_schema with unique attribute {qualifiedName=lsrc-9zwq70:.:100298} does not exist 
│ │ with module.kafka-resources.confluent_business_metadata_binding.business_metadata_binding["test_app_pipeline_2:pilot_application-pipeline-example-avro1"], 
│ on module/confluent.tf line 217, in resource "confluent_business_metadata_binding" "business_metadata_binding": 
│ 217: resource "confluent_business_metadata_binding" "business_metadata_binding" {

At this point, business metadata resource with tainted status is in the state file, and the metadata is correctly showing in the UI.

Subsequent run with no changes to configuration:

1. Plan to replace the tainted business metadata resource and add tag binding resource.
2. Failure with errors:

• 403 Forbidden: User is denied operation DELETE for business metadata binding resource
• 404 Not Found: Instance sr_schema with unique attribute {qualifiedName=lsrc-9zwq70:.:100298} does not exist for tag binding resource

3. Both resource block present in the state file with tainted status, both resources created in the UI.

The schema identifier cited as non existent in the errors was the correct id. We have tested the setup before and have not encountered this issue.
Additionally, running Terraform in the debug mode produced this warning:

[WARN] Provider "registry.terraform.io/confluentinc/confluent" produced an invalid plan for module.kafka-resources.confluent_schema.this["pilot_application-pipeline-example-avro1"], 
but we are tolerating it because it is using the legacy plugin SDK. The following problems may be the cause of any confusing errors from downstream operations: 
- .hard_delete: planned value cty.False for a non-computed attribute 
- .recreate_on_update: planned value cty.False for a non-computed attribute

Any clues as to what could be causing this?
Is there any way deprecating confluent_schema_registry_region data source and confluent_schema_registry_cluster could have impact?

[linouk23]

@kaat-nnit thanks for creating the issue!

Error: error creating Tag Binding 403 Forbidden: User is denied operation POST on resource
catalog/v1/entity/tags/

It might imply SR API Key's owner lacks permissions 🤔 Could you confirm what RBAC roles it has?

404 Not Found: Instance sr_schema with unique attribute {qualifiedName=lsrc-9zwq70:.:100298} does not exist for tag binding resource

404 should result in an automated resource removal from TF state file, that seems like a bug.

[kaat-nnit]

@linouk23 Thank you for the quick reply.
It's ResourceOwner role for the schema registry. And in a setup with multiple schemas and bindings, most of them got created correctly, except for 2 last ones throwing the exact same errors as in the issue.
So it's strange that for the same SR some bindings are applied correctly, and some throw this error.

[S1M0NM]

Yesterday I also tried to create tags and tag bindings for topics.

Unfortunately, I noticed that some of the tags I created disappear in the Confluent Cloud UI and reappear after 5-10 minutes, and then some disappear again. The situation is similar with Terraform, where the plan states that the resource has been deleted outside of Terraform.

As a result, tag bindings cannot be created at all or are inconsistent. The data portal and topics have the bindings one moment and apparently no longer have them the next.

However, if you then try to create them via the UI, you get an error message saying that the tag or binding already exists.