Apply fails silently when applying old version of existing schema

[ondrejbilcik]

Hi,

We're running into an issue when reverting changes to a schema, when terraform apply succeeds however the schema does not revert to the previous version in the registry.
We can work around this using an arbitrary version attribute, however the terraform apply should still throw an error in cases when the attribute isn't changed and thus the change isn't applied to the schema.

When TF_LOG=warn is set, the following warning is displayed when applying:

2024-06-17T17:18:19.977+0200 [WARN] Provider "provider["registry.terraform.io/confluentinc/confluent"]" produced an unexpected new value for module.schemas.confluent_schema.this["event.initiative.InitiativeUpdated"], but we are tolerating it because it is using the legacy plugin SDK.

[ConfluentSpencer]

@ondrejbilcik can you please share us some example configs and the expected outcomes? This will help us troubleshoot. CC @channingdong

[ondrejbilcik]

Sure. I set up some test configs to illustrate the issue:

1. I created the following schema:

{
  "name": "Test",
  "type": "record",
  "namespace": "test.one",
  "doc": "This is a test",
  "fields": [
    {
      "name": "test1",
      "type": "string",
      "doc": "Random string to test the schema"
    }
  ]
}

2. I updated the schema with the following:

{
	"type": "record",
	"name": "Test",
	"namespace": "test.one",
	"doc": "This is a test",
	"fields": [
		{
			"name": "test1",
			"type": "string",
			"doc": "Random string to test the schema"
		},
		{
			"name": "test2",
			"type": "string",
			"doc": "Random second string to test the schema",
			"default": null
		}
	]
}

3. I'd like to revert the change from step two, the apply succeeds

Terraform will perform the following actions:

  # module.schemas.confluent_schema.this["ondrej.test"] will be updated in-place
  ~ resource "confluent_schema" "this" {
        id                 = "1/ondrej.test/latest"
      ~ schema             = jsonencode(
          ~ {
              ~ fields    = [
                    {
                        doc  = "Random string to test the schema"
                        name = "test1"
                        type = "string"
                    },
                  - {
                      - default = null
                      - doc     = "Random second string to test the schema"
                      - name    = "test2"
                      - type    = "string"
                    },
                ]
                name      = "Test"
                # (3 unchanged attributes hidden)
            }
        )
        # (6 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.schemas.confluent_schema.this["ondrej.test"]: Modifying... [id=1/ondrej.test/latest]
module.schemas.confluent_schema.this["ondrej.test"]: Still modifying... [id=1/ondrej.test/latest, 10s elapsed]
module.schemas.confluent_schema.this["ondrej.test"]: Modifications complete after 10s [id=1/ondrej.test/latest]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

As you can see below, the schema is still on version two. I would expect either the schema to revert to the previous version, or for terraform to throw an error stating that the change wasn't successfully applied.

Let me know if you need any more data.

Cheers,
Ondrej