terraform test: Additional erroneous error message when referencing invalid attributes

[liamcervante]

Terraform Version

v1.8.3

Terraform Configuration Files

# main.tf

variable "number" {
    type = number
}

data "external" "resource" {
    count = var.number

    program = [ "bash", "${path.module}/out.sh" ]
}

output "out" {
    value = data.external.resource
}

#!/bin/bash
# out.sh

echo "{\"hello\":\"world\"}"

# main.tftest.hcl

run "test" {
    variables {
        number = 1
    }

    assert {
        condition = data.external.resource[0].hello == "world"
        error_message = "wrong"
    }
}

Debug Output

N/A

Expected Behavior

main.tftest.hcl... in progress
  run "test"... fail
╷
│ Error: Unsupported attribute
│ 
│   on main.tftest.hcl line 8, in run "test":
│    8:         condition = data.external.resource[0].hello == "world"
│ 
│ This object has no argument, nested block, or exported attribute named "hello".
╵
main.tftest.hcl... tearing down
main.tftest.hcl... fail

Failure! 0 passed, 1 failed.

Actual Behavior

main.tftest.hcl... in progress
  run "test"... fail
╷
│ Error: Unknown variable
│ 
│   on main.tftest.hcl line 8, in run "test":
│    8:         condition = data.external.resource[0].hello == "world"
│ 
│ There is no variable named "data".
╵
╷
│ Error: Unsupported attribute
│ 
│   on main.tftest.hcl line 8, in run "test":
│    8:         condition = data.external.resource[0].hello == "world"
│ 
│ This object has no argument, nested block, or exported attribute named "hello".
╵
main.tftest.hcl... tearing down
main.tftest.hcl... fail

Failure! 0 passed, 1 failed.

Steps to Reproduce

1. terraform init
2. terraform test

Additional Context

No response

References

https://discuss.hashicorp.com/t/tf-test-1-8-cannot-test-custom-data-source-output/67461

[liamcervante]

We are constructing a HCL evaluation context for the test command here. In this case, the actual construction of the context does some validation of its own which results in it returning the expected diagnostic along with an empty context. The error then comes as we are not exiting the evaluation early when the context isn't computed properly. Currently, we are just adding the diagnostics and then attempting to evaluate the condition anyway. This results in the confusing message we get as the context does not contain any variables.

The fix would be to skip the evaluation of the rule if we cannot build the context properly:

		hclCtx, moreDiags := scope.EvalContext(refs)
		ruleDiags = ruleDiags.Append(moreDiags)
                if moreDiags.HasErrors() {
                    // If we couldn't create the context properly then we can't
                    // evaluate the condition.
                    continue
                }

[gogingersnap777]

Making changes for this issue! I was able to reproduced the behavior locally and working on the changes.