[SPARK-48709][SQL] Fix varchar type resolution mismatch for DataSourceV2 CTAS #47082
Conversation
|
cc @cloud-fan |
| @@ -1739,6 +1739,16 @@ class DataSourceV2SQLSuiteV1Filter | |||
| } | |||
| } | |||
|
|
|||
| test("SPARK-48709: varchar resolution mismatch for DataSourceV2 CTAS") { | |||
| withSQLConf( | |||
| SQLConf.STORE_ASSIGNMENT_POLICY.key -> SQLConf.StoreAssignmentPolicy.LEGACY.toString) { | |||
There was a problem hiding this comment.
just for my education, why it's only a problem with the legacy store assignment?
There was a problem hiding this comment.
Maybe we can test all options here
There was a problem hiding this comment.
We will validate store assignment policy if !v2Write.outputResolved is true.
Similar to outType, we should get RawType to make output resolved.
There was a problem hiding this comment.
How does this end up with the error org.apache.spark.sql.AnalysisException: LEGACY store assignment policy is disallowed in Spark data source V2.?
There was a problem hiding this comment.
I mean, it looks like the right error.
There was a problem hiding this comment.
oh nvm, when we don't need to do store assignment, then the mode doesn't matter.
|
Merged to master, Thank you @wangyum @cloud-fan Could you send a backport PR to branch-3.5? @wangyum |
…SourceV2 CTAS Backport of #47082. ### What changes were proposed in this pull request? This PR fixes varchar type resolution mismatch for DataSourceV2 CTAS. For example: ```sql set spark.sql.storeAssignmentPolicy=LEGACY; CREATE TABLE testcat.ns.t1 (d1 string, d2 varchar(200)) USING parquet; CREATE TABLE testcat.ns.t2 USING foo as select * from testcat.ns.t1 ``` Error message: ``` org.apache.spark.sql.AnalysisException: LEGACY store assignment policy is disallowed in Spark data source V2. Please set the configuration spark.sql.storeAssignmentPolicy to other values. ``` ### Why are the changes needed? Avoid query failures. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit test. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47103 from wangyum/SPARK-48709-branch-3.5. Authored-by: Yuming Wang <[email protected]> Signed-off-by: Kent Yao <[email protected]>
What changes were proposed in this pull request?
This PR fixes varchar type resolution mismatch for DataSourceV2 CTAS. For example:
Error message:
Why are the changes needed?
Avoid query failures.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Unit test.
Was this patch authored or co-authored using generative AI tooling?
No.