Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AVRO-3869 [chsarp] Fix logical fixed schema support #2512

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

paulohumberto42
Copy link

What is the purpose of the change

Logical types works only for primitive types, and I can't deserialize an fixed type, like this one:

{
 "type": "fixed",
 "name": "LogicalFixed",
 "logicalType": "decimal",
 "precision": 4,
 "scale": 2,
  "size": 10
}
 

A SchemaParseException is raised ("Undefined name: fixed at 'type'")
This schema is autogenerated by PySpark, from a decimal type, and can't be parsed.
Spark docs: https://spark.apache.org/docs/latest/sql-data-sources-avro.html

Verifying this change

This change added tests and can be verified as follows:

[TestCase("{ \"type\": \"fixed\", \"name\": \"LogicalFixed\", \"logicalType\": \"decimal\", \"precision\": 4, \"scale\": 2, \"size\": 10 }", "decimal", 10)]
public void TestLogicalFixed(string s, string logicalType, int size)
{
    Schema sc = Schema.Parse(s);
    Assert.AreEqual(Schema.Type.Logical, sc.Tag);
    LogicalSchema logsc = sc as LogicalSchema;
    FixedSchema basesc = logsc.BaseSchema as FixedSchema;
    Assert.AreEqual(Schema.Type.Fixed, basesc.Tag);
    Assert.AreEqual(size, basesc.Size);
    Assert.AreEqual(logicalType, logsc.LogicalType.Name);

    testEquality(s, sc);
    testToString(sc);
}

@github-actions github-actions bot added the C# label Sep 21, 2023
@KalleOlaviNiemitalo
Copy link

This pull request changes only parsing, not Schema.ToString(), whose result might still not be interoperable with other libraries. I suppose ToString can be considered out of scope for AVRO-3869 and fixed later in AVRO-3570.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants