dataframe_parser
s that rename columns conflict with type coercion
#1666
Labels
bug
Something isn't working
I'm not sure if this qualifies as a bug, or is just the expected behavior, but I thought I would report just in case this behavior was unintended. Using the new
dataframe_parser
decorator introduced recently, I tried making a parser that maps some of the column names of the data I am ingesting to new names. On this same schema model, I also enabled schema-wide type coercion by settingcoerce = True
in theConfig
class. Basically, it seems like due to the order of operations of core parsers and custom parsers, it isn't possible to perform coercion with this setup since core parsers operate before custom parsers and the columns defined in the schema are not present in the input DataFrame before the custom parsers operate, which ultimately leads to aSchemaError
when attempting to validate data.It seems like this could potentially be solved by allowing users a greater degree of control over the order of custom parsers and core parsers. Again, I'm not sure if this is the expected behavior, and I can also appreciate that renaming columns in this manner may not be an intended use case for this functionality. If that's the case or there's some other workaround, please feel free to close this issue.
Code Sample, a copy-pastable example
Expected behavior
I would expect
FailingSchema
to validate successfully, because I have asserted thatcol
should be coerced into anint
.Desktop (please complete the following information):
The text was updated successfully, but these errors were encountered: