Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Image cast storage faster #6786

Open
wants to merge 11 commits into
base: main
Choose a base branch
from

Conversation

Modexus
Copy link
Contributor

@Modexus Modexus commented Apr 5, 2024

PR for issue #6782.
Makes cast_storage of the Image class faster by removing the slow call to .pylist.
Instead directly convert each ListArray item to either Array2DExtensionType or Array3DExtensionType.

This also preserves the dtype removing the warning if the array is already uint8.

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq
Copy link
Member

lhoestq commented Jun 26, 2024

Hi ! Thanks for diving into this, this conversion to python lists is indeed quite slow.

Array2DExtensionType and Array3DExtensionType currently rely on pyarrow lists, but we will soon modify them to use FixedShapeTensorArray instead which is more efficient (e.g. doesn't need to store an offset for each value). So ideally it would be cool to speed this code up without using those extension types or it will be blocking to improve Array2DExtensionType and Array3DExtensionType.

If I understand correctly you just need the logic from ArrayExtensionArray.to_numpy ? If so feel free to make a separate function and ArrayExtensionArray.to_numpy can call it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants