Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for HEIF based images #2633

Open
wants to merge 96 commits into
base: main
Choose a base branch
from
Open

Conversation

ynse01
Copy link
Contributor

@ynse01 ynse01 commented Dec 29, 2023

Prerequisites

  • I have written a descriptive pull-request title
  • I have verified that there are no overlapping pull-requests open
  • I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
  • I have provided test coverage for my change (where applicable)

Description

This PR is an attempt to start with #1320

Implemented decoding and encoding of images based on a HEIF (ISO/IEC 23008-12:2022) container. These include (amongst others): HEIC, HIF and AVIF.

Please note this PR does NOT add any new compression algorithm to ImageSharp. For now it will take a JPEG thumbnail as its pixel source.
Needless to say, much more work is needed after this PR to reach the goal. I'm try to help there also.

Please do let me know any comments, also on whether to support the HEIF file format in the first place.

@brianpopow
Copy link
Collaborator

@ynse01 The Av1BitStreamReader seems to work as it should, I was able to parse Av1CodecConfiguration with it d8e7078. Maybe I can help with this PR, but I still in the process of reading the spec, there is a lot to grasp.

@ynse01
Copy link
Contributor Author

ynse01 commented Jun 24, 2024

@ynse01 The Av1BitStreamReader seems to work as it should, I was able to parse Av1CodecConfiguration with it d8e7078. Maybe I can help with this PR, but I still in the process of reading the spec, there is a lot to grasp.

There sure is a lot of grasp for AVIF.

The parsing of ObuSequenceHeader seems to work OK indeed, but somewhere in either ObuFrameHeader or in the handling of TileGroups there is a bug. The way AVIF (or technically Av1) intertwines BitStream coding with Symbol coding is pretty hard to debug. Merged in my latest changes even though some tests are still failing, as I did some minor refactoring.

@ynse01
Copy link
Contributor Author

ynse01 commented Jun 25, 2024

I fixed the bug :-) Tests are passing again.

@brianpopow There are a few TODO items in the Heif container area, if you like you could have a look at:

  • Item locations can be relative to multiple origins, this is incompletely implemented in the HeifLocation class.
  • Sorting of item locations, such that we can read the stream sequentially.
  • There is no connection between HeifDecoderCore and Av1Decoder classes. Ideally, this could be designed such that LegacyJpeg (and potentially Jpeg-XR) can be implemented similarly, thinking of an interface here. Look in 'ParseMediaData', which is currently hardcoded into LegacyJPEG.
  • We need code to split the incoming pixels into their respective Y, U and V channels, as Av1 uses the YUV pixel format exclusively.

I'll dive deeper into the Symbol coding and parsing the coefficients of the TileInfoBlocks.

@brianpopow
Copy link
Collaborator

I fixed the bug :-) Tests are passing again.

@brianpopow There are a few TODO items in the Heif container area, if you like you could have a look at:
...
I'll dive deeper into the Symbol coding and parsing the coefficients of the TileInfoBlocks.

Good job in finding the bug! I had some time on the weekend too look into your code. It's impressive howmuch is already done with the OBU reader. I thought maybe I could start with something small, maybe implementing parsing 5.9.14. Segmentation params as a start?

@ynse01
Copy link
Contributor Author

ynse01 commented Jun 25, 2024

I fixed the bug :-) Tests are passing again.
@brianpopow There are a few TODO items in the Heif container area, if you like you could have a look at:
...
I'll dive deeper into the Symbol coding and parsing the coefficients of the TileInfoBlocks.

Good job in finding the bug! I had some time on the weekend too look into your code. It's impressive howmuch is already done with the OBU reader. I thought maybe I could start with something small, maybe implementing parsing 5.9.14. Segmentation params as a start?

Great !
I see the implement-another-spec-section-virus has gotten to you also. If you like more of these, feel free to do so.
As a goal, we could support images that do not use the reduced still picture header feature and explicitly contain every OBU module out there. The test image Irvine_CA.avif is such an example, in ObuFrameHeaderTest you can uncomment it and work your way through the the OBU headers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants