Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Chocolatey is using checksum.exe which is slow and can't cope with larger than 2GB files #23

Open
6 tasks done
flcdrg opened this issue Jun 10, 2024 · 2 comments · May be fixed by chocolatey/choco#3463
Open
6 tasks done

Chocolatey is using checksum.exe which is slow and can't cope with larger than 2GB files #23

flcdrg opened this issue Jun 10, 2024 · 2 comments · May be fixed by chocolatey/choco#3463
Labels
Improvement Issues that enhances existing functionality, or adds new features

Comments

@flcdrg
Copy link
Member

flcdrg commented Jun 10, 2024

Checklist

  • I confirm there are no unresolved issues reported on the Chocolatey Status page.
  • I have verified this is the correct repository for opening this issue.
  • I have verified no other issues exist related to my problem.
  • I have verified this is not an issue for a specific package.
  • I have verified this issue is not security related.
  • I confirm I am using official, and not unofficial, or modified, Chocolatey products.

What You Are Seeing?

Package fails to validate or install because download is larger than 2GB

What is Expected?

Chocolatey should be able to cope with large files for calculating checksum/filehash

How Did You Get This To Happen?

Tried to install package with downloaded file larger than 2GB eg- https://community.chocolatey.org/packages/resharper-platform/241.0.20240528.180551

System Details

As per verifier environment

Installed Packages

n/a

Output Log

2024-06-10 01:03:39,185 4952 [INFO ] - Download of JetBrains.dotUltimate.2024.1.3.exe (2.55 GB) completed.
2024-06-10 01:03:42,544 4952 [DEBUG] - No runtime virus checking built into FOSS Chocolatey. Check out Pro/Business - https://chocolatey.org/compare
2024-06-10 01:03:42,637 4952 [DEBUG] - Verifying package provided checksum of '5342ede24c95a30261c0d43402c5cd3687d4b208233f028d7cb1ef7fe99a5a25' for 'C:\ProgramData\chocolatey\lib\resharper-platform\JetBrains.dotUltimate.2024.1.3.exe'.
2024-06-10 01:03:42,793 4952 [DEBUG] - Running Get-ChecksumValid -file 'C:\ProgramData\chocolatey\lib\resharper-platform\JetBrains.dotUltimate.2024.1.3.exe' -checksum '5342ede24c95a30261c0d43402c5cd3687d4b208233f028d7cb1ef7fe99a5a25' -checksumType 'sha256' -originalUrl 'https://download.jetbrains.com/resharper/dotUltimate.2024.1.3/JetBrains.dotUltimate.2024.1.3.exe' 
2024-06-10 01:03:43,091 4952 [DEBUG] - checksum.exe found at 'C:\ProgramData\chocolatey\helpers\..\tools\checksum.exe'
2024-06-10 01:03:43,123 4952 [DEBUG] - Executing command ['C:\ProgramData\chocolatey\helpers\..\tools\checksum.exe' -c="5342ede24c95a30261c0d43402c5cd3687d4b208233f028d7cb1ef7fe99a5a25" -t="sha256" -f="C:\ProgramData\chocolatey\lib\resharper-platform\JetBrains.dotUltimate.2024.1.3.exe"]
2024-06-10 01:40:08,840 4952 [WARN ] - Chocolatey timed out waiting for the command to finish. The timeout
 specified (or the default value) was '2700' seconds. Perhaps try a
 higher `--execution-timeout`? See `choco -h` for details.
2024-06-10 01:40:08,888 4952 [DEBUG] - Calling command ['"C:\Windows\System32\shutdown.exe" /a']
2024-06-10 01:40:09,955 4952 [DEBUG] - Command ['"C:\Windows\System32\shutdown.exe" /a'] exited with '1116'
2024-06-10 01:40:10,683 4952 [DEBUG] - Capturing package files in 'C:\ProgramData\chocolatey\lib\resharper-platform'
2024-06-10 01:40:10,745 4952 [WARN ] - Error computing hash for 'C:\ProgramData\chocolatey\lib\resharper-platform\JetBrains.dotUltimate.2024.1.3.exe'
 Hash will be special code for locked file or file too big instead.
 Captured error:
  The file is too long. This operation is currently limited to supporting files less than 2 gigabytes in size.


### Additional Context

_No response_
@flcdrg flcdrg added the Bug Issues where something has happened which was not expected or intended label Jun 10, 2024
flcdrg referenced this issue in flcdrg/choco Jun 10, 2024
@flcdrg
Use Get-FileHash cmdlet
9f359fe 
- Use this cmdlet if it is available in preference to checksum.exe as it should be faster and support larger file sizes

Resolves #3462
@flcdrg flcdrg linked a pull request Jun 10, 2024 that will close this issue
Open
10 tasks
@TheCakeIsNaOH
Copy link
Member

checksum.exe can hash files that are larger than 2gb, see https://github.com/chocolatey/choco/issues

The warning about Error computing hash for is from the file hashing built into Chocolatey CLI itself: chocolatey/choco#747
I have a PR open to fix it: chocolatey/choco#2714

The hashing taking way longer than alternative hashing methods would be an issue (although it probably would need to be benchmarked), which could be related tochecksum.exe (or some aspect of how it is called).

flcdrg referenced this issue in flcdrg/choco Jun 11, 2024
@flcdrg
Use Get-FileHash cmdlet
dc5b744 
- Use this cmdlet if it is available in preference to checksum.exe as it should be faster and support larger file sizes

Resolves #3462
@pauby pauby transferred this issue from chocolatey/choco Jun 11, 2024
@pauby pauby added Improvement Issues that enhances existing functionality, or adds new features and removed Bug Issues where something has happened which was not expected or intended labels Jun 11, 2024
@flcdrg
Copy link
Member Author

flcdrg commented Jun 11, 2024

Might be worth looking at the Get-FileHash source at https://github.com/PowerShell/PowerShell/blob/master/src/Microsoft.PowerShell.Commands.Utility/commands/utility/GetHash.cs to see if there's tweaks you can adopt here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Improvement Issues that enhances existing functionality, or adds new features
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

(#3462) Use Get-FileHash cmdlet where possible flcdrg/choco
3 participants
@flcdrg @pauby @TheCakeIsNaOH