Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement chunked and parallel Upload in serve WebDAV . #7860

Open
MythodeaLoL opened this issue May 20, 2024 · 0 comments
Open

Implement chunked and parallel Upload in serve WebDAV . #7860

MythodeaLoL opened this issue May 20, 2024 · 0 comments

Comments

@MythodeaLoL
Copy link

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/add-general-webdav-serve-chunked-upload-support/46089

What is your current rclone version (output from rclone version)?

v1.66.0.0

What problem are you are trying to solve?

rclone webdav does not support receiving Upload in chunks.
The backend server where rclone serves webdav only accepts 100MB POST uploads.

How do you think rclone should be changed to solve that?

rclone webdav does not support receiving Upload in chunks.
The backend server where rclone serves webdav only accepts 100MB POST uploads.

Add support for sending chunked uploads to the webdav backend, without the remote file in webdav being chunked/fragmented.

Add support for receiving uploads in chunks by the webdav server rclone serves webdav with vendor = other and type = webdav, merging the fragments into a single file.

Currently it is only possible to send uploads with chunker mode:

[chunker]
type = chunker
remote = Remote_Webdav_Server:
chunk_size=100M

However, this method creates several parts of files on the webdav backend server, the files are not merged.

What I'm proposing is that webdav works normally, however with a command --webdav-upload-chunk-size or --webdav-upload-cutoff the upload would be done in parts, and received in parts by the webdav backend works, and written normally in the filesystem without creating file parts.
--webdav-upload-cutoff=100M activates the sending of the file in chunks over the network, through Accept-Ranges: bytes

I also propose the option of sending multiple parts in parallel, speeding up file upload. --webdav-upload-parallels
The default would be 1 part at a time.
--webdav-upload-parallels=5 would enable 5-part parallel upload.

Implementation already present in part in nextcloud webdav, related #3503

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant