-
Notifications
You must be signed in to change notification settings - Fork 231
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Halve model loading time for llama demo #4032
base: main
Are you sure you want to change the base?
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/4032
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 4d922b4 with merge base 3eec95a ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This pull request was exported from Phabricator. Differential Revision: D58826044 |
This pull request was exported from Phabricator. Differential Revision: D58826044 |
b8da11a
to
01ff5b5
Compare
Summary: Pull Request resolved: pytorch#4032 mmap is not recommended for large sequential workloads -- you have to take a bunch of page faults. Surprisingly, this doesn't seem to hurt reported peak memory usage. Differential Revision: D58826044
01ff5b5
to
8daba50
Compare
Summary: Pull Request resolved: pytorch#4032 mmap is not recommended for large sequential workloads -- you have to take a bunch of page faults. Surprisingly, this doesn't seem to hurt reported peak memory usage. Differential Revision: D58826044
This pull request was exported from Phabricator. Differential Revision: D58826044 |
Summary: Pull Request resolved: pytorch#4032 mmap is not recommended for large sequential workloads -- you have to take a bunch of page faults. I originally assumed this would hurt peak memory usage (we read all the weights into memory at once and then pack them; packing is basically copying them), but it doesn't. In retrospect, this makes sense because we actually operate on one weights tensor at a time, and the individual tensors aren't gigantic, there are just a lot of them. Reviewed By: larryliu0820 Differential Revision: D58826044
This pull request was exported from Phabricator. Differential Revision: D58826044 |
8daba50
to
4d922b4
Compare
Summary:
mmap is not recommended for large sequential workloads -- you
have to take a bunch of page faults. Surprisingly, this doesn't seem
to hurt reported peak memory usage.
Differential Revision: D58826044