Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TunableOp hotfix #129281

Closed
wants to merge 3 commits into from
Closed

TunableOp hotfix #129281

wants to merge 3 commits into from
+55 −18

Conversation

jeffdaily
Copy link
Collaborator

Fixes.

  • PYTORCH_TUNABLEOP_NUMERICAL_CHECK=1 had a memory leak.
  • The strided batched gemm size calculation for buffer rotation was incorrect resulting in a mem fault.

@jeffdaily jeffdaily requested a review from eqy as a code owner June 21, 2024 22:15
@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Jun 21, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129281

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 315ba1b with merge base 920ebcc (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@jeffdaily jeffdaily changed the title Tunableop hotfix TunableOp hotfix Jun 21, 2024
@jeffdaily jeffdaily requested a review from mxz297 June 21, 2024 22:16
xw285cornell
xw285cornell approved these changes Jun 21, 2024
View reviewed changes
Copy link
Contributor

@xw285cornell xw285cornell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jeffdaily thanks for the fix! I wonder if we can add unit tests to tunable op?

@jeffdaily
Copy link
Collaborator Author

@pytorchbot rebase

@pytorchmergebot
Copy link
Collaborator

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

@pytorchmergebot
Copy link
Collaborator

Successfully rebased tunableop_hotfix2 onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout tunableop_hotfix2 && git pull --rebase)

@facebook-github-bot
Copy link
Contributor

@mxz297 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@mxz297
Copy link

mxz297 commented Jun 24, 2024

Still ran into memory access fault at model level. It looks like my previous repo did not fully capture the tensor layout. A new repo has been provided to @jeffdaily

@facebook-github-bot
Copy link
Contributor

@mxz297 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@mxz297
Copy link

mxz297 commented Jun 25, 2024

Resolved memory access fault observed for tunable op in internal workload.

@jeffdaily
Copy link
Collaborator Author

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 25, 2024
@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: This PR needs a release notes: label
If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Details for Dev Infra team Raised by workflow job

@jeffdaily jeffdaily added the topic: not user facing topic category label Jun 25, 2024
@jeffdaily
Copy link
Collaborator Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@pytorchmergebot
Copy link
Collaborator

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / macos-py3-arm64-mps / test (mps, 1, 1, macos-m1-13)

Details for Dev Infra team Raised by workflow job

@jeffdaily
Copy link
Collaborator Author

@pytorchbot merge -f "unrelated mac failure"

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@jeffdaily
Copy link
Collaborator Author

@pytorchmergebot cherry-pick --onto release/2.4 -c critical

pytorchbot pushed a commit that referenced this pull request Jun 25, 2024
@jeffdaily @pytorchbot
TunableOp hotfix (#129281)
519516c 
Fixes.
- PYTORCH_TUNABLEOP_NUMERICAL_CHECK=1 had a memory leak.
- The strided batched gemm size calculation for buffer rotation was incorrect resulting in a mem fault.
Pull Request resolved: #129281
Approved by: https://github.com/xw285cornell, https://github.com/eqy, https://github.com/mxz297

(cherry picked from commit e68ee2c)
@pytorchbot pytorchbot mentioned this pull request Jun 25, 2024
Open
@pytorchbot
Copy link
Collaborator

Cherry picking #129281

The cherry pick PR is at #129499 and it is recommended to link a critical cherry pick PR with an issue. The following tracker issues are updated:

Details for Dev Infra team Raised by workflow job

jeffdaily added a commit to ROCm/pytorch that referenced this pull request Jun 26, 2024
@jeffdaily
TunableOp hotfix, unit test follow-up
a5c3e9f 
PR pytorch#129281 was landed to fix critical issues but did not contain unit
tests to exercise those issues.  This is a follow-up set of unit tests
that would exercise the problems seen previously.
@jeffdaily jeffdaily mentioned this pull request Jun 26, 2024
Closed
jeffdaily added a commit to ROCm/pytorch that referenced this pull request Jun 26, 2024
@jeffdaily
TunableOp hotfix, unit test follow-up
96cd3d3 
PR pytorch#129281 was landed to fix critical issues but did not contain unit
tests to exercise those issues.  This is a follow-up set of unit tests
that would exercise the problems seen previously.
@jeffdaily jeffdaily mentioned this pull request Jun 26, 2024
Merged
pytorchmergebot pushed a commit that referenced this pull request Jun 27, 2024
@jeffdaily @pytorchmergebot
TunableOp hotfix, unit test follow-up (#129606)
04206d1 
PR #129281 was landed to fix critical issues but did not contain unit tests to exercise those issues.  This is a follow-up set of unit tests that would exercise the problems seen previously.
Pull Request resolved: #129606
Approved by: https://github.com/atalman
atalman pushed a commit that referenced this pull request Jun 27, 2024
@pytorchbot @jeffdaily
TunableOp hotfix (#129499)
ca8d4d1 
TunableOp hotfix (#129281)

Fixes.
- PYTORCH_TUNABLEOP_NUMERICAL_CHECK=1 had a memory leak.
- The strided batched gemm size calculation for buffer rotation was incorrect resulting in a mem fault.
Pull Request resolved: #129281
Approved by: https://github.com/xw285cornell, https://github.com/eqy, https://github.com/mxz297

(cherry picked from commit e68ee2c)

Co-authored-by: Jeff Daily <[email protected]>
atalman pushed a commit that referenced this pull request Jun 27, 2024
@jeffdaily
Tunableop hotfix2 unit tests, release/2.4 (#129607)
d0831d6 
TunableOp hotfix, unit test follow-up

PR #129281 was landed to fix critical issues but did not contain unit
tests to exercise those issues.  This is a follow-up set of unit tests
that would exercise the problems seen previously.
@fxmarty fxmarty mentioned this pull request Jun 28, 2024
Closed
fxmarty pushed a commit to fxmarty/pytorch that referenced this pull request Jun 28, 2024
@jeffdaily @fxmarty
TunableOp hotfix (pytorch#129281)
d877191 
Fixes.
- PYTORCH_TUNABLEOP_NUMERICAL_CHECK=1 had a memory leak.
- The strided batched gemm size calculation for buffer rotation was incorrect resulting in a mem fault.
Pull Request resolved: pytorch#129281
Approved by: https://github.com/xw285cornell, https://github.com/eqy, https://github.com/mxz297
fxmarty pushed a commit to fxmarty/pytorch that referenced this pull request Jun 28, 2024
@jeffdaily @fxmarty
TunableOp hotfix, unit test follow-up (pytorch#129606)
adbb514 
PR pytorch#129281 was landed to fix critical issues but did not contain unit tests to exercise those issues.  This is a follow-up set of unit tests that would exercise the problems seen previously.
Pull Request resolved: pytorch#129606
Approved by: https://github.com/atalman
fxmarty pushed a commit to fxmarty/pytorch that referenced this pull request Jun 28, 2024
@jeffdaily @fxmarty
TunableOp hotfix (pytorch#129281)
ceaa1e4 
Fixes.
- PYTORCH_TUNABLEOP_NUMERICAL_CHECK=1 had a memory leak.
- The strided batched gemm size calculation for buffer rotation was incorrect resulting in a mem fault.
Pull Request resolved: pytorch#129281
Approved by: https://github.com/xw285cornell, https://github.com/eqy, https://github.com/mxz297
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mxz297 mxz297 mxz297 approved these changes

@eqy eqy eqy approved these changes

@xw285cornell xw285cornell xw285cornell approved these changes

Assignees
No one assigned
Labels
ciflow/rocm ciflow/trunk Trigger trunk jobs on your pull request Merged open source topic: not user facing topic category
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@jeffdaily @pytorchmergebot @facebook-github-bot @mxz297 @pytorchbot @eqy @xw285cornell