Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] optimize PP overlap #5735

Merged
merged 24 commits into from
Jun 26, 2024
Merged

[Feature] optimize PP overlap #5735

merged 24 commits into from
Jun 26, 2024

Conversation

Edenzzzz
Copy link
Contributor

@Edenzzzz Edenzzzz commented May 20, 2024 •
edited
Loading

📌 Checklist before creating the PR

  • I have created an issue for this PR for traceability
  • The title follows the standard format: [doc/gemini/tensor/...]: A concise description
  • I have added relevant tags if possible for us to better distinguish different PRs
  • I have installed pre-commit: pip install pre-commit && pre-commit install

🚨 Issue number

📝 What does this PR do?

  1. Largely refurbish and clean up interfaces for clarity and maintainability.
  2. Optimize Pipeline Parallel, overlapping communication with computation.

💥 Checklist before requesting a review

  • I have linked my PR to an issue (instruction)
  • My issue clearly describes the problem/feature/proposal, with diagrams/charts/table/code if possible
  • I have performed a self-review of my code
  • I have added thorough tests.
  • I have added docstrings for all the functions/methods I implemented

⭐️ Do you enjoy contributing to Colossal-AI?

  • 🌝 Yes, I do.
  • 🌚 No, I don't.

Tell us more if you don't enjoy contributing to Colossal-AI.

@Edenzzzz Edenzzzz requested a review from a team as a code owner May 20, 2024 08:03
@Edenzzzz Edenzzzz changed the title [Feature] optimizer overlap in PP [Feature] optimize PP overlap May 20, 2024
@Edenzzzz Edenzzzz force-pushed the optimize_3d branch 2 times, most recently from 48c5241 to 3f97cf7 Compare May 22, 2024 09:50
@Edenzzzz
Copy link
Contributor Author

Edenzzzz commented May 24, 2024 •
edited
Loading

Before overlapping:

image

After overlapping:

img_v3_02b6_eefa4ebc-19a6-4ac2-90d8-e455c19dcd7g

ver217
ver217 reviewed Jun 7, 2024
View reviewed changes
colossalai/pipeline/p2p.py Outdated Show resolved Hide resolved
colossalai/pipeline/schedule/one_f_one_b.py Show resolved Hide resolved
colossalai/booster/plugin/hybrid_parallel_plugin.py Outdated Show resolved Hide resolved
colossalai/pipeline/schedule/interleaved_pp.py Outdated Show resolved Hide resolved
colossalai/pipeline/schedule/interleaved_pp.py Show resolved Hide resolved
examples/language/llama/benchmark.py Outdated Show resolved Hide resolved
examples/language/llama/benchmark.py Outdated Show resolved Hide resolved
examples/language/llama/benchmark.py Outdated Show resolved Hide resolved
examples/language/llama/benchmark.py Outdated Show resolved Hide resolved
examples/language/llama/pp.sh Outdated Show resolved Hide resolved
@Edenzzzz
Copy link
Contributor Author

Edenzzzz commented Jun 18, 2024 •
edited
Loading

More benchmarking results:
image

@Edenzzzz Edenzzzz enabled auto-merge (squash) June 18, 2024 06:53
@Edenzzzz Edenzzzz merged commit 2a25a2a into hpcaitech:main Jun 26, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ver217 ver217 ver217 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@Edenzzzz @ver217