-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add chatglm2 & chatglm3 autotp #5540
base: master
Are you sure you want to change the base?
Conversation
@@ -476,7 +481,9 @@ def _replace_module(self, r_module, prev_name='', prev_class_name=''): | |||
|
|||
def get_model_num_kv_heads(self, config): | |||
num_kv_heads = None | |||
kv_head_names = ['num_kv_heads', 'num_key_value_heads', 'num_attention_heads', 'n_heads'] | |||
kv_head_names = [ | |||
'multi_query_group_num', 'num_kv_heads', 'num_key_value_heads', 'num_attention_heads', 'n_heads' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Yejing-Lai as this list gets longer, it would help if newly added entries can have comments for what model this new entry serves. In case one day this piece information is needed.
Hi @loadams , this PR is ready for review. Can this PR be reviewed? Thanks! |
Hi @loadams. The CI error seems like a network issue. Could you rerun the CI? Thanks! |
This PR aims to enable chatglm2 & chatglm3 autotp. Similar to the phi3, this model uses the chunk MLP layer, so we adjust the weight order by 'shard_mlp_chunk' func. Please kindly review~ Thanks!