support qwen2 1.5b

[lvhan028]

qwen2 1.5b set tie_word_embeddings=True.
In this case, the output layer and the token embedding layer share the same weight

[irexyc]

This will affect many models, like internlm2, internvl

[lvhan028]

Yes, I understand. However, I believe it is necessary to ensure that the original model configuration is accessible to all source models. Otherwise, the model_info() function should be capable of handling all edge cases.

[lvhan028]

TODO: full test all supported models

[zhyncs]

ref https://qwenlm.github.io/blog/qwen2/#model-information

Models	Qwen2-0.5B	Qwen2-1.5B	Qwen2-7B	Qwen2-57B-A14B	Qwen2-72B
Params	0.49B	1.54B	7.07B	57.41B	72.71B
Non-Emb Params	0.35B	1.31B	5.98B	56.32B	70.21B
GQA	True	True	True	True	True
Tie Embedding	True	True	False	False	False
Context Length	32K	32K	128K	64K	128K

For small models, we prefer the application of tying embedding as the large sparse embeddings take up a large proportion of the total model parameters.

[dawnranger]

any plan to support qwen2-0.5b? @lvhan028

[lvhan028]

any plan to support qwen2-0.5b? @lvhan028

qwen2-0.5b defines head_dim 64. But lmdeploy turbomind engine now requires head_dim 128.
So it doesn't support qwen2-0.5b now. You may use the pytorch engine for the qwen2-0.5b model