New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

使用lmdeploy serve api_server部署chat1.5模型，同时参数设置 Lora 适配器路径，不起作用 #294

Open

njzfw1024 opened this issue Jun 21, 2024 · 0 comments

njzfw1024 commented Jun 21, 2024 •

edited

Loading

CUDA_VISIBLE_DEVICES=0,1 lmdeploy serve api_server /mnt/pretrainmodel/InternVL-Chat-V1-5/ 
--server-port 8080 
--tp 2 
--cache-max-entry-count 0.2
 --session-len 4096 
--vision-max-batch-size 1 
--max-batch-size 64  
-- adapters  /mnt/pretrainmodel/InternVL-Chat-V1-5-lora`
``` `
adapters 路径为lora微调后的模型地址

备注： 测试过在 官方提供的@app.post('/worker_generate_stream')接口  用训练数据测试，是起作用的

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment