在langchain-chatchat里得到的回答，和大模型直接推理得到的回答，不一样

[KevinFanng]

问题描述 / Problem Description
在langchain-chatchat里得到的回答，和大模型直接推理得到的回答，不一样

复现问题的步骤 / Steps to Reproduce
1.使用LLaMA-Factory的推理程序CUDA_VISIBLE_DEVICES=0 llamafactory-cli chat examples/inference/llama3.yaml
2.在langchain-chatchat的LLM对话界面
输入同样的问题，得到不一样的答案

预期的结果 / Expected Result
应该得到同样的回答

实际结果 / Actual Result
回答不一样

环境信息 / Environment Information
项目版本：v0.2.10
langchain版本：0.0.354. fastchat版本：0.2.35

• 是否使用 Docker 部署（是/否）：否

• 使用的模型（ChatGLM2-6B / Qwen-7B 等）：LLaMA3-8B-Chat、Qwen1.5-0.5B-Chat

• 使用的 Embedding 模型（moka-ai/m3e-base 等）：bge-reranker-large

• 使用的向量库类型 (faiss / milvus / pg_vector 等)： faiss

• 操作系统及版本 / Operating system and version:操作系统：Linux-4.19.24-7.34.cbp.al7.x86_64-x86_64-with-glibc2.35.

• python版本：3.11.9 (main, Apr 19 2024, 16:48:06) [GCC 11.2.0]

[dosubot]

To address the discrepancy in answers between Langchain-Chatchat and direct model inference, consider the following steps:

1. Model Configuration: Ensure both Langchain-Chatchat and your direct inference setup use identical model configurations, including parameters and version.

2. Input Processing: Verify that input data preprocessing, such as tokenization and normalization, is consistent across both setups. Langchain-Chatchat has specific preprocessing steps, including handling message history and constructing prompts, which should be mirrored in your direct inference process.

3. Inference Engine Settings: Align the settings of the underlying inference engine (e.g., PyTorch, TensorFlow) between both setups to ensure consistency.

4. Model Versions: Update both Langchain-Chatchat and your direct inference setup to the latest model versions if discrepancies exist.

5. External Dependencies: Ensure that all external dependencies, such as language model libraries, are the same version in both environments.

6. Knowledge Base: If Langchain-Chatchat uses a local knowledge base, make sure it's also used in your direct inference setup.

7. Debugging and Logs: Review logs from both setups for any errors or warnings that could explain the output differences.

8. Documentation and Community: Consult the official documentation and community forums for advice on achieving consistent inference results.

For detailed guidance, refer to Langchain-Chatchat's documentation and source code, especially sections related to chat processing, configuration examples, and knowledge base integration. This approach will help you identify and resolve the differences in answers between the two setups.

Details

Sources

• [BUG] 使用微调模型进行对话异常，其他工具调用正常
• knowledge_base/samples/content/test_files/langchain-ChatGLM_open.jsonl
• knowledge_base/samples/content/test_files/langchain-ChatGLM_closed.jsonl
• server/chat/chat.py
• configs/kb_config.py.example
• server/chat/search_engine_chat.py
• README_en.md
• README_ja.md
• README.md
• Dockerfile
• .gitmodules
• requirements_lite.txt
• server/api.py
• server/chat/knowledge_base_chat.py
• server/agent/tools/search_knowledgebase_once.py
• configs/basic_config.py.example
• startup.py
• server/agent/tools/search_knowledgebase_complex.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[suiquantong]

我也发现这个问题了