Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new section to change LLM model such as deepseek based on validated model table in LLM microservice #1501

Merged
merged 4 commits into from
Feb 12, 2025
Merged

Add a new section to change LLM model such as deepseek based on validated model table in LLM microservice #1501

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
11a58be
Update README.md for changing llm model based on validated model table
louie-tsai Feb 7, 2025
09c9685
Update ChatQnA/docker_compose/intel/cpu/xeon/README.md
louie-tsai Feb 11, 2025
49a7715
Update ChatQnA/docker_compose/intel/hpu/gaudi/README.md
louie-tsai Feb 11, 2025
16d4685
Merge branch 'main' into change_model
xiguiw Feb 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions ChatQnA/docker_compose/intel/cpu/xeon/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,22 @@ To set up environment variables for deploying ChatQnA services, follow these ste
```

3. Set up other environment variables:

```bash
source ./set_env.sh
```

4. Change Model for LLM serving

By default, Meta-Llama-3-8B-Instruct is used for LLM serving, the default model can be changed to other validated LLM models.
Please pick a [validated llm models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#validated-llm-models) from the table.
To change the default model defined in set_env.sh, overwrite it by exporting LLM_MODEL_ID to the new model or by modifying set_env.sh, and then repeat step 3.
For example, change to Llama-2-7b-chat-hf using the following command.

```bash
export LLM_MODEL_ID="meta-llama/Llama-2-7b-chat-hf"
```

## Quick Start: 2.Run Docker Compose

```bash
Expand Down
19 changes: 19 additions & 0 deletions ChatQnA/docker_compose/intel/hpu/gaudi/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,25 @@ To set up environment variables for deploying ChatQnA services, follow these ste
source ./set_env.sh
```

4. Change Model for LLM serving

By default, Meta-Llama-3-8B-Instruct is used for LLM serving, the default model can be changed to other validated LLM models.
Please pick a [validated llm models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#validated-llm-models) from the table.
To change the default model defined in set_env.sh, overwrite it by exporting LLM_MODEL_ID to the new model or by modifying set_env.sh, and then repeat step 3.
For example, change to DeepSeek-R1-Distill-Qwen-32B using the following command.

```bash
export LLM_MODEL_ID="deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
```

Please also check [required gaudi cards for different models](https://github.com/opea-project/GenAIComps/tree/main/comps/llms/src/text-generation#system-requirements-for-llm-models) for new models.
It might be necessary to increase the number of Gaudi cards for the model by exporting NUM_CARDS to the new model or by modifying set_env.sh, and then repeating step 3. For example, increase the number of Gaudi cards for DeepSeek-R1-
Distill-Qwen-32B using the following command:

```bash
export NUM_CARDS=4
```

## Quick Start: 2.Run Docker Compose

```bash
Expand Down