yang/CodeGeeX4
1
0
Fork 0
mirror of https://github.com/JasonYANG170/CodeGeeX4.git synced 2024-10-19 14:01:48 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

add vllm implement in README

Browse Source
This commit is contained in:
xinpeng-zhang 2024-07-09 20:14:27 +08:00 committed by GitHub
parent b07f58b5a8
commit 5ab6406fde
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 2 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
README.md
View File

@ -39,7 +39,7 @@ with torch.no_grad():
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) print(tokenizer.decode(outputs[0], skip_special_tokens=True))
``` ```
Use `vllm==0.5.1` to quickly launch Use `vllm==0.5.1` to quickly launch [codegeex4-all-9b](https://huggingface.co/THUDM/codegeex4-all-9b):
``` ```
from transformers import AutoTokenizer from transformers import AutoTokenizer
from vllm import LLM, SamplingParams from vllm import LLM, SamplingParams
@ -70,7 +70,7 @@ outputs = llm.generate(prompts=inputs, sampling_params=sampling_params)
print(outputs[0].outputs[0].text) print(outputs[0].outputs[0].text)
``` ```
Set up OpenAI Compatible Server via vllm using following command, detailed please check [OpenAI Compatible Server Via vllm docs](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html) Set up OpenAI Compatible Server via vllm, detailed please check [OpenAI Compatible Server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html)
``` ```
python -m vllm.entrypoints.openai.api_server \ python -m vllm.entrypoints.openai.api_server \
--model THUDM/codegeex4-all-9b \ --model THUDM/codegeex4-all-9b \