MiniCPM-V 2.6 Deployment Tutorial
输入“/”快速插入内容
🔥MiniCPM-V 2.6 Deployment Tutorial
MiniCPM-V 2.6 Deployment Tutorial
飞书用户309
飞书用户69872024年8月20日修改
🙌
Bilibili Accompanying Video: https://www.bilibili.com/video/BV1sM4m1172r/?vd_source=cd29f4e20ef69babd26f4f34cc7c8b3f
Suitable for Individuals: Those who are capable of modifying simple parameters in Python scripts and can use the most basic Bash .
vllm
vllm python code inference
1.
First go to huggingface to download the model weights：
代码块
git clone https://huggingface.co/openbmb/MiniCPM-V-2_6
Or : You can also download the quantized awq model, which is twice as fast and requires only 7G of video memory.
代码块
git clone https://www.modelscope.cn/models/linglingdan/MiniCPM-V_2_6_awq_int4
git clone https://github.com/LDLINGLINGLING/AutoAWQ.git
cd AutoAWQ
pip install e .
Simple comparison between fp16 and awq using vllm (4090 single gpu)
| Awq int4 | fp16 | |
Input image | | ||
prompt | 这张图片描述了什么（What is depicted in the picture?） | | |
speed | input: 172.74 toks/s, output: 28.21 toks/s | input: 113.63 toks/s, output: 32.49 toks/s | |
Time use | 00:07 | 00:12 | |
output | 这幅图片展示了一架商用客机在晴朗的蓝天下飞行。这架飞机是一架空中客车A330-300，可以通过其双发配置、翼尖小翼和机身长度来识别。机身上的中文字符表明这架飞机属于中国南方航空公司，这是一家总部位于中国广州的航空公司。这架飞机的涂装包括蓝色和白色的配色方案，尾翼上有一个红色的标志，这是中国南方航空公司的标志。这幅图片可能是在飞机起飞或降落时拍摄的，因为起落架是可见的。 | 这幅图片展示了一架商用客机在晴朗的蓝天下飞行。飞机的涂装和标志表明它属于中国南方航空公司，这是一家总部位于中国的航空公司。飞机的机型是空中客车A330-300，这是一款中远程宽体双发喷气式客机。飞机的起落架已展开，表明它要么正在起飞，要么正在降落。晴朗的天空和良好的能见度表明飞行条件良好 | |
output translation | This image shows a commercial airliner flying under a clear blue sky. The aircraft, an Airbus A330-300, can be identified by its twin-engine configuration, winglets and fuselage length. Chinese characters on the fuselage indicated the plane belonged to China Southern Airlines, an airline based in Guangzhou, China. The aircraft's livery includes a blue and white color scheme with a red logo on the tail, which is the logo of China Southern Airlines. The image was probably taken during takeoff or landing, as the landing gear is visible. | This image shows a commercial airliner flying under a clear blue sky. The plane's livery and markings indicate it belongs to China Southern Airlines, an airline based in China. The aircraft model is Airbus A330-300, which is a medium- to long-range wide-body twin-engine jetliner. The plane's landing gear was deployed, indicating it was either taking off or landing. Clear skies and good visibility indicate good flying conditions | |
Memory use of model | 7Gb | 16gb | |
Max length of 24g memory | 2048*3 | 2048 | |
Max batch_size of 24g memory | 52 | 2 | |
2.
pip install vllm
代码块
pip install vllm==0.5.4
3.
Create python code to inference by vllm
代码块
from PIL import Image
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
IMAGES = [
"/root/ld/ld_project/MiniCPM-V/assets/airplane.jpeg", # local_path
]
MODEL_NAME = "/root/ld/ld_model_pretrained/Minicpmv2_6" # local_model_path or huggingface_model_name
image = Image.open(IMAGES[0]).convert("RGB")
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, trust_remote_code=True)
llm = LLM(model=MODEL_NAME,
gpu_memory_utilization=1,
trust_remote_code=True,
max_model_len=2048) # if your memory is not enough,reduce it
messages = [{'role': 'user', 'content': '(<image>./</image>)\n' + 'please describe the picture'}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# set stop token id
# 2.0
# stop_token_ids = [tokenizer.eos_id]
# 2.5
#stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
# 2.6
stop_tokens = ['<|im_end|>', '<|endoftext|>']
stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]
# set generate param
sampling_params = SamplingParams(
stop_token_ids=stop_token_ids,
# temperature=0.7,
# top_p=0.8,
# top_k=100,
# seed=3472,
max_tokens=1024,
# min_tokens=150,
temperature=0,
use_beam_search=True,
# length_penalty=1.2,
best_of=3)
# get output
outputs = llm.generate({
"prompt": prompt,
"multi_modal_data": {
"image": image
}
}, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)