【Edge deployment】MiniCPM-Llama3-V 2.5
输入“/”快速插入内容
【Edge deployment】MiniCPM-Llama3-V 2.5
【Edge deployment】MiniCPM-Llama3-V 2.5
用户3092024年8月26日修改
👇
Suitable for: Those familiar with the Linux system and have a basic understanding of file management on Android phones.
一、Download the app and deploy directly
Equipment requirements: Android phone, preferably Snapdragon 8 or higher chip
1.
Download apk
2.
Install apk file on mobile phone
3.
Open the wall-facing smart app on your mobile phone and select the miniCPM-V-2.5 model to download.
4.
Wait for the download to complete and you can select pictures for conversation.
二、lama.cpp Android deployment
Equipment requirements: The quantified version of the model requires 8g of memory, and the non-quantified model requires more than 16g of memory.
1.
Get llama.cpp code：
代码块
git clone https://github.com/ggerganov/llama.cpp.git
2.
Get the gguf weight of Minicpmv2.5
You can go directly to huggingface.co to download all the required models, skip the following a, b, c
a.
First go to huggingface or modelscope to download pytorch weights:
代码块
git clone https://huggingface.co/openbmb/MiniCPM-Llama3-V-2_5
#or
git clone https://modelscope.cn/models/OpenBMB/MiniCPM-Llama3-V-2_5.git
b.
Use the above llama.cpp to convert the model weights into a gguf file:
代码块
# The first line is to obtain intermediate model outputs, preparing for conversion to gguf format
python ./examples/minicpmv/minicpmv-surgery.py -m ../MiniCPM-Llama3-V-2_5
# Convert the siglip model to gguf format
python ./examples/minicpmv/minicpmv-convert-image-encoder-to-gguf.py -m ../MiniCPM-Llama3-V-2_5 --minicpmv-projector ../MiniCPM-Llama3-V-2_5/minicpmv.projector --output-dir ../MiniCPM-Llama3-V-2_5/ --image-mean 0.5 0.5 0.5 --image-std 0.5 0.5 0.5
# Convert the language model to gguf format
python ./convert.py ../MiniCPM-Llama3-V-2_5/model --outtype f16 --vocab-type bpe
c.
On the mobile phone, the language module needs to be quantified:
代码块
./quantize ../MiniCPM-Llama3-V-2_5/model/model-8B-F16.gguf ../MiniCPM-Llama3-V-2_5/model/ggml-model-Q4_K_M.gguf Q4_K_M
3.
4.
After downloading and unzipping, enter llama.cpp and create a new folder build_android
代码块
cd llama.cpp
mkdir build_android
5.
Deployment of compilation environment
代码块
cd build_android
export NDK=/your_ndk_path
# Example of the line above:export NDK=/root/ld/ld_project/llama.cpp/build_android
# Use cmake to configure the build environment
cmake -DCMAKE_TOOLCHAIN_FILE=$NDK/build/cmake/android.toolchain.cmake -DANDROID_ABI=arm64-v8a -DANDROID_PLATFORM=android-23 -DCMAKE_C_FLAGS=-march=armv8.4a+dotprod ..
6.
Compile llama.cpp at this time：
代码块
make
7.
Check whether llama.cpp/build_android/bin/minicpmv-cli exists. If it exists, the compilation is successful.: