How to convert YOLOv5 model to ncnn model

ncnn

This article explains how to convert a YOLOv5 PyTorch model to an ncnn model. Once the PyTorch model (.pt) is converted to an ONNX model, and then from the ONNX model to the ncnn model (.param and .bin). In addition, some notes on YOLOv5 are explained.

Environment

Setup

In this article, ncnn was installed on Jetson Nano, but this time ncnn is installed on the host PC to check the operation on the host PC.

sudo apt update sudo apt install -y cmake build-essential libprotobuf-dev protobuf-compiler libopencv-dev git clone https://github.com/Tencent/ncnn cd ncnn mkdir build cd build cmake -DCMAKE_BUILD_TYPE=Release -DNCNN_VULKAN=OFF -DNCNN_SYSTEM_GLSLANG=ON -DNCNN_BUILD_EXAMPLES=ON .. make -j make install
Code language: Bash (bash)

For detailed installation instructions, please refer to this article and this article.

operation check

Execute the following. Repository shaoshengsong/yolov5_62_export_ncnn has yolov5s_6.2.param and yolov5s_6.2.bin which are referenced by the yolov5 sample code in ncnn.

git clone https://github.com/shaoshengsong/yolov5_62_export_ncnn cd yolov5_62_export_ncnn ../examples/yolov5 bus.jpg
Code language: Bash (bash)

The following images are shown.

bus.jpg with ncnn yolov5

Conversion from PyTorch model to ONNX model

To convert a PyTorch model (.pt) to an ONNX model, use the shaoshengsong/yolov5_62_export_ncnn repository that was cloned above. the original YOLOv5 repository also has a script to convert to ONNX, but the output results differently.

$ pwd yolov5_62_export_ncnn
Code language: Bash (bash)

Install the necessary packages.

sudo apt install -y python3-pip python3 -m pip install -U pip python3 -m pip install -r requirements.txt
Code language: Bash (bash)

Retrieve a PyTorch model of YOLOv5 and convert it to an ONNX model.

$ get https://github.com/ultralytics/yolov5/releases/download/v6.2/yolov5s.pt $ python3 export.py --weights yolov5s.pt --img 640 --batch 1 --include onnx --simplify $ ls yolov5s.onnx yolov5s.onnx
Code language: Bash (bash)

Conversion from ONNX model to ncnn model

The conversion from ONNX models to ncnn models is done using onnx2ncnn, which is created in build/install/bin/onnx2ncnn when ncnn is built & installed.

../install/bin/onnx2ncnn yolov5s.onnx yolov5s.param yolov5s.bin
Code language: Bash (bash)

Now, if we look at ncnn’s YOLOv5 sample code, we can see that the model is loaded by directly specifying yolov5s_6.2.param and yolov5s_6.2.bin. The files created this time are yolov5s.param and yolov5s.bin, so the file names should be changed as follows

diff --git a/examples/yolov5.cpp b/examples/yolov5.cpp index 88f6db2..ef5b2d5 100644 --- a/examples/yolov5.cpp +++ b/examples/yolov5.cpp @@ -280,9 +280,9 @@ static int detect_yolov5(const cv::Mat& bgr, std::vector<Object>& objects) // original pretrained model from https://github.com/ultralytics/yolov5 // the ncnn model https://github.com/nihui/ncnn-assets/tree/master/models #if YOLOV5_V62 - if (yolov5.load_param("yolov5s_6.2.param")) + if (yolov5.load_param("yolov5s.param")) exit(-1); - if (yolov5.load_model("yolov5s_6.2.bin")) + if (yolov5.load_model("yolov5s.bin")) exit(-1); #elif YOLOV5_V60 if (yolov5.load_param("yolov5s_6.0.param"))
Code language: Diff (diff)

Build and run again.

$ make -C .. $ ../examples/yolov5 bus.jpg find_blob_index_by_name 353 failed Try ex.extract("output", out0); ex.extract("354", out1); ex.extract("366", out2); Floating point exception (core dumped)
Code language: Bash (bash)

Unlike before, an error was displayed and bus.jpg did not show up.

Here is a brief explanation of how to fix it.

First, let’s examine yolov5s.param for the above error.

$ grep -w -e output -e 354 -e 366 yolov5s.param Permute /model.24/Transpose 1 1 /model.24/Reshape_output_0 output 0=1 Permute /model.24/Transpose_1 1 1 /model.24/Reshape_1_output_0 354 0=1 Permute /model.24/Transpose_2 1 1 /model.24/Reshape_2_output_0 366 0=1
Code language: Bash (bash)

It contained the numbers output, 354 and 366. let’s visualize yolov5s.onnx at https://netron.app.

The yolov5s.onnx seems to have three outputs: output, 354, and 366.

On the other hand, examine yolov5s_6.2.param in shaoshengsong/yolov5_62_export_ncnn.

$ grep Permute yolov5s_6.2.param Permute Transpose_200 1 1 338 output 0=1 Permute Transpose_203 1 1 352 353 0=1 Permute Transpose_206 1 1 366 367 0=1
Code language: Bash (bash)

The display is slightly different, but output, 353 and 367 seem to be the outputs. This differs from the values (354, 366) examined yolov5s.param.

If we look at the sample code for YOLOv5 in ncnn, the following code exists.

$ grep -w -e 353 -e 367 ../../examples/yolov5.cpp ex.extract("353", out); ex.extract("367", out);
Code language: Bash (bash)

It seems that output is specified here. yolov5.cpp should be modified as follows.

@@ -366,7 +366,7 @@ static int detect_yolov5(const cv::Mat& bgr, std::vector<Object>& objects) ncnn::Mat out; #if YOLOV5_V62 - ex.extract("353", out); + ex.extract("354", out); #elif YOLOV5_V60 ex.extract("376", out); #else @@ -391,7 +391,7 @@ static int detect_yolov5(const cv::Mat& bgr, std::vector<Object>& objects) { ncnn::Mat out; #if YOLOV5_V62 - ex.extract("367", out); + ex.extract("366", out); #elif YOLOV5_V60 ex.extract("401", out); #else
Code language: Diff (diff)

Build and run again.

$ make -C .. $ ../examples/yolov5 bus.jpg 67 = 0.99632 at 656.76 509.26 152.24 x 0.00 67 = 0.99476 at 644.50 499.47 164.50 x 0.00 67 = 0.99158 at 809.00 464.90 0.00 x 0.00 67 = 0.98984 at 809.00 452.97 0.00 x 0.00 67 = 0.98339 at 195.63 596.97 222.76 x 0.00 67 = 0.98278 at 749.81 304.04 59.19 x 0.00 67 = 0.98148 at 809.00 294.05 0.00 x 0.00 67 = 0.97992 at 0.00 523.36 195.36 x 0.00 67 = 0.97671 at 726.73 307.79 82.27 x 0.00 67 = 0.97668 at 45.13 341.54 48.62 x 0.00 ...
Code language: Bash (bash)

The error no longer appears, but the following image is shown.

The details can be found in this article, but we will modify the parameters of Reshape. The format of the parameters can be found here.

$ grep ^Reshape yolov5s.param Reshape /model.24/Reshape 1 1 /model.24/m.0/Conv_output_0 /model.24/Reshape_output_0 0=6400 1=85 2=3 Reshape /model.24/Reshape_1 1 1 /model.24/m.1/Conv_output_0 /model.24/Reshape_1_output_0 0=1600 1=85 2=3 Reshape /model.24/Reshape_2 1 1 /model.24/m.2/Conv_output_0 /model.24/Reshape_2_output_0 0=400 1=85 2=3 $ $ grep ^Reshape yolov5s_6.2.param Reshape Reshape_199 1 1 326 338 0=-1 1=85 2=3 Reshape Reshape_202 1 1 340 352 0=-1 1=85 2=3 Reshape Reshape_205 1 1 354 366 0=-1 1=85 2=3
Code language: Bash (bash)

The yolov5s.param has 0=6400, 0=1600, and 0=400.

On the other hand, yolov5s_6.2.param has 0=-1, 0=-1, and 0=-1.

Modify yolov5s.param to 0=-1.

@@ -165,11 +165,11 @@ Convolution /model.23/cv3/conv/Conv 1 1 /model.23/Concat_output_0 /model.23/cv3/conv/Conv_output_0 0=512 1=1 11=1 2=1 12=1 3=1 13=1 4=0 14=0 15=0 16=0 5=1 6=262144 Swish /model.23/cv3/act/Mul 1 1 /model.23/cv3/conv/Conv_output_0 /model.23/cv3/act/Mul_output_0 Convolution /model.24/m.0/Conv 1 1 /model.17/cv3/act/Mul_output_0_splitncnn_0 /model.24/m.0/Conv_output_0 0=255 1=1 11=1 2=1 12=1 3=1 13=1 4=0 14=0 15=0 16=0 5=1 6=32640 -Reshape /model.24/Reshape 1 1 /model.24/m.0/Conv_output_0 /model.24/Reshape_output_0 0=6400 1=85 2=3 +Reshape /model.24/Reshape 1 1 /model.24/m.0/Conv_output_0 /model.24/Reshape_output_0 0=-1 1=85 2=3 Permute /model.24/Transpose 1 1 /model.24/Reshape_output_0 output 0=1 Convolution /model.24/m.1/Conv 1 1 /model.20/cv3/act/Mul_output_0_splitncnn_0 /model.24/m.1/Conv_output_0 0=255 1=1 11=1 2=1 12=1 3=1 13=1 4=0 14=0 15=0 16=0 5=1 6=65280 -Reshape /model.24/Reshape_1 1 1 /model.24/m.1/Conv_output_0 /model.24/Reshape_1_output_0 0=1600 1=85 2=3 +Reshape /model.24/Reshape_1 1 1 /model.24/m.1/Conv_output_0 /model.24/Reshape_1_output_0 0=-1 1=85 2=3 Permute /model.24/Transpose_1 1 1 /model.24/Reshape_1_output_0 354 0=1 Convolution /model.24/m.2/Conv 1 1 /model.23/cv3/act/Mul_output_0 /model.24/m.2/Conv_output_0 0=255 1=1 11=1 2=1 12=1 3=1 13=1 4=0 14=0 15=0 16=0 5=1 6=130560 -Reshape /model.24/Reshape_2 1 1 /model.24/m.2/Conv_output_0 /model.24/Reshape_2_output_0 0=400 1=85 2=3 +Reshape /model.24/Reshape_2 1 1 /model.24/m.2/Conv_output_0 /model.24/Reshape_2_output_0 0=-1 1=85 2=3 Permute /model.24/Transpose_2 1 1 /model.24/Reshape_2_output_0 366 0=1
Code language: Diff (diff)

This is a modification of yolov5s.param, so no build is required. Once corrected, run it again.

../examples/yolov5 bus.jpg
Code language: Bash (bash)

It showed up well!

Below is a video of YOLOv5 running on Jetson Nano using the ncnn model with the results trained with own dataset.

ncnn YOLOv5 with own dataset on Jetson Nano

That’s all.

Reference

GitHub - Tencent/ncnn: ncnn is a high-performance neural network inference framework optimized for the mobile platform
ncnn is a high-performance neural network inference framework optimized for the mobile platform - GitHub - Tencent/ncnn: ncnn is a high-performance neural netwo...
how to build
ncnn is a high-performance neural network inference framework optimized for the mobile platform - Tencent/ncnn
Tutorial for compiling NCNN library - Zekun Wang's Blog | Data Science
Across the Great Wall, we can reach every corner in the world
详细记录u版YOLOv5目标检测ncnn实现
详细记录u版YOLOv5目标检测ncnn实现 允许在不修改内容前提下转载本文 0x0 u版YOLOv5众所周知,原版YOLO系列是 darknet 框架训练的,而广泛使用的是 YOLOv4 作者 AlexeyAB 的版本 AlexeyAB 首字母是a,于是也被叫做…
operation param weight table
ncnn is a high-performance neural network inference framework optimized for the mobile platform - Tencent/ncnn
Netron
Visualizer for neural network, deep learning and machine learning models.