RK3588 AI 应用开发 （FCN-语义分割）一、模型训练与转换FCN（全卷积网络，Fully Convolutional Networks）是用于语义分割任务的一种深度学习模型架构，引入了跳跃结构（Skip Architecture），通过融合浅层和深层的特征图，保留更多的细节信息，提升分割精度。此外，FCN还利用多尺度上下文聚合，捕捉不同层级的特征，增强了对不同大小目标的识别能力。FCN的成功推动了语义分割领域的发展，成为后续许多先进模型的基础。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、应用开发1. 开发 Gradio 界面import cv2 import json import base64 import requests import numpy as np import gradio as gr def test_image(image_path): try: image_bgr = cv2.imread(image_path) image_string = cv2.imencode('.jpg', image_bgr)[1].tobytes() image_base64 = base64.b64encode(image_string).decode('utf-8') params = {"image_base64": image_base64} response = requests.post(f'http://{ip}:{port}{url}', data=json.dumps(params), headers={"Content-Type": "application/json"}) if response.status_code == 200: image_base64 = response.json().get("image_base64") image_binary = base64.b64decode(image_base64) image_array = np.frombuffer(image_binary, dtype=np.uint8) image_rgb = cv2.imdecode(image_array, cv2.IMREAD_COLOR) else: image_rgb = None except Exception as e: return None else: return image_rgb if __name__ == "__main__": port = 8000 ip = "127.0.0.1" url = "/v1/FCN" demo = gr.Interface(fn=test_image, inputs=gr.Image(type="filepath"), outputs=["image"], title="FCN 果蔬病虫害分割") demo.launch(share=False, server_port=3000) /home/orangepi/miniconda3/envs/python-3.10.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm * Running on local URL: http://127.0.0.1:3000 * To create a public link, set `share=True` in `launch()`. 2. 编写推理代码class FCN: def __init__(self, model_path): self.num_classes = 117 self.rknn_lite = RKNNLite() self.rknn_lite.load_rknn(model_path) self.rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) self.color_list = np.random.randint(0, 255, size=(self.num_classes, 3), dtype=np.uint8).tolist() def preprocess(self, image): image = image[:, :, ::-1] image = cv2.resize(image, (224, 224)) return np.expand_dims(image, axis=0) def rknn_infer(self, data): outputs = self.rknn_lite.inference(inputs=[data]) return outputs[0] def post_process(self, pred): feat = pred.squeeze() return np.argmax(feat, axis=-1).astype(np.uint8) def predict(self, image): # 图像预处理 data = self.preprocess(image) # 模型推理 pred = self.rknn_infer(data) # 模型后处理 feat = self.post_process(pred) # 生成图像分割结果 canv = np.zeros_like(image) mask = cv2.resize(feat, image.shape[:2][::-1], interpolation=cv2.INTER_NEAREST) for i in range(1, self.num_classes): canv[mask==i] = self.color_list[i] return cv2.addWeighted(image[..., ::-1], 0.5, canv, 0.5, 0) def release(self): self.rknn_lite.release() 3. 图片批量预测import os import cv2 import numpy as np import matplotlib.pyplot as plt from rknnlite.api import RKNNLite model = FCN('model/FCN.rknn') for image in os.listdir("image"): image = cv2.imread(os.path.join("image", image)) image = model.predict(image) plt.imshow(image) plt.axis('off') plt.show() model.release() 4. 创建 Flask 服务import cv2 import base64 import numpy as np from rknnlite.api import RKNNLite from flask import Flask, request, jsonify from flask_cors import CORS app = Flask(__name__) CORS(app) @app.route('/v1/FCN', methods=['POST']) def inference(): data = request.get_json() image_base64 = data.get("image_base64") image_binary = base64.b64decode(image_base64) image_array = np.frombuffer(image_binary, dtype=np.uint8) image_bgr = cv2.imdecode(image_array, cv2.IMREAD_COLOR) image_rgb = model.predict(image_bgr) image_string = cv2.imencode('.jpg', image_rgb)[1].tobytes() image_base64 = base64.b64encode(image_string).decode('utf-8') return jsonify({ "image_base64": image_base64 }), 200 if __name__ == '__main__': model = FCN('model/FCN.rknn') app.run(host='0.0.0.0', port=8000) model.release() W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [00:06:51.738] RKNN Runtime Information: librknnrt version: 1.4.0 (a10f100eb@2022-09-09T09:07:14) I RKNN: [00:06:51.738] RKNN Driver Information: version: 0.9.6 I RKNN: [00:06:51.739] RKNN Model Information: version: 1, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: TFLite, framework layout: NHWC * Serving Flask app '__main__' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8000 * Running on http://192.168.3.50:8000 Press CTRL+C to quit 127.0.0.1 - - [02/May/2025 00:07:17] "POST /v1/FCN HTTP/1.1" 200 - 127.0.0.1 - - [02/May/2025 00:07:24] "POST /v1/FCN HTTP/1.1" 200 - 127.0.0.1 - - [02/May/2025 00:07:31] "POST /v1/FCN HTTP/1.1" 200 - 127.0.0.1 - - [02/May/2025 00:07:39] "POST /v1/FCN HTTP/1.1" 200 - 5. 上传图片预测三、小结本章介绍了基于RK3588平台使用FCN模型进行语义分割的AI应用开发全流程，包括模型训练与转换、Gradio界面开发、推理代码编写、批量预测实现及Flask服务部署。通过该流程，开发者可实现高效的图像分割任务，并在本地或云端进行预测和展示。----转自博客：https://bbs.huaweicloud.com/blogs/451998

RK3588 AI 应用开发 （YOLOX-目标检测）一、模型训练和转换YOLOX是YOLO系列的优化版本，引入了解耦头、数据增强、无锚点以及标签分类等目标检测领域的优秀进展，拥有较好的精度表现，同时对工程部署友好。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、应用开发1. 开发 Gradio 界面import cv2 import json import base64 import requests import numpy as np import gradio as gr def test_image(image_path): try: image_bgr = cv2.imread(image_path) image_string = cv2.imencode('.jpg', image_bgr)[1].tobytes() image_base64 = base64.b64encode(image_string).decode('utf-8') params = {"image_base64": image_base64} response = requests.post(f'http://{ip}:{port}{url}', data=json.dumps(params), headers={"Content-Type": "application/json"}) if response.status_code == 200: image_base64 = response.json().get("image_base64") image_binary = base64.b64decode(image_base64) image_array = np.frombuffer(image_binary, dtype=np.uint8) image_rgb = cv2.imdecode(image_array, cv2.IMREAD_COLOR) else: image_rgb = None except Exception as e: return None else: return image_rgb if __name__ == "__main__": port = 8000 ip = "127.0.0.1" url = "/v1/fish_det" demo = gr.Interface(fn=test_image, inputs=gr.Image(type="filepath"), outputs=["image"], title="YOLOX 深海鱼类检测") demo.launch(share=False, server_port=3000) * Running on local URL: http://127.0.0.1:3000 * To create a public link, set `share=True` in `launch()`. 2. 编写推理代码%%writefile YOLOX/yolox/data/datasets/voc_classes.py #!/usr/bin/env python3 # -*- coding:utf-8 -*- # Copyright (c) Megvii, Inc. and its affiliates. # VOC_CLASSES = ( '__background__', # always index 0 VOC_CLASSES = ( "fish", ) Overwriting YOLOX/yolox/data/datasets/voc_classes.pyimport sys sys.path.append("YOLOX") from yolox.utils import demo_postprocess, multiclass_nms, vis from yolox.data.data_augment import preproc as preprocess from yolox.data.datasets.voc_classes import VOC_CLASSESimport cv2 import numpy as np import ipywidgets as widgets from rknnlite.api import RKNNLite from IPython.display import display class YOLOX: def __init__(self, model_path): self.ratio = None self.rknn_lite = RKNNLite() self.rknn_lite.load_rknn(model_path) self.rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) def preprocess(self, image): start_img, self.ratio = preprocess(image, (320, 320), swap=(0, 1, 2)) return np.expand_dims(start_img, axis=0) def rknn_infer(self, data): outputs = self.rknn_lite.inference(inputs=[data]) return outputs[0] def post_process(self, pred): predictions = demo_postprocess(pred.squeeze(), (320, 320)) boxes = predictions[:, :4] scores = predictions[:, 4:5] * predictions[:, 5:] boxes_xyxy = np.ones_like(boxes) boxes_xyxy[:, 0] = boxes[:, 0] - boxes[:, 2] / 2. boxes_xyxy[:, 1] = boxes[:, 1] - boxes[:, 3] / 2. boxes_xyxy[:, 2] = boxes[:, 0] + boxes[:, 2] / 2. boxes_xyxy[:, 3] = boxes[:, 1] + boxes[:, 3] / 2. boxes_xyxy /= self.ratio dets = multiclass_nms(boxes_xyxy, scores, nms_thr=0.45, score_thr=0.25) return dets def predict(self, image): # 图像预处理 data = self.preprocess(image) # 模型推理 pred = self.rknn_infer(data) # 模型后处理 dets = self.post_process(pred) # 绘制目标检测结果 if dets is not None: final_boxes = dets[:, :4] final_scores, final_cls_inds = dets[:, 4], dets[:, 5] image = vis(image, final_boxes, final_scores, final_cls_inds, conf=0.25, class_names=VOC_CLASSES) return image[..., ::-1] def img2bytes(self, image): """将图片转换为字节码""" return bytes(cv2.imencode('.jpg', image)[1]) def infer_video(self, video_path): """视频推理""" image_widget = widgets.Image(format='jpeg', width=800, height=600) display(image_widget) cap = cv2.VideoCapture(video_path) while True: ret, img_frame = cap.read() if not ret: break image_pred = self.predict(img_frame) image_widget.value = self.img2bytes(image_pred) cap.release() def release(self): """释放资源""" self.rknn_lite.release() 3. 图像预测4. 视频推理5. 创建 Flask 服务import cv2 import base64 import numpy as np from rknnlite.api import RKNNLite from flask import Flask, request, jsonify from flask_cors import CORS app = Flask(__name__) CORS(app) @app.route('/v1/fish_det', methods=['POST']) def inference(): data = request.get_json() image_base64 = data.get("image_base64") image_binary = base64.b64decode(image_base64) image_array = np.frombuffer(image_binary, dtype=np.uint8) image_bgr = cv2.imdecode(image_array, cv2.IMREAD_COLOR) image_rgb = model.predict(image_bgr) image_string = cv2.imencode('.jpg', image_rgb)[1].tobytes() image_base64 = base64.b64encode(image_string).decode('utf-8') return jsonify({ "image_base64": image_base64 }), 200 if __name__ == '__main__': model = YOLOX('model/yolox_fish.rknn') app.run(host='0.0.0.0', port=8000) model.release() 6. 上传图片预测三、小结本章介绍了基于RK3588平台使用YOLOX进行目标检测的全流程，包括模型训练与转换、Gradio界面开发、推理代码编写、图像和视频预测实现，以及Flask服务部署。整体实现了高效的鱼类检测应用，适用于嵌入式设备部署与实际场景应用。----转自博客：https://bbs.huaweicloud.com/blogs/452001

RK3588 AI 应用开发 （InceptionV3-图像分类）一、模型训练与转换Inception V3，GoogLeNet的改进版本，采用InceptionModule和全局平均池化层，v3一个最重要的改进是分解（Factorization），将7x7分解成两个一维的卷积（1x7,7x1），3x3也是一样（1x3,3x1），这样的好处，既可以加速计算（多余的计算能力可以用来加深网络），又可以将1个conv拆成2个conv，使得网络深度进一步增加，增加了网络的非线性。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、应用开发1. 开发 Gradio 界面import cv2 import json import base64 import requests import numpy as np import gradio as gr def test_image(image_path): try: image_bgr = cv2.imread(image_path) image_string = cv2.imencode('.jpg', image_bgr)[1].tobytes() image_base64 = base64.b64encode(image_string).decode('utf-8') params = {"image_base64": image_base64} response = requests.post(f'http://{ip}:{port}{url}', data=json.dumps(params), headers={"Content-Type": "application/json"}) if response.status_code == 200: image_base64 = response.json().get("image_base64") image_binary = base64.b64decode(image_base64) image_array = np.frombuffer(image_binary, dtype=np.uint8) image_rgb = cv2.imdecode(image_array, cv2.IMREAD_COLOR) else: image_rgb = None except Exception as e: return None else: return image_rgb if __name__ == "__main__": port = 8000 ip = "127.0.0.1" url = "/v1/InceptionV3" demo = gr.Interface(fn=test_image, inputs=gr.Image(type="filepath"), outputs=["image"], title="InceptionV3 动物分类") demo.launch(share=False, server_port=3000) /home/orangepi/miniconda3/envs/python-3.10.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm * Running on local URL: http://127.0.0.1:3000 * To create a public link, set `share=True` in `launch()`. 2. 编写推理代码class InceptionV3: def __init__(self, model_path): self.rknn_lite = RKNNLite() self.rknn_lite.load_rknn(model_path) self.rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) self.label = ['antelope', 'badger', 'bat', 'bear', 'bee', 'beetle', 'bison', 'boar', 'butterfly', 'cat', 'caterpillar', 'chimpanzee', 'cockroach', 'cow', 'coyote', 'crab', 'crow', 'deer', 'dog', 'dolphin', 'donkey', 'dragonfly', 'duck', 'eagle', 'elephant', 'flamingo', 'fly', 'fox', 'goat', 'goldfish', 'goose', 'gorilla', 'grasshopper', 'hamster', 'hare', 'hedgehog', 'hippopotamus', 'hornbill', 'horse', 'hummingbird', 'hyena', 'jellyfish', 'kangaroo', 'koala', 'ladybugs', 'leopard', 'lion', 'lizard', 'lobster', 'mosquito', 'moth', 'mouse', 'octopus', 'okapi', 'orangutan', 'otter', 'owl', 'ox', 'oyster', 'panda', 'parrot', 'pelecaniformes', 'penguin', 'pig', 'pigeon', 'porcupine', 'possum', 'raccoon', 'rat', 'reindeer', 'rhinoceros', 'sandpiper', 'seahorse', 'seal', 'shark', 'sheep', 'snake', 'sparrow', 'squid', 'squirrel', 'starfish', 'swan', 'tiger', 'turkey', 'turtle', 'whale', 'wolf', 'wombat', 'woodpecker', 'zebra'] def preprocess(self, image): image = image[:, :, ::-1] image = cv2.resize(image, (224, 224)) return np.expand_dims(image, axis=0) def rknn_infer(self, data): outputs = self.rknn_lite.inference(inputs=[data]) return outputs[0] def post_process(self, pred): clsse = np.argmax(pred, axis=-1) score = pred[0][clsse[0]].item() return self.label[clsse[0]], round(score * 100, 2) def predict(self, image): # 图像预处理 data = self.preprocess(image) # 模型推理 pred = self.rknn_infer(data) # 模型后处理 label, score = self.post_process(pred) # 绘制识别结果 print(f'{label}:{score}%') image = cv2.putText(image, f'{label}:{score}%', (0, 100), cv2.FONT_HERSHEY_TRIPLEX, 4, (0, 255, 0), 8) return image[..., ::-1] def release(self): self.rknn_lite.release() 3. 图片批量预测import os import cv2 import numpy as np import matplotlib.pyplot as plt from rknnlite.api import RKNNLite model = InceptionV3('model/InceptionV3.rknn') for image in os.listdir("image"): image = cv2.imread(os.path.join("image", image)) image = model.predict(image) plt.imshow(image) plt.axis('off') plt.show() model.release() 4. 创建 Flask 服务import cv2 import base64 import numpy as np from rknnlite.api import RKNNLite from flask import Flask, request, jsonify from flask_cors import CORS app = Flask(__name__) CORS(app) @app.route('/v1/InceptionV3', methods=['POST']) def inference(): data = request.get_json() image_base64 = data.get("image_base64") image_binary = base64.b64decode(image_base64) image_array = np.frombuffer(image_binary, dtype=np.uint8) image_bgr = cv2.imdecode(image_array, cv2.IMREAD_COLOR) image_rgb = model.predict(image_bgr) image_string = cv2.imencode('.jpg', image_rgb)[1].tobytes() image_base64 = base64.b64encode(image_string).decode('utf-8') return jsonify({ "image_base64": image_base64 }), 200 if __name__ == '__main__': model = InceptionV3('model/InceptionV3.rknn') app.run(host='0.0.0.0', port=8000) model.release() W rknn-toolkit-lite2 version: 2.3.2 * Serving Flask app '__main__' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8000 * Running on http://192.168.3.50:8000 Press CTRL+C to quit 127.0.0.1 - - [01/May/2025 20:37:00] "POST /v1/InceptionV3 HTTP/1.1" 200 - pig:99.95% 127.0.0.1 - - [01/May/2025 20:37:09] "POST /v1/InceptionV3 HTTP/1.1" 200 - swan:99.95% 127.0.0.1 - - [01/May/2025 20:37:20] "POST /v1/InceptionV3 HTTP/1.1" 200 - cat:97.02% 5. 上传图片预测三、小结本章介绍了基于RK3588平台的InceptionV3图像分类应用开发全流程，包括模型训练与格式转换、Gradio界面设计、推理代码实现、批量预测处理及Flask服务部署，实现了从本地到Web端的高效AI推理应用。----转自博客：https://bbs.huaweicloud.com/blogs/451978

松材线虫病检测1. 数据切分无人机广角拍摄的影像分辨率较高（4000x3000），首先对人工标注好的松材线虫病数据集进行切分，将大图切分成小图并设置不同的切分尺寸（例如：1000x1000、1500x1500、2000x2000）和重叠比例（例如：0%、10%、20%、30%）送入模型进行训练。2. 模型训练YOLOv8自2023年推出后，经过多次优化迭代，其架构设计（如C2F模块、动态标签分配）与训练流程已趋于成熟。例如嵌入式设备依赖v8的轻量化特效，在医疗检测领域，v8的高召回率已被临床验证。YOLO12等虽在理论上超越YOLOv8，但是v8的推理速度仍具不可替代性，目前在工业界广泛采用该版本进行部署。我们使用YOLOv8对等比例缩放后的原始图像和切分后的松材线虫病检测数据集进行训练，提高模型对不同大小目标的泛化能力，每次迭代训练s和m两种尺寸的模型，分别用于视频直播检测和图像的自动标注。目前我们的模型已经适配国产昇腾和英伟达的算力卡，可以实现模型的自动化训练作业，并针对不同算力芯片进行模型的自动转换和量化。3. 云上标注我们的模型可以对无人机回传的图片和视频进行切分检测和自动标注，针对不同大小的目标和类别可以设置不同的切分尺寸和重叠比例，实现无人机影像的细粒度检测。4. 直播推理我们的AI直播推理业务Pipeline并发运行，使用Python结合C++进行开发，功能模块化，业务运行更高效，可以在RK3588、Jetson系列开发板上进行部署。目前针对松材线虫病检测的场景，已经支持对9种疫木的实时识别。----转自博客：https://bbs.huaweicloud.com/blogs/458003

如何使用 Python 开发 AI 图编排应用本文将介绍使用Python开发一个简单的AI图编排应用，我们的目标是实现AI应用在RK3588上灵活编排和高效部署。首先我们定义的图是由边和节点组成的有向无环图，边代表任务队列，表示数据在节点之间的流动关系，每个节点都是一个计算单元，用于处理特定的任务。之后我们可以定义一组的处理特定任务的函数节点也称为计算单元，例如：read_frame、model_infer、kf_tracker、draw_boxes、push_frame、redis_push，分别用于读取视频、模型检测、目标跟踪、图像绘制、视频输出以及结果推送。每个节点可以有一个输入和多个输出，数据在节点之间是单向流动的，节点之间通过边进行连接，每个节点通过队列消费和传递数据。代码地址：https://github.com/HouYanSong/modelbox-rk3588一. 计算节点的实现我们在Json文件中定义每一个节点的的数据结构并使用Python进行代码实现：读流计算单元有4个参数：pull_video_url、height、width、fps，分别代表视频地址、视频高度和宽度以及读取帧率，它仅作为生产者，产生的数据可以输出到多个队列。"read_frame": { "config": { "pull_video_url": { "type": "str", "required": true, "default": null, "desc": "pull video url", "source": "mp4|flv|rtmp|rtsp" }, "height": { "type": "int", "required": true, "default": null, "max": 1440, "min": 720, "desc": "video height" }, "width": { "type": "int", "required": true, "default": null, "max": 1920, "min": 960, "desc": "video width" }, "fps": { "type": "int", "required": true, "default": null, "max": 15, "min": 5, "desc": "frame rate" } }, "multi_output": [] } 函数代码的实现如下，我们可以对视频文件或者视频流使用ffmpeg进行硬件解码，并将解码后的帧数据写入到队列中，用于后续任务节点的计算。def read_frame(share_dict, flowunit_data, queue_dict, data): pull_video_url = flowunit_data["config"]["pull_video_url"] height = flowunit_data["config"]["height"] width = flowunit_data["config"]["width"] fps = flowunit_data["config"]["fps"] ffmpeg_cmd = [ 'ffmpeg', '-c:v', 'h264_rkmpp', '-i', pull_video_url, '-r', f'{fps}', '-loglevel', 'info', '-s', f'{width}x{height}', '-an', '-f', 'rawvideo', '-pix_fmt', 'bgr24', 'pipe:' ] ffmpeg_process = sp.Popen(ffmpeg_cmd, stdout=sp.PIPE, stderr=sp.DEVNULL, bufsize=10**7) index = 0 while True: index += 1 raw_frame = ffmpeg_process.stdout.read(width * height * 3) if not raw_frame: break else: frame = np.frombuffer(raw_frame, dtype=np.uint8).reshape((height, width, -1)) data["frame"] = frame for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) # 读取结束，图片数据置为None data["frame"] = None for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) ffmpeg_process.stdout.close() ffmpeg_process.terminate() 推理计算单元的函数定义如下，它有一个输入和多个输出，我们可以指定模型和配置文件路径以及单次图像推理的批次大小等参数。"model_infer": { "config": { "model_file": { "type": "str", "required": true, "default": null, "desc": "model file path, rk3588 mostly ends with .rknn" }, "model_info": { "type": "str", "required": true, "default": null, "desc": "model info file path, mostly use json file" }, "batch_size": { "type": "int", "required": true, "default": null, "max": 8, "min": 1, "desc": "batch size" } }, "single_input": null, "multi_output": [] } 对应的函数实现如下，这里我们通过创建线程池的方式对图像进行批量推理，BatchSize的大小代表创建线程池的数量，将一个批次的推理结果写入到输出队列中，输出队列不唯一，可以为空或有多个输出队列。def model_infer(share_dict, flowunit_data, queue_dict, data): model_file = flowunit_data["config"]["model_file"] model_info = flowunit_data["config"]["model_info"] batch_size = flowunit_data["config"]["batch_size"] rknn_lite_list = [] for i in range(batch_size): rknn_lite = RKNNLite() rknn_lite.load_rknn(model_file) rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) rknn_lite_list.append(rknn_lite) with open(model_info, "r") as f: model_info = json.load(f) labels = [] for label in list(model_info["model_classes"].values()): labels.append(label) IMG_SIZE = model_info["input_shape"][0][-2:] OBJ_THRESH = model_info["conf_threshold"] NMS_THRESH = model_info["nms_threshold"] exist = False index = 0 while True: index += 1 image_batch = [] if flowunit_data["single_input"] is not None: for i in range(batch_size): data = queue_dict[flowunit_data["single_input"]].get() # 图片数据为None就退出循环 if data["frame"] is None: exist = True break image_batch.append(data) else: break with ThreadPoolExecutor(max_workers=batch_size) as executor: results = list(executor.map(infer_single_image, [(data["frame"], rknn_lite_list[i % batch_size], IMG_SIZE, OBJ_THRESH, NMS_THRESH) for i, data in enumerate(image_batch)])) for i, (boxes, classes, scores) in enumerate(results): classes = [labels[class_id] for class_id in classes] data = image_batch[i] if data.get("boxes") is None: data["boxes"] = boxes data["classes"] = classes data["scores"] = scores else: data["boxes"].extend(boxes) data["classes"].extend(classes) data["scores"].extend(scores) for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) if exist: break # 读取结束，图片数据置为None data["frame"] = None for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) for rknn_lite in rknn_lite_list: rknn_lite.release() 跟踪功能单元的可以对推理结果添加跟踪ID，如果没有推理结果，则直接返回原始数据，其定义如下："kf_tracker": { "config": {}, "single_input": null, "multi_output": [] } 对应的函数代码实现如下：def kf_tracker(share_dict, flowunit_data, queue_dict, data): tracker = CentroidKF_Tracker(max_lost=30) index = 0 while True: index += 1 if flowunit_data["single_input"] is not None: data = queue_dict[flowunit_data["single_input"]].get() else: break # 图片数据为None就退出循环 if data["frame"] is None: break boxes, classes, scores = data.get("boxes"), data.get("classes"), data.get("scores") boxes = np.array(boxes) classes = np.array(classes) scores = np.array(scores) boxes[:, 2] = boxes[:, 2] - boxes[:, 0] boxes[:, 3] = boxes[:, 3] - boxes[:, 1] results = tracker.update(boxes, scores, classes) boxes = [] classes = [] scores = [] tracks = [] for result in results: frame_num, id, bb_left, bb_top, bb_width, bb_height, confidence, x, y, z, class_id = result boxes.append([bb_left, bb_top, bb_left + bb_width, bb_top + bb_height]) classes.append(class_id) scores.append(confidence) tracks.append(id) data["boxes"] = boxes data["classes"] = classes data["scores"] = scores data["tracks"] = tracks for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) # 读取结束，图片数据置为None data["frame"] = None for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) 绘制功能单元可以对检测和跟踪结果进行绘制，如果检测结果或跟踪结果为空，则直接返回原始数据，其定义如下："draw_boxes": { "single_input": null, "config": {}, "multi_output": [] } 代码逻辑如下：def draw_boxes(share_dict, flowunit_data, queue_dict, data): index = 0 while True: index += 1 if flowunit_data["single_input"] is not None: data = queue_dict[flowunit_data["single_input"]].get() else: break # 图片数据为None就退出循环 if data["frame"] is None: break boxes, classes, scores = data.get("boxes"), data.get("classes"), data.get("scores") if boxes is not None: tracks = data.get("tracks") if tracks is not None: for boxe, clss, track in zip(boxes, classes, tracks): cv2.rectangle(data["frame"], (boxe[0], boxe[1]), (boxe[2], boxe[3]), (0, 255, 0), 2) cv2.putText(data["frame"], f"{clss} {track}", (boxe[0], boxe[1] - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2) else: for boxe, clss, conf in zip(boxes, classes, scores): cv2.rectangle(data["frame"], (boxe[0], boxe[1]), (boxe[2], boxe[3]), (0, 255, 0), 2) cv2.putText(data["frame"], f"{clss} {conf * 100:.2f}%", (boxe[0], boxe[1] - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2) for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) # 读取结束，图片数据置为None data["frame"] = None for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) 输出功能单元可以将视频帧编码成视频输到到视频文件或者推流到RTMP服务器，其参数定义如下："push_frame": { "config": { "push_video_url": { "type": "str", "required": true, "default": null, "desc": "push video url", "source": "rtmp|flv|mp4" }, "format": { "type": "str", "required": true, "default": null, "desc": "vodeo format", "source": "flv|mp4" }, "height": { "type": "int", "required": true, "default": null, "max": 1920, "min": 720, "desc": "video height" }, "width": { "type": "int", "required": true, "default": null, "max": 1920, "min": 960, "desc": "video width" }, "fps": { "type": "int", "required": true, "default": null, "max": 15, "min": 5, "desc": "frame rate" } }, "single_input": null } push_video_url参数是推流地址，也可以输出到本地视频文件。format参数指定视频格式，支持flv和mp4。height和width为视频分辨率，fps是输出帧率。它仅作为消费者，具体函数代码实现如下：def push_frame(share_dict, flowunit_data, queue_dict, data): push_video_url = flowunit_data["config"]["push_video_url"] format = flowunit_data["config"]["format"] height = flowunit_data["config"]["height"] width = flowunit_data["config"]["width"] fps = flowunit_data["config"]["fps"] process_stdin = ( ffmpeg .input('pipe:', format='rawvideo', pix_fmt='bgr24', s="{}x{}".format(width, height), framerate=fps) .filter('fps', fps=fps, round='up') .output( push_video_url, vcodec='h264_rkmpp', bitrate='2500k', f=format, g=fps, an=None, timeout='0' ) .overwrite_output() .run_async(cmd=["ffmpeg", "-re"], pipe_stdin=True) ) index = 0 while True: index += 1 if flowunit_data["single_input"] is not None: data = queue_dict[flowunit_data["single_input"]].get() else: break # 图片数据为None就退出循环 if data["frame"] is None: break frame = data["frame"] frame = cv2.resize(frame, (width, height)) process_stdin.stdin.write(frame.tobytes()) process_stdin.stdin.close() process_stdin.terminate() 消息功能单元可以将检测或跟踪结果发送到Redis服务器，具体可以根据实际情况进行调整。"redis_push": { "config": { "task_id": { "type": "str", "required": true, "default": null, "desc": "task id" }, "host": { "type": "str", "required": true, "default": null, "desc": "redis host" }, "port": { "type": "int", "required": true, "default": null, "desc": "redis port" }, "username": { "type": "str", "required": true, "default": null, "desc": "redis username" }, "password": { "type": "str", "required": true, "default": null, "desc": "redis password" }, "db": { "type": "int", "required": true, "default": null, "desc": "redis db" } }, "single_input": null } 同样，它也仅作为消费者，只有一个输入，具体函数代码如下：def redis_push(share_dict, flowunit_data, queue_dict, data): task_id = flowunit_data["config"]["task_id"] host = flowunit_data["config"]["host"] port = flowunit_data["config"]["port"] username = flowunit_data["config"]["username"] password = flowunit_data["config"]["password"] db = flowunit_data["config"]["db"] r = redis.Redis( host = host, port = port, username = username, password = password, db = db, decode_responses = True ) index = 0 while True: index += 1 if flowunit_data["single_input"] is not None: data = queue_dict[flowunit_data["single_input"]].get() else: break # 图片数据为None就退出循环 if data["frame"] is None: break track_objs = [] height, width = data["frame"].shape[:2] boxes, classes, scores, tracks = data.get("boxes"), data.get("classes"), data.get("scores"), data.get("tracks") if boxes is not None: for boxe, clss, conf, track in zip(boxes, classes, scores, tracks): x1 = float(boxe[0] / width) y1 = float(boxe[1] / height) x2 = float(boxe[2] / width) y2 = float(boxe[3] / height) track_obj = { "bbox": [x1, y1, x2, y2], "track_id": int(track), "class_id": 0, "class_name": str(clss) } track_objs.append(track_obj) key = 'vision:track:' + str(task_id) + ':frame:' + str(index) value = json.dumps({"track_result": track_objs}) r.set(key, value) r.expire(key, 2) print(track_objs) r.close() 二、流程图编排定义好节点，我们就可以定义管道也就是“边”将“节点”的输入和输出连接起来，这里我们定义6条边也就是实例化6个队列，在配置文件中声明每条管道的名称以及队列的最大容量。"queue_size": 16, "queue_list": [ "frame_queue", "infer_queue_1", "infer_queue_2", "track_queue", "draw_queue_1", "draw_queue_2" ] 之后就是对每一个节点的参数进行配置，并定义功能单元的输入和输出。"graph_edge": { "读流功能单元": { "read_frame": { "config": { "pull_video_url": "/home/orangepi/workspace/modelbox/data/car.mp4", "height": 720, "width": 1280, "fps": 20 }, "multi_output": [ "frame_queue" ] } }, "推理功能单元": { "model_infer": { "config": { "model_file": "/home/orangepi/workspace/modelbox/model/yolov8n_800x800_int8.rknn", "model_info": "/home/orangepi/workspace/modelbox/model/yolov8n_800x800_int8.json", "batch_size": 8 }, "single_input": "frame_queue", "multi_output": [ "infer_queue_1", "infer_queue_2" ] } }, "跟踪功能单元_2": { "kf_tracker": { "config": {}, "single_input": "infer_queue_2", "multi_output": [ "track_queue" ] } }, "绘图功能单元_1": { "draw_boxes": { "config": {}, "single_input": "infer_queue_1", "multi_output": [ "draw_queue_1" ] } }, "绘图功能单元_2": { "draw_boxes": { "single_input": "track_queue", "config": {}, "multi_output": [ "draw_queue_2" ] } }, "推流功能单元_1": { "push_frame": { "config": { "push_video_url": "/home/orangepi/workspace/modelbox/output/det_result.mp4", "format": "mp4", "height": 720, "width": 1280, "fps": 20 }, "single_input": "draw_queue_1" } }, "推流功能单元_2": { "push_frame": { "config": { "push_video_url": "/home/orangepi/workspace/modelbox/output/track_result.mp4", "format": "mp4", "height": 720, "width": 1280, "fps": 20 }, "single_input": "draw_queue_2" } } } 每个功能单元需要起一个节点名称用于功能单元的创建，每个节点名称保证全局唯一，正如字典中的键值不能重复。之后根据这份图文件编排启动AI应用，Python代码如下：import os import sys import json import argparse sys.path.append(os.path.join(os.path.dirname(__file__), '..')) from etc.flowunit import * from multiprocessing import Process, Queue, Manager if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('graph_path', type=str, nargs='?', default='/home/orangepi/workspace/modelbox/graph/person_car.json') args = parser.parse_args() # 初始化数据 data = {"frame": None} config = {} # 读取流程图 with open(args.graph_path) as f: graph = json.load(f) # 创建队列 queue_dict = {} queue_size = graph["queue_size"] for queue_name in graph["queue_list"]: queue_dict[queue_name] = Queue(maxsize=queue_size) with Manager() as manager: # 创建共享字典 share_dict = manager.dict() # 创建进程 process_list = [] graph_edge = graph["graph_edge"] for process in graph_edge.keys(): p = Process(target=eval(list(graph_edge[process].keys())[0]), args=(share_dict, list(graph_edge[process].values())[0], queue_dict, data,)) process_list.append(p) print("=============Start Process...=============") # 启动进程 for p in process_list: p.start() # 等待进程结束 for p in process_list: p.join() print("==========All Process Finished.===========") 这里我们读取一段测试视频分别将检测结果和跟踪结果保存为两个视频文件输出到output目录下：(python-3.9.15) orangepi@orangepi5plus:~$ python /home/orangepi/workspace/modelbox/graph/graph.py /home/orangepi/workspace/modelbox/graph/person_car.json =============Start Process...============= ffmpeg version 04f5eaa Copyright (c) 2000-2023 the FFmpeg developers built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04) configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga libavutil 58. 29.100 / 58. 29.100 libavcodec 60. 31.102 / 60. 31.102 libavformat 60. 16.100 / 60. 16.100 libavdevice 60. 3.100 / 60. 3.100 libavfilter 9. 12.100 / 9. 12.100 libswscale 7. 5.100 / 7. 5.100 libswresample 4. 12.100 / 4. 12.100 libpostproc 57. 3.100 / 57. 3.100 ffmpeg version 04f5eaa Copyright (c) 2000-2023 the FFmpeg developers built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04) configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga libavutil 58. 29.100 / 58. 29.100 libavcodec 60. 31.102 / 60. 31.102 libavformat 60. 16.100 / 60. 16.100 libavdevice 60. 3.100 / 60. 3.100 libavfilter 9. 12.100 / 9. 12.100 libswscale 7. 5.100 / 7. 5.100 libswresample 4. 12.100 / 4. 12.100 libpostproc 57. 3.100 / 57. 3.100 W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.190] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.191] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.192] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.248] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.338] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.338] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.339] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.384] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.459] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.459] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.460] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.504] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.606] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.606] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.608] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.658] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.761] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.761] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.762] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.814] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.910] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.910] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.912] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.962] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:48.069] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:48.070] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:48.071] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:48.122] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:48.228] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:48.228] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:48.229] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:48.280] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) Input #0, rawvideo, from 'pipe:': Duration: N/A, start: 0.000000, bitrate: 442368 kb/s Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24, 1280x720, 442368 kb/s, 20 tbr, 20 tbn Stream mapping: Stream #0:0 (rawvideo) -> fps:default fps:default -> Stream #0:0 (h264_rkmpp) Output #0, mp4, to '/home/orangepi/workspace/modelbox/output/det_result.mp4': Metadata: encoder : Lavf60.16.100 Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), bgr24(progressive), 1280x720, q=2-31, 2000 kb/s, 20 fps, 10240 tbn Metadata: encoder : Lavc60.31.102 h264_rkmpp Input #0, rawvideo, from 'pipe:': 0kB time=N/A bitrate=N/A speed=N/A Duration: N/A, start: 0.000000, bitrate: 442368 kb/s Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24, 1280x720, 442368 kb/s, 20 tbr, 20 tbn Stream mapping: Stream #0:0 (rawvideo) -> fps:default fps:default -> Stream #0:0 (h264_rkmpp) Output #0, mp4, to '/home/orangepi/workspace/modelbox/output/track_result.mp4': Metadata: encoder : Lavf60.16.100 Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), bgr24(progressive), 1280x720, q=2-31, 2000 kb/s, 20 fps, 10240 tbn Metadata: encoder : Lavc60.31.102 h264_rkmpp [out#0/mp4 @ 0x558f2625e0] video:2330kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.062495% frame= 132 fps= 19 q=-0.0 Lsize= 2331kB time=00:00:06.55 bitrate=2915.8kbits/s speed=0.924x Exiting normally, received signal 15. [out#0/mp4 @ 0x557e4875e0] video:1990kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.072192% frame= 131 fps= 18 q=-0.0 Lsize= 1991kB time=00:00:06.50 bitrate=2509.6kbits/s speed= 0.9x ==========All Process Finished.=========== Exiting normally, received signal 15.应用推理的帧率取决于视频读取的帧率以及耗时最久的功能单元，实测FPS约为20左右，满足AI实时检测的场景。

无人机巡检数据集：空中语义分割由计算机图形与视觉研究所（ICG开发的语义无人机数据集，旨在推动城市场景的语义理解研究，提升自主无人机飞行与着陆的安全性。以下从数量、类别、分布及分辨率四个核心维度展开说明：一、数据数量与划分该数据集包含600张高分辨率图像，按用途分为训练集和测试集两部分：训练集：400张图像，公开可获取，包含完整的标注信息，支持模型训练与算法验证。测试集：200张图像，为私有数据，主要用于评估模型的泛化能力，确保研究结果的客观性。此外，数据集还提供了丰富的辅助数据，包括1Hz采集的高分辨率图像序列、5Hz的鱼眼立体图像（同步配有IMU测量数据）、1Hz的热成像图像，以及3栋房屋的地面控制点和全站仪获取的3D真值数据，进一步扩展了其应用场景。二、语义类别与标注数据集针对语义分割任务定义了20个核心类别，覆盖城市场景中的典型元素，具体分类如下：自然元素：树（tree）、草（gras）、其他植被（other vegetation）、泥土（dirt）、 gravel、岩石（rocks）、水（water）；人工构造：铺装区域（paved area）、泳池（pool）、屋顶（roof）、墙（wall）、栅栏（fence）、栅栏柱（fence-pole）、窗户（window）、门（door）、障碍物（obstacle）；动态目标：人（person）、狗（dog）、汽车（car）、自行车（bicycle）。标注精度达到像素级，确保语义分割任务的准确性；同时，针对人物检测任务，提供了训练集和测试集的边界框（bounding box）标注，支持多任务研究。三、数据分布特点数据集的图像均通过无人机从天底视角（鸟瞰视角） 采集，覆盖超过20栋房屋的城市区域，拍摄高度在地面以上5至30米之间，确保场景的真实性与多样性。从分布来看：场景覆盖：包含居民区内的建筑、植被、道路、休闲区域（如泳池）等，兼顾自然与人工环境的混合场景；目标密度：图像中包含不同数量的动态目标（人、动物、车辆）和静态结构，适合测试算法在复杂目标交互场景下的表现；辅助数据分布：热成像、立体图像等辅助数据与主图像时空同步，可用于多模态融合研究，提升模型对环境的感知能力。四、分辨率与数据格式数据集的核心图像采用高分辨率相机采集，单张图像尺寸为6000×4000像素（2400万像素），确保细节信息的完整性，满足精细语义分割的需求。训练集提供多种格式的标注文件，包括：Python pickle格式的边界框数据；可选的XML格式边界框标注；可选的掩码图像（mask images），便于不同算法框架的直接使用。五、数据集下载地址# 数据集地址 https://developer.huaweicloud.com/develop/aigallery/dataset/detail?id=0efe9613-5248-4a43-925e-4d6d377b6996综上，该数据集凭借大尺寸、多类别、高精度的特点，为无人机视觉、语义分割、目标检测等领域的研究提供了高质量的基准数据，其丰富的辅助信息也为多模态感知与3D重建等任务奠定了基础。

华为云开发者空间☁️昇腾NPU实现AI工业质检本案例将在华为云开发者空间工作台⚙️AI Notebook 中使用免费的🎉 昇腾 NPU 910B ✨完成YOLO11模型训练，利用SAHI切片辅助超推理框架实现PCB缺陷检测。1. 下载模型和数据集📦首先在Notebook的代码块中粘贴并运行下面的代码，下载解压本案例所需的训练数据和模型文件：import os import zipfile if not os.path.exists('yolo11_train_ascend.zip'): os.system('wget -q https://orangepi-ascend.obs.cn-north-4.myhuaweicloud.com/yolo11_train_ascend.zip') if not os.path.exists('yolo11_train_ascend'): zip_file = zipfile.ZipFile('yolo11_train_ascend.zip') zip_file.extractall() zip_file.close() 2. 安装依赖包🛠️安装YOLO11所需的依赖包以及SAHI库，构建项目的运行环境：!pip install ultralytics==8.3.160 ultralytics-thop==2.0.14 sahi==0.11.26 numpy==1.26.4 Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://mirrors.huaweicloud.com/repository/pypi/simple Collecting ultralytics==8.3.160 Downloading https://mirrors.huaweicloud.com/repository/pypi/packages/7b/8d/924524ff26c0ed0ba43b90cc598887e2b06f3bf00dd51a505a754ecb138d/ultralytics-8.3.160-py3-none-any.whl (1.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 9.8 MB/s eta 0:00:00 Collecting ultralytics-thop==2.0.14 Downloading https://mirrors.huaweicloud.com/repository/pypi/packages/a6/10/251f036b4c5d77249f9a119cc89dafe8745dc1ad1f1a5f06b6a3988ca454/ultralytics_thop-2.0.14-py3-none-any.whl (26 kB) Collecting sahi==0.11.26 Downloading https://mirrors.huaweicloud.com/repository/pypi/packages/5e/8a/9782c8088af52e6f41fee59c77b5117783c0d6eafde45c96ca3912ec197f/sahi-0.11.26-py3-none-any.whl (115 kB) Collecting numpy==1.26.4 Downloading https://mirrors.huaweicloud.com/repository/pypi/packages/fc/a5/4beee6488160798683eed5bdb7eead455892c3b4e1f78d79d8d3f3b084ac/numpy-1.26.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (14.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 14.2/14.2 MB 42.9 MB/s eta 0:00:00 0:00:01 Requirement already satisfied: matplotlib>=3.3.0 in /home/service/.local/lib/python3.10/site-packages (from ultralytics==8.3.160) (3.10.0) Requirement already satisfied: opencv-python>=4.6.0 in /home/service/.local/lib/python3.10/site-packages (from ultralytics==8.3.160) (4.10.0.84) Requirement already satisfied: pillow>=7.1.2 in /usr/local/python3.10/lib/python3.10/site-packages (from ultralytics==8.3.160) (11.0.0) Requirement already satisfied: pyyaml>=5.3.1 in /usr/local/python3.10/lib/python3.10/site-packages (from ultralytics==8.3.160) (6.0.2) Requirement already satisfied: requests>=2.23.0 in /home/service/.local/lib/python3.10/site-packages (from ultralytics==8.3.160) (2.32.3) Requirement already satisfied: scipy>=1.4.1 in /usr/local/python3.10/lib/python3.10/site-packages (from ultralytics==8.3.160) (1.14.1) Requirement already satisfied: torch>=1.8.0 in /usr/local/python3.10/lib/python3.10/site-packages (from ultralytics==8.3.160) (2.1.0) Requirement already satisfied: torchvision>=0.9.0 in /home/service/.local/lib/python3.10/site-packages (from ultralytics==8.3.160) (0.16.0) Requirement already satisfied: tqdm>=4.64.0 in /usr/local/python3.10/lib/python3.10/site-packages (from ultralytics==8.3.160) (4.67.1) Requirement already satisfied: psutil in /home/service/.local/lib/python3.10/site-packages (from ultralytics==8.3.160) (5.9.8) Requirement already satisfied: py-cpuinfo in /home/service/.local/lib/python3.10/site-packages (from ultralytics==8.3.160) (9.0.0) Requirement already satisfied: pandas>=1.1.4 in /usr/local/python3.10/lib/python3.10/site-packages (from ultralytics==8.3.160) (2.2.3) Requirement already satisfied: click in /usr/local/python3.10/lib/python3.10/site-packages (from sahi==0.11.26) (8.1.8) Collecting fire (from sahi==0.11.26) Downloading https://mirrors.huaweicloud.com/repository/pypi/packages/6b/b6/82c7e601d6d3c3278c40b7bd35e17e82aa227f050aa9f66cb7b7fce29471/fire-0.7.0.tar.gz (87 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Collecting pybboxes==0.1.6 (from sahi==0.11.26) Downloading https://mirrors.huaweicloud.com/repository/pypi/packages/3c/3f/46f6613b41a3c2b4f7af3b526035771ca5bb12d6fdf3b23145899f785e36/pybboxes-0.1.6-py3-none-any.whl (24 kB) Collecting shapely>=2.0.0 (from sahi==0.11.26) Downloading https://mirrors.huaweicloud.com/repository/pypi/packages/29/51/0b158a261df94e33505eadfe737db9531f346dfa60850945ad25fd4162f1/shapely-2.1.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (2.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.9/2.9 MB 22.0 MB/s eta 0:00:00 Collecting terminaltables (from sahi==0.11.26) Downloading https://mirrors.huaweicloud.com/repository/pypi/packages/c4/fb/ea621e0a19733e01fe4005d46087d383693c0f4a8f824b47d8d4122c87e0/terminaltables-3.1.10-py2.py3-none-any.whl (15 kB) Requirement already satisfied: contourpy>=1.0.1 in /home/service/.local/lib/python3.10/site-packages (from matplotlib>=3.3.0->ultralytics==8.3.160) (1.3.1) Requirement already satisfied: cycler>=0.10 in /home/service/.local/lib/python3.10/site-packages (from matplotlib>=3.3.0->ultralytics==8.3.160) (0.12.1) Requirement already satisfied: fonttools>=4.22.0 in /home/service/.local/lib/python3.10/site-packages (from matplotlib>=3.3.0->ultralytics==8.3.160) (4.55.3) Requirement already satisfied: kiwisolver>=1.3.1 in /home/service/.local/lib/python3.10/site-packages (from matplotlib>=3.3.0->ultralytics==8.3.160) (1.4.8) Requirement already satisfied: packaging>=20.0 in /usr/local/python3.10/lib/python3.10/site-packages (from matplotlib>=3.3.0->ultralytics==8.3.160) (24.2) Requirement already satisfied: pyparsing>=2.3.1 in /home/service/.local/lib/python3.10/site-packages (from matplotlib>=3.3.0->ultralytics==8.3.160) (3.2.0) Requirement already satisfied: python-dateutil>=2.7 in /usr/local/python3.10/lib/python3.10/site-packages (from matplotlib>=3.3.0->ultralytics==8.3.160) (2.9.0.post0) Requirement already satisfied: pytz>=2020.1 in /usr/local/python3.10/lib/python3.10/site-packages (from pandas>=1.1.4->ultralytics==8.3.160) (2024.2) Requirement already satisfied: tzdata>=2022.7 in /usr/local/python3.10/lib/python3.10/site-packages (from pandas>=1.1.4->ultralytics==8.3.160) (2024.2) Requirement already satisfied: charset-normalizer<4,>=2 in /home/service/.local/lib/python3.10/site-packages (from requests>=2.23.0->ultralytics==8.3.160) (3.4.1) Requirement already satisfied: idna<4,>=2.5 in /usr/local/python3.10/lib/python3.10/site-packages (from requests>=2.23.0->ultralytics==8.3.160) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /home/service/.local/lib/python3.10/site-packages (from requests>=2.23.0->ultralytics==8.3.160) (2.3.0) Requirement already satisfied: certifi>=2017.4.17 in /home/service/.local/lib/python3.10/site-packages (from requests>=2.23.0->ultralytics==8.3.160) (2024.12.14) Requirement already satisfied: filelock in /usr/local/python3.10/lib/python3.10/site-packages (from torch>=1.8.0->ultralytics==8.3.160) (3.16.1) Requirement already satisfied: typing-extensions in /usr/local/python3.10/lib/python3.10/site-packages (from torch>=1.8.0->ultralytics==8.3.160) (4.12.2) Requirement already satisfied: sympy in /usr/local/python3.10/lib/python3.10/site-packages (from torch>=1.8.0->ultralytics==8.3.160) (1.13.3) Requirement already satisfied: networkx in /usr/local/python3.10/lib/python3.10/site-packages (from torch>=1.8.0->ultralytics==8.3.160) (3.4.2) Requirement already satisfied: jinja2 in /usr/local/python3.10/lib/python3.10/site-packages (from torch>=1.8.0->ultralytics==8.3.160) (3.1.5) Requirement already satisfied: fsspec in /home/service/.local/lib/python3.10/site-packages (from torch>=1.8.0->ultralytics==8.3.160) (2024.9.0) Collecting termcolor (from fire->sahi==0.11.26) Downloading https://mirrors.huaweicloud.com/repository/pypi/packages/4f/bd/de8d508070629b6d84a30d01d57e4a65c69aa7f5abe7560b8fad3b50ea59/termcolor-3.1.0-py3-none-any.whl (7.7 kB) Requirement already satisfied: six>=1.5 in /usr/local/python3.10/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib>=3.3.0->ultralytics==8.3.160) (1.16.0) Requirement already satisfied: MarkupSafe>=2.0 in /home/service/.local/lib/python3.10/site-packages (from jinja2->torch>=1.8.0->ultralytics==8.3.160) (3.0.2) Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/python3.10/lib/python3.10/site-packages (from sympy->torch>=1.8.0->ultralytics==8.3.160) (1.3.0) Building wheels for collected packages: fire Building wheel for fire (pyproject.toml) ... done Created wheel for fire: filename=fire-0.7.0-py3-none-any.whl size=114330 sha256=a1f27d511635da524f8f51fa2d35ae22862e400cec55285acbc05ced6ef91371 Stored in directory: /home/service/.cache/pip/wheels/9b/dc/c7/06491fe82713723ab64494dbcfd521bdbe80cf26b5fcb5f564 Successfully built fire Installing collected packages: terminaltables, termcolor, numpy, shapely, pybboxes, fire, ultralytics-thop, sahi, ultralytics Attempting uninstall: numpy Found existing installation: numpy 1.24.4 Uninstalling numpy-1.24.4: Successfully uninstalled numpy-1.24.4 WARNING: The script f2py is installed in '/home/service/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The script sahi is installed in '/home/service/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. WARNING: The scripts ultralytics and yolo are installed in '/home/service/.local/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. gradio 5.9.1 requires markupsafe~=2.0, but you have markupsafe 3.0.2 which is incompatible. openmind 0.9.1 requires datasets<=2.21.0,>=2.18.0, but you have datasets 3.2.0 which is incompatible. openmind 0.9.1 requires openmind-hub==0.9.0, but you have openmind-hub 0.9.1 which is incompatible. openmind-datasets 0.7.1 requires datasets==2.18.0, but you have datasets 3.2.0 which is incompatible. openmind-evaluate 0.7.0 requires datasets==2.18.0, but you have datasets 3.2.0 which is incompatible. Successfully installed fire-0.7.0 numpy-1.26.4 pybboxes-0.1.6 sahi-0.11.26 shapely-2.1.1 termcolor-3.1.0 terminaltables-3.1.10 ultralytics-8.3.160 ultralytics-thop-2.0.14 [notice] A new release of pip is available: 24.3.1 -> 25.1.1 [notice] To update, run: pip install --upgrade pip3. 修改配置文件📝我们在配置文件中指定数据集路径和类别等信息，用于后续模型的训练：%%writefile yolo11_train_ascend/pcb.yaml # Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..] path: /opt/huawei/edu-apaas/src/init/yolo11_train_ascend/pcb_sliced # dataset root dir (absolute path) train: images/train # train images (relative to 'path') val: images/val # val images (relative to 'path') test: # test images (optional) # Classes，类别 names: 0: mouse_bite 1: open_circuit 2: short 3: spur 4: spurious_copperWriting yolo11_train_ascend/pcb.yaml4. 下载 Arial.ttf 字体🖋️为了避免影响训练进展，可以先提前下载字体文件并拷贝到 /home/service/.config/Ultralytics 路径下。!wget https://orangepi-ascend.obs.cn-north-4.myhuaweicloud.com/Arial.ttf !mkdir -p /home/service/.config/Ultralytics !cp Arial.ttf /home/service/.config/Ultralytics/Arial.ttf--2025-06-28 05:55:59-- https://pcb-sahi-public.obs.cn-southwest-2.myhuaweicloud.com/Arial.ttf Resolving pcb-sahi-public.obs.cn-southwest-2.myhuaweicloud.com (pcb-sahi-public.obs.cn-southwest-2.myhuaweicloud.com)... 100.125.6.3, 100.125.7.3, 100.125.6.131 Connecting to pcb-sahi-public.obs.cn-southwest-2.myhuaweicloud.com (pcb-sahi-public.obs.cn-southwest-2.myhuaweicloud.com)|100.125.6.3|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 773236 (755K) [application/x-font-ttf] Saving to: 'Arial.ttf' Arial.ttf 100%[===================>] 755.11K --.-KB/s in 0.004s 2025-06-28 05:55:59 (188 MB/s) - 'Arial.ttf' saved [773236/773236] 5. 模型训练🧠🔥我们使用yolo11n.pt预训练模型，利用昇腾NPU进行模型加速，设置模型的训练次数为10轮、图像的大小为640x640、开启8个数据加载线程每次送入模型32张图像进行迭代优化。%cd yolo11_train_ascend import torch import torch_npu from torch_npu.contrib import transfer_to_npu from ultralytics import YOLO # Load a model model = YOLO('yolo11n.pt') # load a pretrained model (recommended for training) # Train the model results = model.train(data='pcb.yaml', epochs=10, imgsz=640, workers=8, batch=32) %cd .. /home/service/.local/lib/python3.10/site-packages/IPython/core/magics/osm.py:417: UserWarning: This is now an optional IPython functionality, setting dhist requires you to install the `pickleshare` library. self.shell.db['dhist'] = compress_dhist(dhist)[-100:] /opt/huawei/edu-apaas/src/init/yolo11_train_ascend /home/service/.local/lib/python3.10/site-packages/torch_npu/utils/path_manager.py:82: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/latest owner does not match the current user. warnings.warn(f"Warning: The {path} owner does not match the current user.") /home/service/.local/lib/python3.10/site-packages/torch_npu/utils/path_manager.py:82: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/8.0.RC3/aarch64-linux/ascend_toolkit_install.info owner does not match the current user. warnings.warn(f"Warning: The {path} owner does not match the current user.") /home/service/.local/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:301: ImportWarning: ************************************************************************************************************* The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now.. The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now.. The backend in torch.distributed.init_process_group set to hccl now.. The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now.. The device parameters have been replaced with npu in the function below: torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty ************************************************************************************************************* warnings.warn(msg, ImportWarning) /home/service/.local/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:260: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu. warnings.warn(msg, RuntimeWarning) Creating new Ultralytics Settings v0.0.6 file ✅ View Ultralytics Settings with 'yolo settings' or at '/home/service/.config/Ultralytics/settings.json' Update Settings with 'yolo settings key=value', i.e. 'yolo settings runs_dir=path/to/dir'. For help see https://docs.ultralytics.com/quickstart/#ultralytics-settings. [W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator()) Ultralytics 8.3.160 🚀 Python-3.10.15 torch-2.1.0 CUDA:0 (Ascend910B3, 62432MiB) engine/trainer: agnostic_nms=False, amp=True, augment=False, auto_augment=randaugment, batch=32, bgr=0.0, box=7.5, cache=False, cfg=None, classes=None, close_mosaic=10, cls=0.5, conf=None, copy_paste=0.0, copy_paste_mode=flip, cos_lr=False, cutmix=0.0, data=pcb.yaml, degrees=0.0, deterministic=True, device=None, dfl=1.5, dnn=False, dropout=0.0, dynamic=False, embed=None, epochs=10, erasing=0.4, exist_ok=False, fliplr=0.5, flipud=0.0, format=torchscript, fraction=1.0, freeze=None, half=False, hsv_h=0.015, hsv_s=0.7, hsv_v=0.4, imgsz=640, int8=False, iou=0.7, keras=False, kobj=1.0, line_width=None, lr0=0.01, lrf=0.01, mask_ratio=4, max_det=300, mixup=0.0, mode=train, model=yolo11n.pt, momentum=0.937, mosaic=1.0, multi_scale=False, name=train, nbs=64, nms=False, opset=None, optimize=False, optimizer=auto, overlap_mask=True, patience=100, perspective=0.0, plots=True, pose=12.0, pretrained=True, profile=False, project=None, rect=False, resume=False, retina_masks=False, save=True, save_conf=False, save_crop=False, save_dir=runs/detect/train, save_frames=False, save_json=False, save_period=-1, save_txt=False, scale=0.5, seed=0, shear=0.0, show=False, show_boxes=True, show_conf=True, show_labels=True, simplify=True, single_cls=False, source=None, split=val, stream_buffer=False, task=detect, time=None, tracker=botsort.yaml, translate=0.1, val=True, verbose=True, vid_stride=1, visualize=False, warmup_bias_lr=0.1, warmup_epochs=3.0, warmup_momentum=0.8, weight_decay=0.0005, workers=8, workspace=None Overriding model.yaml nc=80 with nc=5 from n params module arguments 0 -1 1 464 ultralytics.nn.modules.conv.Conv [3, 16, 3, 2] 1 -1 1 4672 ultralytics.nn.modules.conv.Conv [16, 32, 3, 2] 2 -1 1 6640 ultralytics.nn.modules.block.C3k2 [32, 64, 1, False, 0.25] 3 -1 1 36992 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2] 4 -1 1 26080 ultralytics.nn.modules.block.C3k2 [64, 128, 1, False, 0.25] 5 -1 1 147712 ultralytics.nn.modules.conv.Conv [128, 128, 3, 2] 6 -1 1 87040 ultralytics.nn.modules.block.C3k2 [128, 128, 1, True] 7 -1 1 295424 ultralytics.nn.modules.conv.Conv [128, 256, 3, 2] 8 -1 1 346112 ultralytics.nn.modules.block.C3k2 [256, 256, 1, True] 9 -1 1 164608 ultralytics.nn.modules.block.SPPF [256, 256, 5] 10 -1 1 249728 ultralytics.nn.modules.block.C2PSA [256, 256, 1] 11 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest'] 12 [-1, 6] 1 0 ultralytics.nn.modules.conv.Concat [1] 13 -1 1 111296 ultralytics.nn.modules.block.C3k2 [384, 128, 1, False] 14 -1 1 0 torch.nn.modules.upsampling.Upsample [None, 2, 'nearest'] 15 [-1, 4] 1 0 ultralytics.nn.modules.conv.Concat [1] 16 -1 1 32096 ultralytics.nn.modules.block.C3k2 [256, 64, 1, False] 17 -1 1 36992 ultralytics.nn.modules.conv.Conv [64, 64, 3, 2] 18 [-1, 13] 1 0 ultralytics.nn.modules.conv.Concat [1] 19 -1 1 86720 ultralytics.nn.modules.block.C3k2 [192, 128, 1, False] 20 -1 1 147712 ultralytics.nn.modules.conv.Conv [128, 128, 3, 2] 21 [-1, 10] 1 0 ultralytics.nn.modules.conv.Concat [1] 22 -1 1 378880 ultralytics.nn.modules.block.C3k2 [384, 256, 1, True] 23 [16, 19, 22] 1 431647 ultralytics.nn.modules.head.Detect [5, [64, 128, 256]] /home/service/.local/lib/python3.10/site-packages/torch_npu/utils/storage.py:38: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage() if self.device.type != 'cpu': YOLO11n summary: 181 layers, 2,590,815 parameters, 2,590,799 gradients, 6.4 GFLOPs Transferred 448/499 items from pretrained weights Freezing layer 'model.23.dfl.conv.weight' AMP: running Automatic Mixed Precision (AMP) checks... [W compiler_depend.ts:51] Warning: CAUTION: The operator 'torchvision::nms' is not currently supported on the NPU backend and will fall back to run on the CPU. This may have performance implications. (function npu_cpu_fallback) AMP: checks passed ✅ train: Fast image access ✅ (ping: 0.0±0.0 ms, read: 620.7±42.3 MB/s, size: 454.2 KB) train: Scanning /opt/huawei/edu-apaas/src/init/yolo11_train_ascend/pcb_sliced/labels/train... 4646 images, 0 backgrounds, 0 corrupt: 100%|██████████| 4646/4646 [00:05<00:00, 848.20it/s] train: New cache created: /opt/huawei/edu-apaas/src/init/yolo11_train_ascend/pcb_sliced/labels/train.cache val: Fast image access ✅ (ping: 0.0±0.0 ms, read: 471.4±135.5 MB/s, size: 448.2 KB) val: Scanning /opt/huawei/edu-apaas/src/init/yolo11_train_ascend/pcb_sliced/labels/val... 422 images, 0 backgrounds, 0 corrupt: 100%|██████████| 422/422 [00:00<00:00, 520.44it/s] val: New cache created: /opt/huawei/edu-apaas/src/init/yolo11_train_ascend/pcb_sliced/labels/val.cache Plotting labels to runs/detect/train/labels.jpg... optimizer: 'optimizer=auto' found, ignoring 'lr0=0.01' and 'momentum=0.937' and determining best 'optimizer', 'lr0' and 'momentum' automatically... optimizer: AdamW(lr=0.001111, momentum=0.9) with parameter groups 81 weight(decay=0.0), 88 weight(decay=0.0005), 87 bias(decay=0.0) Image sizes 640 train, 640 val Using 8 dataloader workers Logging results to runs/detect/train Starting training for 10 epochs... Closing dataloader mosaic Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 0%| | 0/146 [00:00<?, ?it/s] . /home/service/.local/lib/python3.10/site-packages/ultralytics/utils/tal.py:274: UserWarning: AutoNonVariableTypeMode is deprecated and will be removed in 1.10 release. For kernel implementations please use AutoDispatchBelowADInplaceOrView instead, If you are looking for a user facing API to enable running your inference-only workload, please use c10::InferenceMode. Using AutoDispatchBelowADInplaceOrView in user code is under risk of producing silent wrong result in some edge cases. See Note [AutoDispatchBelowAutograd] for more details. (Triggered internally at build/CMakeFiles/torch_npu.dir/compiler_depend.ts:74.) target_scores = torch.where(fg_scores_mask > 0, target_scores, 0) [W compiler_depend.ts:103] Warning: Non finite check and unscale on NPU device! (function operator()) 1/10 7.77G 2.238 5.333 1.761 8 640: 100%|██████████| 146/146 [01:31<00:00, 1.60it/s] Class Images Instances Box(P R mAP50 mAP50-95): 0%| | 0/7 [00:00<?, ?it/s] ..... Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:58<00:00, 8.39s/it] all 422 604 0.39 0.0656 0.0888 0.023 Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 2/10 8.2G 1.876 2.724 1.462 8 640: 100%|██████████| 146/146 [01:16<00:00, 1.92it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:04<00:00, 1.58it/s] all 422 604 0.451 0.238 0.214 0.0639 Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 3/10 8.2G 1.825 1.912 1.445 8 640: 100%|██████████| 146/146 [01:12<00:00, 2.00it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:04<00:00, 1.64it/s] all 422 604 0.339 0.291 0.244 0.0742 Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 4/10 8.2G 1.748 1.571 1.398 4 640: 100%|██████████| 146/146 [01:12<00:00, 2.02it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:04<00:00, 1.58it/s] all 422 604 0.409 0.361 0.335 0.117 Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 5/10 8.2G 1.703 1.343 1.372 6 640: 100%|██████████| 146/146 [01:11<00:00, 2.05it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:04<00:00, 1.66it/s] all 422 604 0.442 0.34 0.321 0.118 Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 6/10 8.2G 1.673 1.26 1.343 5 640: 100%|██████████| 146/146 [01:11<00:00, 2.03it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:04<00:00, 1.62it/s] all 422 604 0.605 0.49 0.53 0.224 Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 7/10 8.2G 1.614 1.145 1.316 6 640: 100%|██████████| 146/146 [01:12<00:00, 2.00it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:04<00:00, 1.58it/s] all 422 604 0.595 0.542 0.525 0.206 Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 8/10 8.2G 1.578 1.067 1.294 7 640: 100%|██████████| 146/146 [01:11<00:00, 2.03it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:04<00:00, 1.73it/s] all 422 604 0.754 0.629 0.685 0.307 Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 9/10 8.21G 1.551 1.009 1.275 8 640: 100%|██████████| 146/146 [01:11<00:00, 2.04it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:04<00:00, 1.57it/s] all 422 604 0.782 0.618 0.703 0.315 Epoch GPU_mem box_loss cls_loss dfl_loss Instances Size 10/10 8.21G 1.5 0.9621 1.255 6 640: 100%|██████████| 146/146 [01:12<00:00, 2.02it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:03<00:00, 1.83it/s] all 422 604 0.8 0.661 0.732 0.354 10 epochs completed in 0.236 hours. Optimizer stripped from runs/detect/train/weights/last.pt, 5.5MB Optimizer stripped from runs/detect/train/weights/best.pt, 5.5MB Validating runs/detect/train/weights/best.pt... Ultralytics 8.3.160 🚀 Python-3.10.15 torch-2.1.0 CUDA:0 (Ascend910B3, 62432MiB) YOLO11n summary (fused): 100 layers, 2,583,127 parameters, 0 gradients, 6.3 GFLOPs ... Class Images Instances Box(P R mAP50 mAP50-95): 0%| | 0/7 [00:00<?, ?it/s] . Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 7/7 [00:07<00:00, 1.14s/it] all 422 604 0.799 0.663 0.732 0.355 mouse_bite 107 169 0.806 0.785 0.829 0.4 open_circuit 73 101 0.656 0.471 0.492 0.219 short 69 87 0.889 0.54 0.701 0.314 spur 95 134 0.864 0.714 0.76 0.342 spurious_copper 95 113 0.782 0.805 0.88 0.5 Speed: 0.1ms preprocess, 8.3ms inference, 0.0ms loss, 2.5ms postprocess per image Results saved to runs/detect/train /opt/huawei/edu-apaas/src/init /home/service/.local/lib/python3.10/site-packages/IPython/core/magics/osm.py:417: UserWarning: This is now an optional IPython functionality, setting dhist requires you to install the `pickleshare` library. self.shell.db['dhist'] = compress_dhist(dhist)[-100:] 模型训练好后，可以在runs/detect/train目录下查看训练结果，例如损失函数的变化曲线、mAP等评价指标📈💪。6. 图像切分检测✂️🔍最后我们利用SAHI框架对高清PCB图像进行切片推理，从而更精准地检测出PCB的瑕疵类别。import torch import torch_npu from torch_npu.contrib import transfer_to_npu from sahi.predict import get_sliced_prediction from sahi import AutoDetectionModel from PIL import Image detection_model = AutoDetectionModel.from_pretrained( model_type = 'ultralytics', model_path = "yolo11_train_ascend/runs/detect/train/weights/best.pt", confidence_threshold = 0.4, device = "cuda:0" ) 这里我们使用滑窗检测🔍的技术，将原始图像切分成640x640大小的子图🖼️，同时设置一定的重叠度，再分别预测每张子图，最后将所有的检测结果进行合并处理🛠️。image_path = "https://orangepi-ascend.obs.cn-north-4.myhuaweicloud.com/001.bmp" result = get_sliced_prediction( image_path, detection_model, slice_height = 640, slice_width = 640, overlap_height_ratio = 0.1, overlap_width_ratio = 0.1, perform_standard_pred = False, postprocess_class_agnostic = True, postprocess_match_threshold = 0.1, ) result.export_visuals(export_dir="output/", file_name="sliced_result") Image.open("output/sliced_result.png") Performing prediction on 24 slices.可以看到，模型准确无误的预测出PCB缺陷的位置、类别和置信度😄7. 小结📌本案例借助华为云开发者空间💡昇腾910B NPU完成YOLO11模型训练与PCB缺陷检测，并且结合SAHI实现高效切片推理🚀，华为云开发者空间💻AI Notebook开箱即用，大家快来体验吧！🤗 ----转自博客：https://bbs.huaweicloud.com/blogs/455280

1      YOLO介绍YOLO（You Only Look Once）是一种流行的实时目标检测算法，以其高速和高精度著称。与传统的目标检测方法（如R-CNN系列）不同，YOLO将目标检测任务视为单一的回归问题，直接从图像像素中预测边界框和类别概率，实现了“端到端”的检测。         YOLO将输入图像划分为 S×S 的网格（例如7×7），每个网格负责预测多个边界框（Bounding Box）及其置信度（Confidence Score）和类别概率。边界框：包含框的中心坐标、宽高。置信度：反映框内是否存在目标以及预测的准确性。类别概率：使用Softmax预测框内物体的类别。         传统方法（如滑动窗口）需要多次扫描图像，而YOLO仅需“看一次”（You Only Look Once），通过卷积神经网络一次性输出所有检测结果，因此速度极快。         在昇腾（Ascend）平台上运行YOLO（You Only Look Once）目标检测算法具有重要的技术意义和商业价值，尤其在AI加速计算领域。昇腾是华为推出的高性能AI处理器（如Ascend 910/310），结合昇腾AI软件栈（CANN、MindSpore等），能够显著提升YOLO的推理和训练效率。以下是其核心意义：1. 高性能加速，满足实时性需求；2. 边缘到云的灵活部署；3. 软硬件协同优化。 2      系统环境安装         昇腾平台运行YOLO需要安装这些工具：1.     Ascend-cann-toolkit_8.0.RC3_linux-aarch64，2.     Ascend-cann-kernels-910b_8.0.RC3_linux-aarch64，3.     mindspore=2.5.0，4.     python=3.9， python3.9的环境的安装命令如下，python的版本号为3.10或者3.11会报错：conda create -n yolo20250705python3d9d8 python=3.9conda activate yolo20250705python3d9d8 下载CANN8.0相关工具的网址：https://www.hiascend.com/developer/download/community/result?module=cann&cann=8.0.RC3.beta1；将下载得到的工具包传至服务器，然后安装；使用CANN=8.1或者8.2运行YOLO有可能会报错；安装CANN的命令如下：/tmp/Ascend-cann-toolkit_8.0.RC3_linux-aarch64.run --install/tmp/Ascend-cann-kernels-910b_8.0.RC3_linux-aarch64.run --devel  下载MindSpore的网址：cid:link_0；安装MindSpore的命令如下：pip install https://ms-release.obs.cn-north-4.myhuaweicloud.com/2.5.0/MindSpore/unified/aarch64/mindspore-2.5.0-cp39-cp39-linux_aarch64.whl --trusted-host ms-release.obs.cn-north-4.myhuaweicloud.com -i https://pypi.tuna.tsinghua.edu.cn/simple 代码目录：/apply/yolo20250811/ 进入代码目录：cd /apply/yolo20250811安装环境包资源：pip install -r requirements.txt系统需要安装mesa-libGL工具包，不安装有时会报错：sudo yum install mesa-libGLpython环境需要安装这些工具包，不安装有时会报错；albumentations的版本号>=2.0会报错：pip install sympypip install tepip install albumentations==1.4.24 3      昇腾平台的YOLO的训练与推理训练命令；没有“--ms_mode 1”会报错：python train.py --epochs 600 --config ./configs/yolov11/yolov11-n.yaml --data_dir  ./cache/data/coco --keep_checkpoint_max 1 --auto_accumulate True --per_batch_size 25 --weight ./cache/pretrain_ckpt/yolov11n.ckpt --ms_mode 1 推理命令：python ./demo/predict.py --config ./configs/yolov11/yolov11-n.yaml --weight ./cache/pretrain_ckpt/yolov11n.ckpt --image_path ./cache/data/coco/images/val2017/000000550691.jpg yolo训练日志： (yolo20250705python3d9d8) [root@bms-jp ascendyolo_run_for_v811_20250417a1]# python train.py --epochs 600 --config ./configs/yolov11/yolov11-n.yaml --data_dir  ./cache/data/coco --keep_checkpoint_max 1 --auto_accumulate True --per_batch_size 25 --weight ./cache/pretrain_ckpt/yolov11n.ckpt --ms_mode 1/root/miniconda3/envs/yolo20250705python3d9d8/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.  setattr(self, word, getattr(machar, word).flat[0])/root/miniconda3/envs/yolo20250705python3d9d8/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.  return self._float_to_str(self.smallest_subnormal)/root/miniconda3/envs/yolo20250705python3d9d8/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.  setattr(self, word, getattr(machar, word).flat[0])/root/miniconda3/envs/yolo20250705python3d9d8/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.  return self._float_to_str(self.smallest_subnormal)2025-07-05 22:45:27,447 [INFO] parse_args:2025-07-05 22:45:27,447 [INFO] task                                    detect2025-07-05 22:45:27,447 [INFO] device_target                           Ascend2025-07-05 22:45:27,447 [INFO] save_dir                                ./runs/2025.07.05-22.45.272025-07-05 22:45:27,447 [INFO] log_level                               INFO2025-07-05 22:45:27,447 [INFO] is_parallel                             False2025-07-05 22:45:27,447 [INFO] ms_mode                                 12025-07-05 22:45:27,447 [INFO] max_call_depth                          20002025-07-05 22:45:27,447 [INFO] ms_amp_level                            O22025-07-05 22:45:27,447 [INFO] keep_loss_fp32                          True2025-07-05 22:45:27,447 [INFO] anchor_base                             False2025-07-05 22:45:27,447 [INFO] ms_loss_scaler                          dynamic2025-07-05 22:45:27,447 [INFO] ms_loss_scaler_value                    65536.02025-07-05 22:45:27,447 [INFO] ms_jit                                  True2025-07-05 22:45:27,447 [INFO] ms_enable_graph_kernel                  False2025-07-05 22:45:27,447 [INFO] ms_datasink                             False2025-07-05 22:45:27,447 [INFO] overflow_still_update                   False2025-07-05 22:45:27,447 [INFO] clip_grad                               True2025-07-05 22:45:27,447 [INFO] clip_grad_value                         10.02025-07-05 22:45:27,447 [INFO] ema                                     True2025-07-05 22:45:27,447 [INFO] weight                                  ./cache/pretrain_ckpt/yolov11n.ckpt2025-07-05 22:45:27,447 [INFO] ema_weight                              2025-07-05 22:45:27,447 [INFO] freeze                                  []2025-07-05 22:45:27,447 [INFO] epochs                                  6002025-07-05 22:45:27,447 [INFO] per_batch_size                          252025-07-05 22:45:27,447 [INFO] img_size                                6402025-07-05 22:45:27,447 [INFO] nbs                                     642025-07-05 22:45:27,447 [INFO] accumulate                              3.02025-07-05 22:45:27,447 [INFO] auto_accumulate                         True2025-07-05 22:45:27,447 [INFO] log_interval                            1002025-07-05 22:45:27,447 [INFO] single_cls                              False2025-07-05 22:45:27,447 [INFO] sync_bn                                 False2025-07-05 22:45:27,447 [INFO] keep_checkpoint_max                     12025-07-05 22:45:27,447 [INFO] run_eval                                False2025-07-05 22:45:27,447 [INFO] conf_thres                              0.0012025-07-05 22:45:27,447 [INFO] iou_thres                               0.72025-07-05 22:45:27,447 [INFO] conf_free                               True2025-07-05 22:45:27,447 [INFO] rect                                    False2025-07-05 22:45:27,447 [INFO] nms_time_limit                          20.02025-07-05 22:45:27,447 [INFO] recompute                               False2025-07-05 22:45:27,447 [INFO] recompute_layers                        02025-07-05 22:45:27,447 [INFO] seed                                    22025-07-05 22:45:27,447 [INFO] summary                                 True2025-07-05 22:45:27,447 [INFO] profiler                                False2025-07-05 22:45:27,447 [INFO] profiler_step_num                       12025-07-05 22:45:27,447 [INFO] opencv_threads_num                      02025-07-05 22:45:27,447 [INFO] strict_load                             True2025-07-05 22:45:27,447 [INFO] enable_modelarts                        False2025-07-05 22:45:27,447 [INFO] data_url                                2025-07-05 22:45:27,447 [INFO] ckpt_url                                2025-07-05 22:45:27,447 [INFO] multi_data_url                          2025-07-05 22:45:27,447 [INFO] pretrain_url                            2025-07-05 22:45:27,447 [INFO] train_url                               2025-07-05 22:45:27,447 [INFO] data_dir                                ./cache/data/coco2025-07-05 22:45:27,447 [INFO] ckpt_dir                                /cache/pretrain_ckpt/2025-07-05 22:45:27,447 [INFO] data.dataset_name                       coco2025-07-05 22:45:27,447 [INFO] data.train_set                          /apply/yolo20250811/cache/data/coco/train2017.txt2025-07-05 22:45:27,447 [INFO] data.val_set                            /apply/yolo20250811/cache/data/coco/val2017.txt2025-07-05 22:45:27,447 [INFO] data.test_set                           /apply/yolo20250811/cache/data/coco/test-dev2017.txt2025-07-05 22:45:27,447 [INFO] data.nc                                 802025-07-05 22:45:27,447 [INFO] data.names                              ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush']2025-07-05 22:45:27,447 [INFO] train_transforms.stage_epochs           [590, 10]2025-07-05 22:45:27,447 [INFO] train_transforms.trans_list             [[{'func_name': 'mosaic', 'prob': 1.0}, {'func_name': 'copy_paste', 'prob': 0.1, 'sorted': True}, {'func_name': 'resample_segments'}, {'func_name': 'random_perspective', 'prob': 1.0, 'degrees': 0.0, 'translate': 0.1, 'scale': 0.5, 'shear': 0.0}, {'func_name': 'albumentations'}, {'func_name': 'hsv_augment', 'prob': 1.0, 'hgain': 0.015, 'sgain': 0.7, 'vgain': 0.4}, {'func_name': 'fliplr', 'prob': 0.5}, {'func_name': 'label_norm', 'xyxy2xywh_': True}, {'func_name': 'label_pad', 'padding_size': 160, 'padding_value': -1}, {'func_name': 'image_norm', 'scale': 255.0}, {'func_name': 'image_transpose', 'bgr2rgb': True, 'hwc2chw': True}], [{'func_name': 'letterbox', 'scaleup': True}, {'func_name': 'resample_segments'}, {'func_name': 'random_perspective', 'prob': 1.0, 'degrees': 0.0, 'translate': 0.1, 'scale': 0.5, 'shear': 0.0}, {'func_name': 'albumentations'}, {'func_name': 'hsv_augment', 'prob': 1.0, 'hgain': 0.015, 'sgain': 0.7, 'vgain': 0.4}, {'func_name': 'fliplr', 'prob': 0.5}, {'func_name': 'label_norm', 'xyxy2xywh_': True}, {'func_name': 'label_pad', 'padding_size': 160, 'padding_value': -1}, {'func_name': 'image_norm', 'scale': 255.0}, {'func_name': 'image_transpose', 'bgr2rgb': True, 'hwc2chw': True}]]2025-07-05 22:45:27,447 [INFO] data.test_transforms                    [{'func_name': 'letterbox', 'scaleup': False, 'only_image': True}, {'func_name': 'image_norm', 'scale': 255.0}, {'func_name': 'image_transpose', 'bgr2rgb': True, 'hwc2chw': True}]。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。[INFO] albumentations load success[WARNING] ME(50295:281473090502688,MainProcess):2025-07-05-22:46:03.231.109 [mindspore/run_check/_check_version.py:305] The version 7.6 used for compiling the custom operator does not match Ascend AI software package version 7.5 in the current environment......2025-07-05 22:47:15,474 [WARNING] Epoch 1/600, Step 1/2, accumulate: 3.0, this step grad overflow, drop. Loss scale adjust to 32768.02025-07-05 22:47:15,809 [WARNING] Epoch 1/600, Step 2/2, accumulate: 3.0, this step grad overflow, drop. Loss scale adjust to 16384.02025-07-05 22:47:16,184 [INFO] Epoch 1/600, Step 2/2, imgsize (640, 640), loss: 3.5250, lbox: 1.0629, lcls: 1.3194, dfl: 1.1426, cur_lr: 1.9966999388998374e-052025-07-05 22:47:17,505 [INFO] Epoch 1/600, Step 2/2, step time: 49088.29 ms2025-07-05 22:47:18,444 [INFO] Saving model to ./runs/2025.07.05-22.45.27/weights/yolov11-n-1_2.ckpt2025-07-05 22:47:18,444 [INFO] Epoch 1/600, epoch time: 1.65 min.2025-07-05 22:47:18,710 [WARNING] Epoch 2/600, Step 1/2, accumulate: 3.0, this step grad overflow, drop. Loss scale adjust to 8192.02025-07-05 22:47:19,024 [INFO] Epoch 2/600, Step 2/2, imgsize (640, 640), loss: 3.6963, lbox: 1.0847, lcls: 1.4422, dfl: 1.1694, cur_lr: 3.986799856647849e-052025-07-05 22:47:19,037 [INFO] Epoch 2/600, Step 2/2, step time: 296.27 ms2025-07-05 22:47:19,945 [INFO] Saving model to ./runs/2025.07.05-22.45.27/weights/yolov11-n-2_2.ckpt2025-07-05 22:47:19,946 [INFO] Epoch 2/600, epoch time: 0.03 min.2025-07-05 22:47:20,223 [WARNING] Epoch 3/600, Step 1/2, accumulate: 3.0, this step grad overflow, drop. Loss scale adjust to 4096.0   yolo推理日志： (yolo20250705python3d9d8) [root@bms-jp ascendyolo_run_for_v811_20250417a1]# python ./demo/predict.py --config ./configs/yolov11/yolov11-n.yaml --weight ./cache/pretrain_ckpt/yolov11n.ckpt --image_path ./cache/data/coco/images/val2017/000000550691.jpg                                                                                                                                              /root/miniconda3/envs/yolo20250705python3d9d8/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for  type is zero.     setattr(self, word, getattr(machar, word).flat[0])                                                                                                                                                /root/miniconda3/envs/yolo20250705python3d9d8/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for  type is zero.      return self._float_to_str(self.smallest_subnormal)                                                                                                                                                /root/miniconda3/envs/yolo20250705python3d9d8/lib/python3.9/site-packages/numpy/core/getlimits.py:549: UserWarning: The value of the smallest subnormal for  type is zero.     setattr(self, word, getattr(machar, word).flat[0])                                                                                                                                                /root/miniconda3/envs/yolo20250705python3d9d8/lib/python3.9/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for  type is zero.      return self._float_to_str(self.smallest_subnormal)                                                                                                                                                2025-07-05 22:50:19,157 [WARNING] Parse Model, args: nearest, keep str type                                                                                                                         2025-07-05 22:50:19,204 [WARNING] Parse Model, args: nearest, keep str type                                                                                                                         2025-07-05 22:50:19,584 [INFO] number of network params, total: 2.639747M, trainable: 2.624064M                                                                                                     [WARNING] ME(66771:281473260486688,MainProcess):2025-07-05-22:50:26.245.7 [mindspore/train/serialization.py:1956] For 'load_param_into_net', remove parameter prefix name: ema., continue to load.  2025-07-05 22:50:26,023 [INFO] Load checkpoint from [./cache/pretrain_ckpt/yolov11n.ckpt] success.                                                                                                  .Warning: tiling offset out of range, index: 32                                                                                                                                                     .Warning: tiling offset out of range, index: 32                                                                                                                                                     .Warning: tiling offset out of range, index: 32                                                                                                                                                     Warning: tiling offset out of range, index: 32                                                                                                                                                      Warning: tiling offset out of range, index: 32                                                                                                                                                      Warning: tiling offset out of range, index: 32                                                                                                                                                      ..2025-07-05 22:51:27,507 [INFO] Predict result is: {'category_id': [6, 3, 3, 6, 6], 'bbox': [[194.125, 54.75, 243.875, 354.25], [115.25, 286.5, 82.25, 68.0], [442.0, 283.0, 24.0, 20.0], [3.25, 215.25, 160.75, 64.0], [3.875, 215.5, 159.875, 96.5]], 'score': [0.93115, 0.90283, 0.70898, 0.58154, 0.45508]}                                                                                        2025-07-05 22:51:27,507 [INFO] Speed: 61360.0/11.5/61371.5 ms inference/NMS/total per 640x640 image at batch-size 1;                                                                                2025-07-05 22:51:27,507 [INFO] Detect a image success.                                                                                                                                              2025-07-05 22:51:27,516 [INFO] Infer completed.     4      总结         在昇腾（Ascend）平台上成功运行YOLO模型的训练和推理，通过CANN软件栈和MindSpore框架的深度适配，实现了高效的算子优化及硬件加速（如昇腾910B/310）。关键技术包括动态分片、混合精度训练和DVPP硬件预处理，显著提升了目标检测的推理性能。昇腾NPU在CV任务中的具有强大的竞争力。昇腾显卡在边缘计算、智能安防等场景的AI部署具有重要的产业意义。        

在将segformer-b0算法模型转OM后，在MDC300F MINI上推理耗时1000ms。ONNX中MatMul算子的输入数据的shape带batch维度，将算子的type类型更改为 BatchMatMul后，转OM时，发现MatMul算子前后各出现一个trans_TransData算子。通过profiling计算算子耗时，发现TransposeD、ArgMaxD、TransData、SoftmaxV2、BatchMatMul占用大量的推理时间。请问如何避免带batch维度的MatMul产生TransData算子？针对现在算子耗时，有什么优化的方法吗？

一、简介本项目主要运用华为云 EI 的 ModelArts 的自动学习以及云对象存储的 OBS，实现简单的垃圾分类系统。二、内容描述本垃圾分类图像识别系统主要通过创建图像分类自动学习项目，进行数据标注，进行自动训练和部署测试，再到最后的结束测试。 三、主要流程四、图像分类任务介绍ModelArts 服务之自动学习图像分类项目，是对图像进行检测分类。添加图片并对图像进行分类标注，每个分类识别一种类型的图像。完成图片标注后开始自动训练，即可快速生成图像分类模型。可应用于商品的自动识别、运输车辆种类识别和残次品的自动检测。例如质量检查的场景，则可以上传产品图片，将图片标注“合格”、“不合格”，通过训练部署模型，实现产品的质检。五、系统创建1、创建项目2、添加图片3、数据标注进行“一次性快餐盒-其他垃圾” 的数据标注。 先将左下角的数字选择为 45， 点击图片选择同类的图片（一次可以选择一张或者多张），在标签名栏填写当前选择图片的标签（已有的标签可以直接选择） ， 输入标签名， 点击确定。4、自动训练5、部署测试预测结果：