CNN-RNN 视频动态手势识别人工智能的发展日新月异，也深刻的影响到人机交互领域的发展。手势动作作为一种自然、快捷的交互方式，在智能驾驶、虚拟现实等领域有着广泛的应用。手势识别的任务是，当操作者做出某个手势动作后，计算机能够快速准确的判断出该手势的类型。本文将使用ModelArts开发训练一个视频动态手势识别的算法模型，对上滑、下滑、左滑、右滑、打开、关闭等动态手势类别进行检测，实现类似华为手机隔空手势的功能。算法简介CNN-RNN视频动态手势识别算法首先使用预训练网络InceptionResNetV2逐帧提取视频动作片段特征，然后输入LSTM进行分类。我们使用全栈AI黑客松决赛样例数据对算法进行测试，总共包含108段视频，数据集包含无效手势、上滑、下滑、左滑、右滑、打开、关闭、放大、缩小等9种手势的视频，数据集下载链接如下：https://developer.huaweicloud.com/develop/aigallery/dataset/detail?id=7e9c0d90-461f-4af2-93b3-67a8df76c109代码实现首先我们将采集的视频文件解码抽取关键帧，每隔4帧保存一次，然后对图像进行中心裁剪和预处理，代码如下：def load_video(file_name): cap = cv2.VideoCapture(file_name) # 每隔多少帧抽取一次 frame_interval = 4 frames = [] count = 0 while True: ret, frame = cap.read() if not ret: break # 每隔frame_interval帧保存一次 if count % frame_interval == 0: # 中心裁剪 frame = crop_center_square(frame) # 缩放 frame = cv2.resize(frame, (IMG_SIZE, IMG_SIZE)) # BGR -> RGB [0,1,2] -> [2,1,0] frame = frame[:, :, [2, 1, 0]] frames.append(frame) count += 1 return np.array(frames) 然后我们创建图像特征提取器，使用预训练模型InceptionResNetV2提取图像特征，代码如下：def get_feature_extractor(): feature_extractor = keras.applications.inception_resnet_v2.InceptionResNetV2( weights = 'imagenet', include_top = False, pooling = 'avg', input_shape = (IMG_SIZE, IMG_SIZE, 3) ) preprocess_input = keras.applications.inception_resnet_v2.preprocess_input inputs = keras.Input((IMG_SIZE, IMG_SIZE, 3)) preprocessed = preprocess_input(inputs) outputs = feature_extractor(preprocessed) model = keras.Model(inputs, outputs, name = 'feature_extractor') return model接着提取视频特征向量，如果视频不足40帧就创建全0数组进行补白：def load_data(videos, labels): video_features = [] for video in tqdm(videos): frames = load_video(video) counts = len(frames) # 如果帧数小于MAX_SEQUENCE_LENGTH if counts < MAX_SEQUENCE_LENGTH: # 补白 diff = MAX_SEQUENCE_LENGTH - counts # 创建全0的numpy数组 padding = np.zeros((diff, IMG_SIZE, IMG_SIZE, 3)) # 数组拼接 frames = np.concatenate((frames, padding)) # 获取前MAX_SEQUENCE_LENGTH帧画面 frames = frames[:MAX_SEQUENCE_LENGTH, :] # 批量提取特征 video_feature = feature_extractor.predict(frames) video_features.append(video_feature) return np.array(video_features), np.array(labels) 最后创建LSTM Model，代码如下：def video_cls_model(class_vocab): # 类别数量 classes_num = len(class_vocab) # 定义模型 model = keras.Sequential([ layers.Input(shape=(MAX_SEQUENCE_LENGTH, NUM_FEATURES)), layers.LSTM(64, return_sequences=True), layers.Flatten(), layers.Dense(classes_num, activation='softmax') ]) # 编译模型 model.compile(optimizer = keras.optimizers.Adam(1e-5), loss = keras.losses.SparseCategoricalCrossentropy(from_logits=False), metrics = ['accuracy'] ) return model模型训练体验完整的训练流程可以点击Run in ModelArts运行我发布的Notebook使用云上的免费算力训练，代码链接如下：https://developer.huaweicloud.com/develop/aigallery/notebook/detail?id=e8914ecc-953e-48b1-8e5d-90b37b3bc8e9视频推理首先加载LSTM Model，获取视频类别索引标签：import random # 加载模型 model = tf.keras.models.load_model('saved_model') # 类别标签 label_to_name = {0:'无效手势', 1:'上滑', 2:'下滑', 3:'左滑', 4:'右滑', 5:'打开', 6:'关闭', 7:'放大', 8:'缩小'} 然后使用图像特征提取器InceptionResNetV2提取视频特征：# 获取视频特征 def getVideoFeat(frames): frames_count = len(frames) # 如果帧数小于MAX_SEQUENCE_LENGTH if frames_count < MAX_SEQUENCE_LENGTH: # 补白 diff = MAX_SEQUENCE_LENGTH - frames_count # 创建全0的numpy数组 padding = np.zeros((diff, IMG_SIZE, IMG_SIZE, 3)) # 数组拼接 frames = np.concatenate((frames, padding)) # 取前MAX_SEQ_LENGTH帧 frames = frames[:MAX_SEQUENCE_LENGTH,:] # 计算视频特征 N, 1536 video_feat = feature_extractor.predict(frames) return video_feat最后将视频序列的特征向量输入LSTM进行预测：# 视频预测 def testVideo(): test_file = random.sample(videos, 1)[0] label = test_file.split('_')[-2] print('文件名:{}'.format(test_file) ) print('真实类别:{}'.format(label_to_name.get(int(label))) ) # 读取视频每一帧 frames = load_video(test_file) # 挑选前帧MAX_SEQUENCE_LENGTH显示 frames = frames[:MAX_SEQUENCE_LENGTH].astype(np.uint8) # 保存为GIF imageio.mimsave('animation.gif', frames, duration=10) # 获取特征 feat = getVideoFeat(frames) # 模型推理 prob = model.predict(tf.expand_dims(feat, axis=0))[0] print('预测类别：') for i in np.argsort(prob)[::-1][:5]: print('{}: {}%'.format(label_to_name[i], round(prob[i]*100, 2))) return display(Image(open('animation.gif', 'rb').read())) 运行testVideo()函数，会随机选择一个视频进行预测，并显示模型的预测结果：文件名:hand_gesture/man_005_3_1.mp4 真实类别:左滑 预测类别： 左滑: 78.32% 右滑: 8.54% 下滑: 5.51% 无效手势: 4.33% 上滑: 1.67%文章小结本文介绍了基于CNN-RNN架构的视频动态手势识别算法，通过结合InceptionResNetV2卷积神经网络和LSTM循环神经网络，实现了对多种手势动作（如上滑、下滑、左滑、右滑、打开、关闭等）的高效识别。

OrangePi AI Studio Pro基于MindYolo实现YOLOv8模型训练及验证OrangePi AI Studio Pro是基于 2 个昇腾 310P 处理器的新一代高性能推理解析卡，提供基础通用算力+超强AI算力，整合了训练和推理的全部底层软件栈，实现训推一体。其中AI半精度FP16算力约为176TFLOPS，整数Int8精度可达352TOPS。本章将介绍如何在昇腾310上基于mindyolo实现YOLOv8模型的训练及验证。一、环境准备首先检查昇腾310P的NPU驱动，在命令行中输入：npu-smi info，可以看到两块昇腾310P的AICore的利用率和内存的占用情况。+--------------------------------------------------------------------------------------------------------+ | npu-smi v1.0 Version: 24.1.rc4.b999 | +-------------------------------+-----------------+------------------------------------------------------+ | NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) | | Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) | +===============================+=================+======================================================+ | 30208 310P1 | OK | NA 41 0 / 0 | | 0 0 | 0000:77:00.0 | 0 1416 / 89608 | +-------------------------------+-----------------+------------------------------------------------------+ | 30208 310P1 | OK | NA 40 0 / 0 | | 1 1 | 0000:77:00.0 | 0 1622 / 89085 | +===============================+=================+======================================================+ +-------------------------------+-----------------+------------------------------------------------------+ | NPU Chip | Process id | Process name | Process memory(MB) | +===============================+=================+======================================================+ | No running processes found in NPU 30208 | +===============================+=================+======================================================+之后升级CANN的版本以及更新MindSpore，可以参考我的另一篇文章：如何在OrangePi Studio Pro上升级CANN以及的Pytorch和MindSpore，升级完成后，检查MindSpore的安装情况，我使用的版本是2.7.0。source /usr/local/Ascend/ascend-toolkit/set_env.sh python3 -c "import mindspore;mindspore.set_context(device_target='Ascend');mindspore.run_check()" [WARNING] ME(1621400:139701939115840,MainProcess):2025-09-24-10:46:21.978.000 [mindspore/context.py:1412] For 'context.set_context', the parameter 'device_target' will be deprecated and removed in a future version. Please use the api mindspore.set_device() instead. MindSpore version: 2.7.0 [WARNING] GE_ADPT(1621400,7f0e18710640,python3):2025-09-24-10:46:23.323.570 [mindspore/ops/kernel/ascend/acl_ir/op_api_exec.cc:169] GetAscendDefaultCustomPath] Checking whether the so exists or if permission to access it is available: /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize_vision/op_api/lib/libcust_opapi.so The result of multiplication calculation is correct, MindSpore has been installed on platform [Ascend] successfully! 克隆mindyolo仓库，我们使用由天津大学发布的无人机视觉挑战赛数据集VisDrone-Dataset进行模型的训练及验证。git clone https://github.com/mindspore-lab/mindyolo.git正克隆到 'mindyolo'... remote: Enumerating objects: 3505, done. remote: Counting objects: 100% (157/157), done. remote: Compressing objects: 100% (69/69), done. remote: Total 3505 (delta 114), reused 88 (delta 88), pack-reused 3348 (from 2) 接收对象中: 100% (3505/3505), 6.74 MiB | 8.91 MiB/s, 完成. 处理 delta 中: 100% (2048/2048), 完成.我们将下载后的数据集首先转换成YOLO格式，具体的转换教程可以参考网上的公开资料，经过转换后的visdrone数据集包括以下内容：visdrone ├── train │ ├── images │ │ ├── 000001.jpg │ │ ├── 000002.jpg │ │ ├── ... │ │ └── ... │ └── labels │ ├── 000001.txt │ ├── 000002.txt │ ├── ... │ └── ... └── val ├── images │ ├── 000001.jpg │ ├── 000002.jpg │ ├── ... │ └── ... └── labels ├── 000001.txt ├── 000001.txt ├── ... └── ... 二、数据格式转换由于mindyolo中的train过程使用的数据是yolo格式，而eval过程使用coco数据集中的json文件，因此需要再增加coco格式的标注文件instances_train2017.json、instances_val2017.json以及train.txt和val.txt文件，经过转换后的visdrone数据集包括以下内容：visdrone_COCO_format ├── train.txt ├── val.txt ├── train │ ├── images │ │ ├── 000001.jpg │ │ ├── 000002.jpg │ │ ├── ... │ │ └── ... │ └── labels │ ├── 000001.txt │ ├── 000002.txt │ ├── ... │ └── ... ├── annotations │ ├── instances_train2017.json │ └── instances_val2017.json └── val ├── images │ ├── 000001.jpg │ ├── 000002.jpg │ ├── ... │ └── ... └── labels ├── 000001.txt ├── 000001.txt ├── ... └── ... 我们先把YOLO格式的数据集转换为COCO格式，在mindyolo中实现yolov5_yaml_to_coco.py脚本，具体代码如下：# -*- encoding: utf-8 -*- # @Author: SWHL # @Contact: liekkaskono@163.com import argparse import glob import json import os import shutil import time from pathlib import Path import cv2 import yaml from tqdm import tqdm def read_txt(txt_path): with open(str(txt_path), "r", encoding="utf-8") as f: data = list(map(lambda x: x.rstrip("\n"), f)) return data def mkdir(dir_path): Path(dir_path).mkdir(parents=True, exist_ok=True) def verify_exists(file_path): file_path = Path(file_path).resolve() if not file_path.exists(): raise FileNotFoundError(f"The {file_path} is not exists!!!") class YOLOV5CFG2COCO: def __init__(self, yaml_path): verify_exists(yaml_path) with open(yaml_path, "r", encoding="UTF-8") as f: self.data_cfg = yaml.safe_load(f) self.root_dir = Path(yaml_path).parent.parent self.root_data_dir = Path(self.data_cfg.get("path")) self.train_path = self._get_data_dir("train") self.val_path = self._get_data_dir("val") nc = self.data_cfg["nc"] if "names" in self.data_cfg: self.names = self.data_cfg.get("names") else: # assign class names if missing self.names = [f"class{i}" for i in range(self.data_cfg["nc"])] assert ( len(self.names) == nc ), f"{len(self.names)} names found for nc={nc} dataset in {yaml_path}" # 构建COCO格式目录 self.dst = self.root_dir / f"{Path(self.root_data_dir).stem}_COCO_format" self.coco_train = "train/images" self.coco_val = "val/images" self.coco_annotation = "annotations" self.coco_train_json = ( self.dst / self.coco_annotation / f"instances_train2017.json" ) self.coco_val_json = ( self.dst / self.coco_annotation / f"instances_val2017.json" ) mkdir(self.dst) mkdir(self.dst / self.coco_train) mkdir(self.dst / self.coco_val) mkdir(self.dst / self.coco_annotation) # 构建json内容结构 self.type = "instances" self.categories = [] self._get_category() self.annotation_id = 1 cur_year = time.strftime("%Y", time.localtime(time.time())) self.info = { "year": int(cur_year), "version": "1.0", "description": "For object detection", "date_created": cur_year, } self.licenses = [ { "id": 1, "name": "Apache License v2.0", "url": "https://choosealicense.com/licenses/apache-2.0/", } ] def _get_data_dir(self, mode): data_dir = self.data_cfg.get(mode) if data_dir: if isinstance(data_dir, str): full_path = [str(self.root_data_dir / data_dir)] elif isinstance(data_dir, list): full_path = [str(self.root_data_dir / one_dir) for one_dir in data_dir] else: raise TypeError(f"{data_dir} is not str or list.") else: raise ValueError(f"{mode} dir is not in the yaml.") return full_path def _get_category(self): for i, category in enumerate(self.names, start=1): self.categories.append( { "supercategory": category, "id": i, "name": category, } ) def generate(self): self.train_files = self.get_files(self.train_path) self.valid_files = self.get_files(self.val_path) train_dest_dir = Path(self.dst) / self.coco_train self.gen_dataset( self.train_files, train_dest_dir, self.coco_train_json, mode="train" ) val_dest_dir = Path(self.dst) / self.coco_val self.gen_dataset(self.valid_files, val_dest_dir, self.coco_val_json, mode="val") print(f"The output directory is: {self.dst}") def get_files(self, path): IMG_FORMATS = ["bmp", "dng", "jpeg", "jpg", "mpo", "png", "tif", "tiff", "webp"] f = [] for p in path: p = Path(p) if p.is_dir(): f += glob.glob(str(p / "**" / "*.*"), recursive=True) elif p.is_file(): # file with open(p, "r", encoding="utf-8") as t: t = t.read().strip().splitlines() parent = str(p.parent) + os.sep f += [ x.replace("./", parent) if x.startswith("./") else x for x in t ] else: raise FileExistsError(f"{p} does not exist") im_files = sorted( x.replace("/", os.sep) for x in f if x.split(".")[-1].lower() in IMG_FORMATS ) return im_files def gen_dataset(self, img_paths, target_img_path, target_json, mode): """ https://cocodataset.org/#format-data """ images = [] annotations = [] sa, sb = ( os.sep + "images" + os.sep, os.sep + "labels" + os.sep, ) # /images/, /labels/ substrings for img_id, img_path in enumerate(tqdm(img_paths, desc=mode), 1): label_path = sb.join(img_path.rsplit(sa, 1)).rsplit(".", 1)[0] + ".txt" img_path = Path(img_path) verify_exists(img_path) imgsrc = cv2.imread(str(img_path)) height, width = imgsrc.shape[:2] dest_file_name = f"{img_id:012d}.jpg" save_img_path = target_img_path / dest_file_name if img_path.suffix.lower() == ".jpg": shutil.copyfile(img_path, save_img_path) else: cv2.imwrite(str(save_img_path), imgsrc) images.append( { "date_captured": "2021", "file_name": dest_file_name, "id": img_id, "height": height, "width": width, } ) if Path(label_path).exists(): new_anno = self.read_annotation(label_path, img_id, height, width) if len(new_anno) > 0: annotations.extend(new_anno) else: raise ValueError(f"{label_path} is empty") else: raise FileNotFoundError(f"{label_path} not exists") json_data = { "info": self.info, "images": images, "licenses": self.licenses, "type": self.type, "annotations": annotations, "categories": self.categories, } with open(target_json, "w", encoding="utf-8") as f: json.dump(json_data, f, ensure_ascii=False) def read_annotation(self, txt_file, img_id, height, width): annotation = [] all_info = read_txt(txt_file) for label_info in all_info: # 遍历一张图中不同标注对象 label_info = label_info.split(" ") if len(label_info) < 5: continue category_id, vertex_info = label_info[0], label_info[1:] segmentation, bbox, area = self._get_annotation(vertex_info, height, width) annotation.append( { "segmentation": segmentation, "area": area, "iscrowd": 0, "image_id": img_id, "bbox": bbox, "category_id": int(category_id) + 1, "id": self.annotation_id, } ) self.annotation_id += 1 return annotation @staticmethod def _get_annotation(vertex_info, height, width): cx, cy, w, h = [float(i) for i in vertex_info] cx = cx * width cy = cy * height box_w = w * width box_h = h * height x0 = max(cx - box_w / 2, 0) y0 = max(cy - box_h / 2, 0) x1 = min(x0 + box_w, width) y1 = min(y0 + box_h, height) segmentation = [[x0, y0, x1, y0, x1, y1, x0, y1]] bbox = [x0, y0, box_w, box_h] area = box_w * box_h return segmentation, bbox, area def main(): parser = argparse.ArgumentParser("Datasets converter from YOLOV5 to COCO") parser.add_argument( "--yaml_path", type=str, default="dataset/YOLOV5_yaml/sample.yaml", help="Dataset cfg file", ) args = parser.parse_args() converter = YOLOV5CFG2COCO(args.yaml_path) converter.generate() if __name__ == "__main__": main() 之后在mindyolo目录下创建YOLO格式的配置文件visdrone.yaml：# Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..] path: /root/workspace/dataset/visdrone # dataset root dir (absolute path) train: train/images # train images (relative to 'path') val: val/images # val images (relative to 'path') test: # test images (optional) nc: 12 # Classes，类别 names: 0: ignored regions 1: pedestrian 2: people 3: bicycle 4: car 5: van 6: truck 7: tricycle 8: awning-tricycle 9: bus 10: motor 11: others在终端中运行如下命令将YOLO格式的数据集转换为COCO格式：python3 yolov5_yaml_to_coco.py --yaml_path visdrone.yamltrain: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6471/6471 [01:13<00:00, 88.07it/s] val: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 548/548 [00:03<00:00, 148.22it/s] The output directory is: visdrone_COCO_format再创建coco2yolo.py的Python脚本将COCO格式的标注文件.json导出为labels文件夹中YOLO格式的标注文件.txt：import json import os import argparse parser = argparse.ArgumentParser(description='Test yolo data.') parser.add_argument('-j', help='JSON file', dest='json', required=True) parser.add_argument('-o', help='path to output folder', dest='out',required=True) args = parser.parse_args() json_file = args.json output = args.out class COCO2YOLO: def __init__(self): self._check_file_and_dir(json_file, output) self.labels = json.load(open(json_file, 'r', encoding='utf-8')) self.coco_id_name_map = self._categories() self.coco_name_list = list(self.coco_id_name_map.values()) print("total images", len(self.labels['images'])) print("total categories", len(self.labels['categories'])) print("total labels", len(self.labels['annotations'])) def _check_file_and_dir(self, file_path, dir_path): if not os.path.exists(file_path): raise ValueError("file not found") if not os.path.exists(dir_path): os.makedirs(dir_path) def _categories(self): categories = {} for cls in self.labels['categories']: categories[cls['id']] = cls['name'] return categories def _load_images_info(self): images_info = {} for image in self.labels['images']: id = image['id'] file_name = image['file_name'] if file_name.find('\\') > -1: file_name = file_name[file_name.index('\\')+1:] w = image['width'] h = image['height'] images_info[id] = (file_name, w, h) return images_info def _bbox_2_yolo(self, bbox, img_w, img_h): x, y, w, h = bbox[0], bbox[1], bbox[2], bbox[3] centerx = bbox[0] + w / 2 centery = bbox[1] + h / 2 dw = 1 / img_w dh = 1 / img_h centerx *= dw w *= dw centery *= dh h *= dh return centerx, centery, w, h def _convert_anno(self, images_info): anno_dict = dict() for anno in self.labels['annotations']: bbox = anno['bbox'] image_id = anno['image_id'] category_id = anno['category_id'] image_info = images_info.get(image_id) image_name = image_info[0] img_w = image_info[1] img_h = image_info[2] yolo_box = self._bbox_2_yolo(bbox, img_w, img_h) anno_info = (image_name, category_id, yolo_box) anno_infos = anno_dict.get(image_id) if not anno_infos: anno_dict[image_id] = [anno_info] else: anno_infos.append(anno_info) anno_dict[image_id] = anno_infos return anno_dict def save_classes(self): sorted_classes = list(map(lambda x: x['name'], sorted(self.labels['categories'], key=lambda x: x['id']))) print('coco names', sorted_classes) with open('coco.names', 'w', encoding='utf-8') as f: for cls in sorted_classes: f.write(cls + '\n') f.close() def coco2yolo(self): print("loading image info...") images_info = self._load_images_info() print("loading done, total images", len(images_info)) print("start converting...") anno_dict = self._convert_anno(images_info) print("converting done, total labels", len(anno_dict)) print("saving txt file...") self._save_txt(anno_dict) print("saving done") def _save_txt(self, anno_dict): for k, v in anno_dict.items(): file_name = os.path.splitext(v[0][0])[0] + ".txt" with open(os.path.join(output, file_name), 'w', encoding='utf-8') as f: print(k, v) for obj in v: cat_name = self.coco_id_name_map.get(obj[1]) category_id = self.coco_name_list.index(cat_name) box = ['{:.6f}'.format(x) for x in obj[2]] box = ' '.join(box) line = str(category_id) + ' ' + box f.write(line + '\n') if __name__ == '__main__': c2y = COCO2YOLO() c2y.coco2yolo() 在终端中切换到mindyolo目录下依次运行如下命令导出instances_train2017.json和instances_val2017.json文件对应的YOLO格式的标注文件到labels文件夹中：python3 coco2yolo.py -j ./visdrone_COCO_format/annotations/instances_train2017.json -o ./visdrone_COCO_format/train/labelspython3 coco2yolo.py -j ./visdrone_COCO_format/annotations/instances_val2017.json -o ./visdrone_COCO_format/val/labels最后创建generate_txt.sh脚本在COCO数据集目录下生成train.txt和val.txt，指定训练图片和验证图片的在数据集中的相对路径：#!/bin/bash # 检查是否提供了数据集路径参数 if [ $# -eq 0 ]; then echo "Usage: $0 <dataset_path>" echo "Example: $0 /path/to/visdrone" exit 1 fi # 获取数据集路径 DATASET_PATH="$1" # 检查数据集路径是否存在 if [ ! -d "$DATASET_PATH" ]; then echo "Error: Dataset path '$DATASET_PATH' does not exist." exit 1 fi # 定义训练和验证图片目录 TRAIN_DIR="$DATASET_PATH/train/images" VAL_DIR="$DATASET_PATH/val/images" # 检查训练和验证目录是否存在 if [ ! -d "$TRAIN_DIR" ]; then echo "Error: Train directory '$TRAIN_DIR' does not exist." exit 1 fi if [ ! -d "$VAL_DIR" ]; then echo "Error: Validation directory '$VAL_DIR' does not exist." exit 1 fi # 生成 train.txt TRAIN_TXT="$DATASET_PATH/train.txt" ls "$TRAIN_DIR" | grep '\.jpg$' | sort | sed 's/^/\.\/train\/images\//' > "$TRAIN_TXT" echo "Generated $TRAIN_TXT" # 生成 val.txt VAL_TXT="$DATASET_PATH/val.txt" ls "$VAL_DIR" | grep '\.jpg$' | sort | sed 's/^/\.\/val\/images\//' > "$VAL_TXT" echo "Generated $VAL_TXT" echo "Successfully generated train.txt and val.txt in $DATASET_PATH" 在终端中运行generate_txt.sh，并传入前面COCO数据集的路径：chmod +x generate_txt.sh ./generate_txt.sh visdrone_COCO_formatGenerated visdrone_COCO_format/train.txt Generated visdrone_COCO_format/val.txt Successfully generated train.txt and val.txt in visdrone_COCO_format最终生成的visdrone_COCO_format数据集的格式如下，可以直接用于MindYOLOv8模型的训练：visdrone_COCO_format ├── train.txt ├── val.txt ├── train │ ├── images │ │ ├── 000001.jpg │ │ ├── 000002.jpg │ │ ├── ... │ │ └── ... │ └── labels │ ├── 000001.txt │ ├── 000002.txt │ ├── ... │ └── ... ├── annotations │ ├── instances_train2017.json │ └── instances_val2017.json └── val ├── images │ ├── 000001.jpg │ ├── 000002.jpg │ ├── ... │ └── ... └── labels ├── 000001.txt ├── 000001.txt ├── ... └── ... 三、模型训练MindYOLO支持yaml文件继承机制，因此新编写的配置文件只需要继承MindYOLO提供的原生yaml文件现有配置文件：在configs目录下编写MindYOLO数据集的yaml配置文件，指定训练图片和验证图片的路径以及模型的类别标签：data: dataset_name: visdrone_COCO_format train_set: /root/workspace/mindyolo/visdrone_COCO_format/train.txt val_set: /root/workspace/mindyolo/visdrone_COCO_format/val.txt test_set: /root/workspace/mindyolo/visdrone_COCO_format/val.txt nc: 12 # class names names: ['ignored regions', 'pedestrian', 'people', 'bicycle', 'car', 'van', 'truck', 'tricycle', 'awning-tricycle', 'bus', 'motor', 'others' ] train_transforms: [] test_transforms: [] 修改configs/yolov8s.yaml文件，注释掉原有的coco.yaml配置文件，指定我们自己的数据集，同时添加epochs、img_size、per_batch_size、multi-stage data augment等自定义训练参数：__BASE__: [ # '../coco.yaml', '../visdrone.yaml', './hyp.scratch.low.yaml', './yolov8-base.yaml' ] overflow_still_update: False network: depth_multiple: 0.33 # scales module repeats width_multiple: 0.50 # scales convolution channels max_channels: 1024 epochs: 10 img_size: 1024 per_batch_size: 16 data: num_parallel_workers: 8 # multi-stage data augment train_transforms: { stage_epochs: [ 5, 5 ], trans_list: [ [ { func_name: mosaic, prob: 1.0 }, { func_name: resample_segments }, { func_name: random_perspective, prob: 1.0, degrees: 0.0, translate: 0.1, scale: 0.5, shear: 0.0 }, {func_name: albumentations}, {func_name: hsv_augment, prob: 1.0, hgain: 0.015, sgain: 0.7, vgain: 0.4}, {func_name: fliplr, prob: 0.5}, {func_name: label_norm, xyxy2xywh_: True}, {func_name: label_pad, padding_size: 160, padding_value: -1}, {func_name: image_norm, scale: 255.}, {func_name: image_transpose, bgr2rgb: True, hwc2chw: True} ], [ {func_name: letterbox, scaleup: True}, {func_name: resample_segments}, {func_name: random_perspective, prob: 1.0, degrees: 0.0, translate: 0.1, scale: 0.5, shear: 0.0}, {func_name: albumentations}, {func_name: hsv_augment, prob: 1.0, hgain: 0.015, sgain: 0.7, vgain: 0.4}, {func_name: fliplr, prob: 0.5}, {func_name: label_norm, xyxy2xywh_: True}, {func_name: label_pad, padding_size: 160, padding_value: -1}, {func_name: image_norm, scale: 255.}, {func_name: image_transpose, bgr2rgb: True, hwc2chw: True} ]] } test_transforms: [ {func_name: letterbox, scaleup: False, only_image: True}, {func_name: image_norm, scale: 255.}, {func_name: image_transpose, bgr2rgb: True, hwc2chw: True} ] 在终端中运行train.py进行模型训练，指定模型的配置文件以及使用昇腾NPU：python3 train.py --config ./configs/yolov8/yolov8s.yaml --device_target Ascend默认是跑在0卡上也可以在环境变量中指定DEVICE_ID让模型的训练代码跑在1卡上：import os os.setenv("DEVICE_ID", 1) 如果不想设置环境变量也可以修改mindyolo\mindyolo\utils\utils.py中默认的参数：import os import random import yaml import cv2 from datetime import datetime import numpy as np import mindspore as ms from mindspore import ops, Tensor, nn from mindspore.communication.management import get_group_size, get_rank, init from mindspore import ParallelMode from mindyolo.utils import logger def set_seed(seed=2): np.random.seed(seed) random.seed(seed) ms.set_seed(seed) def set_default(args): # Set Context ms.set_context(mode=args.ms_mode) ms.set_recursion_limit(args.max_call_depth) if args.ms_mode == 0: ms.set_context(jit_config={"jit_level": "O2"}) if args.device_target == "Ascend": ms.set_device("Ascend", int(os.getenv("DEVICE_ID", 1))) ... 2025-10-23 14:48:02,364 [INFO] parse_args: 2025-10-23 14:48:02,364 [INFO] task detect 2025-10-23 14:48:02,364 [INFO] device_target Ascend 2025-10-23 14:48:02,364 [INFO] save_dir ./runs/2025.10.23-14.48.02 2025-10-23 14:48:02,364 [INFO] log_level INFO 2025-10-23 14:48:02,364 [INFO] is_parallel False 2025-10-23 14:48:02,364 [INFO] ms_mode 0 2025-10-23 14:48:02,364 [INFO] max_call_depth 2000 2025-10-23 14:48:02,364 [INFO] ms_amp_level O0 2025-10-23 14:48:02,364 [INFO] keep_loss_fp32 True 2025-10-23 14:48:02,364 [INFO] anchor_base False 2025-10-23 14:48:02,364 [INFO] ms_loss_scaler static 2025-10-23 14:48:02,364 [INFO] ms_loss_scaler_value 1024.0 2025-10-23 14:48:02,364 [INFO] ms_jit True 2025-10-23 14:48:02,364 [INFO] ms_enable_graph_kernel False 2025-10-23 14:48:02,364 [INFO] ms_datasink False 2025-10-23 14:48:02,364 [INFO] overflow_still_update False 2025-10-23 14:48:02,364 [INFO] clip_grad False 2025-10-23 14:48:02,364 [INFO] clip_grad_value 10.0 2025-10-23 14:48:02,364 [INFO] ema True 2025-10-23 14:48:02,364 [INFO] weight 2025-10-23 14:48:02,364 [INFO] ema_weight 2025-10-23 14:48:02,364 [INFO] freeze [] 2025-10-23 14:48:02,364 [INFO] epochs 10 2025-10-23 14:48:02,364 [INFO] per_batch_size 16 2025-10-23 14:48:02,364 [INFO] img_size 1024 2025-10-23 14:48:02,364 [INFO] nbs 64 2025-10-23 14:48:02,364 [INFO] accumulate 1 2025-10-23 14:48:02,364 [INFO] auto_accumulate False 2025-10-23 14:48:02,364 [INFO] log_interval 100 2025-10-23 14:48:02,364 [INFO] single_cls False 2025-10-23 14:48:02,364 [INFO] sync_bn False 2025-10-23 14:48:02,364 [INFO] keep_checkpoint_max 100 2025-10-23 14:48:02,364 [INFO] run_eval False 2025-10-23 14:48:02,364 [INFO] run_eval_interval 1 2025-10-23 14:48:02,364 [INFO] conf_thres 0.001 2025-10-23 14:48:02,364 [INFO] iou_thres 0.7 2025-10-23 14:48:02,364 [INFO] conf_free True 2025-10-23 14:48:02,364 [INFO] rect False 2025-10-23 14:48:02,364 [INFO] nms_time_limit 20.0 2025-10-23 14:48:02,364 [INFO] recompute False 2025-10-23 14:48:02,364 [INFO] recompute_layers 0 2025-10-23 14:48:02,364 [INFO] seed 2 2025-10-23 14:48:02,364 [INFO] summary True 2025-10-23 14:48:02,364 [INFO] profiler False 2025-10-23 14:48:02,364 [INFO] profiler_step_num 1 2025-10-23 14:48:02,364 [INFO] opencv_threads_num 0 2025-10-23 14:48:02,364 [INFO] strict_load True 2025-10-23 14:48:02,364 [INFO] enable_modelarts False 2025-10-23 14:48:02,364 [INFO] data_url 2025-10-23 14:48:02,364 [INFO] ckpt_url 2025-10-23 14:48:02,364 [INFO] multi_data_url 2025-10-23 14:48:02,364 [INFO] pretrain_url 2025-10-23 14:48:02,364 [INFO] train_url 2025-10-23 14:48:02,364 [INFO] data_dir /cache/data/ 2025-10-23 14:48:02,364 [INFO] ckpt_dir /cache/pretrain_ckpt/ 2025-10-23 14:48:02,364 [INFO] data.dataset_name result 2025-10-23 14:48:02,364 [INFO] data.train_set /root/workspace/mindyolo/visdrone_COCO_format/train.txt 2025-10-23 14:48:02,364 [INFO] data.val_set /root/workspace/mindyolo/visdrone_COCO_format/val.txt 2025-10-23 14:48:02,364 [INFO] data.test_set /root/workspace/mindyolo/visdrone_COCO_format/val.txt 2025-10-23 14:48:02,364 [INFO] data.nc 12 2025-10-23 14:48:02,364 [INFO] data.names ['ignored regions', 'pedestrian', 'people', 'bicycle', 'car', 'van', 'truck', 'tricycle', 'awning-tricycle', 'bus', 'motor', 'others'] 2025-10-23 14:48:02,364 [INFO] train_transforms.stage_epochs [5, 5] 2025-10-23 14:48:02,364 [INFO] train_transforms.trans_list [[{'func_name': 'mosaic', 'prob': 1.0}, {'func_name': 'resample_segments'}, {'func_name': 'random_perspective', 'prob': 1.0, 'degrees': 0.0, 'translate': 0.1, 'scale': 0.5, 'shear': 0.0}, {'func_name': 'albumentations'}, {'func_name': 'hsv_augment', 'prob': 1.0, 'hgain': 0.015, 'sgain': 0.7, 'vgain': 0.4}, {'func_name': 'fliplr', 'prob': 0.5}, {'func_name': 'label_norm', 'xyxy2xywh_': True}, {'func_name': 'label_pad', 'padding_size': 160, 'padding_value': -1}, {'func_name': 'image_norm', 'scale': 255.0}, {'func_name': 'image_transpose', 'bgr2rgb': True, 'hwc2chw': True}], [{'func_name': 'letterbox', 'scaleup': True}, {'func_name': 'resample_segments'}, {'func_name': 'random_perspective', 'prob': 1.0, 'degrees': 0.0, 'translate': 0.1, 'scale': 0.5, 'shear': 0.0}, {'func_name': 'albumentations'}, {'func_name': 'hsv_augment', 'prob': 1.0, 'hgain': 0.015, 'sgain': 0.7, 'vgain': 0.4}, {'func_name': 'fliplr', 'prob': 0.5}, {'func_name': 'label_norm', 'xyxy2xywh_': True}, {'func_name': 'label_pad', 'padding_size': 160, 'padding_value': -1}, {'func_name': 'image_norm', 'scale': 255.0}, {'func_name': 'image_transpose', 'bgr2rgb': True, 'hwc2chw': True}]] 2025-10-23 14:48:02,364 [INFO] data.test_transforms [{'func_name': 'letterbox', 'scaleup': False, 'only_image': True}, {'func_name': 'image_norm', 'scale': 255.0}, {'func_name': 'image_transpose', 'bgr2rgb': True, 'hwc2chw': True}] 2025-10-23 14:48:02,364 [INFO] data.num_parallel_workers 8 2025-10-23 14:48:02,364 [INFO] optimizer.optimizer momentum 2025-10-23 14:48:02,364 [INFO] optimizer.lr_init 0.01 2025-10-23 14:48:02,364 [INFO] optimizer.momentum 0.937 2025-10-23 14:48:02,364 [INFO] optimizer.nesterov True 2025-10-23 14:48:02,364 [INFO] optimizer.loss_scale 1.0 2025-10-23 14:48:02,364 [INFO] optimizer.warmup_epochs 3 2025-10-23 14:48:02,364 [INFO] optimizer.warmup_momentum 0.8 2025-10-23 14:48:02,364 [INFO] optimizer.warmup_bias_lr 0.1 2025-10-23 14:48:02,364 [INFO] optimizer.min_warmup_step 1000 2025-10-23 14:48:02,364 [INFO] optimizer.group_param yolov8 2025-10-23 14:48:02,364 [INFO] optimizer.gp_weight_decay 0.0005 2025-10-23 14:48:02,364 [INFO] optimizer.start_factor 1.0 2025-10-23 14:48:02,364 [INFO] optimizer.end_factor 0.01 2025-10-23 14:48:02,364 [INFO] optimizer.epochs 10 2025-10-23 14:48:02,364 [INFO] optimizer.nbs 64 2025-10-23 14:48:02,364 [INFO] optimizer.accumulate 1 2025-10-23 14:48:02,364 [INFO] optimizer.total_batch_size 16 2025-10-23 14:48:02,364 [INFO] loss.name YOLOv8Loss 2025-10-23 14:48:02,364 [INFO] loss.box 7.5 2025-10-23 14:48:02,364 [INFO] loss.cls 0.5 2025-10-23 14:48:02,364 [INFO] loss.dfl 1.5 2025-10-23 14:48:02,364 [INFO] loss.reg_max 16 2025-10-23 14:48:02,364 [INFO] network.model_name yolov8 2025-10-23 14:48:02,364 [INFO] network.nc 80 2025-10-23 14:48:02,364 [INFO] network.reg_max 16 2025-10-23 14:48:02,364 [INFO] network.stride [8, 16, 32] 2025-10-23 14:48:02,364 [INFO] network.backbone [[-1, 1, 'ConvNormAct', [64, 3, 2]], [-1, 1, 'ConvNormAct', [128, 3, 2]], [-1, 3, 'C2f', [128, True]], [-1, 1, 'ConvNormAct', [256, 3, 2]], [-1, 6, 'C2f', [256, True]], [-1, 1, 'ConvNormAct', [512, 3, 2]], [-1, 6, 'C2f', [512, True]], [-1, 1, 'ConvNormAct', [1024, 3, 2]], [-1, 3, 'C2f', [1024, True]], [-1, 1, 'SPPF', [1024, 5]]] 2025-10-23 14:48:02,364 [INFO] network.head [[-1, 1, 'Upsample', ['None', 2, 'nearest']], [[-1, 6], 1, 'Concat', [1]], [-1, 3, 'C2f', [512]], [-1, 1, 'Upsample', ['None', 2, 'nearest']], [[-1, 4], 1, 'Concat', [1]], [-1, 3, 'C2f', [256]], [-1, 1, 'ConvNormAct', [256, 3, 2]], [[-1, 12], 1, 'Concat', [1]], [-1, 3, 'C2f', [512]], [-1, 1, 'ConvNormAct', [512, 3, 2]], [[-1, 9], 1, 'Concat', [1]], [-1, 3, 'C2f', [1024]], [[15, 18, 21], 1, 'YOLOv8Head', ['nc', 'reg_max', 'stride']]] 2025-10-23 14:48:02,364 [INFO] network.depth_multiple 0.33 2025-10-23 14:48:02,364 [INFO] network.width_multiple 0.5 2025-10-23 14:48:02,364 [INFO] network.max_channels 1024 2025-10-23 14:48:02,364 [INFO] config ./configs/yolov8/yolov8s.yaml 2025-10-23 14:48:02,364 [INFO] rank 0 2025-10-23 14:48:02,364 [INFO] rank_size 1 2025-10-23 14:48:02,364 [INFO] total_batch_size 16 2025-10-23 14:48:02,364 [INFO] callback [] 2025-10-23 14:48:02,364 [INFO] 2025-10-23 14:48:02,365 [INFO] Please check the above information for the configurations 2025-10-23 14:48:02,441 [WARNING] Parse Model, args: nearest, keep str type 2025-10-23 14:48:02,451 [WARNING] Parse Model, args: nearest, keep str type 2025-10-23 14:48:02,528 [INFO] number of network params, total: 11.160279M, trainable: 11.140228M [WARNING] GE_ADPT(336686,7ff4350e8740,python3):2025-10-23-14:48:13.472.732 [mindspore/ops/kernel/ascend/acl_ir/op_api_exec.cc:169] GetAscendDefaultCustomPath] Checking whether the so exists or if permission to access it is available: /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize_vision/op_api/lib/libcust_opapi.so 2025-10-23 14:48:14,547 [WARNING] Parse Model, args: nearest, keep str type 2025-10-23 14:48:14,558 [WARNING] Parse Model, args: nearest, keep str type 2025-10-23 14:48:14,646 [INFO] number of network params, total: 11.160279M, trainable: 11.140228M .2025-10-23 14:48:30,416 [INFO] ema_weight not exist, default pretrain weight is currently used. 2025-10-23 14:48:30,421 [INFO] No dataset cache available, caching now... Scanning images: 0%| | 0/6471 [00:00<?, ?it/s]WARNING ⚠️ /root/workspace/mindyolo/visdrone_COCO_format/train/images/000000000335.jpg: 1 duplicate labels removed Scanning '/root/workspace/mindyolo/visdrone_COCO_format/train.cache' images and labels... 397 found, 0 missing, 0 empty, 0 corrupted: 6%|████ | 397/6471 [00:00<00:01, 3960.91it/s]WARNING ⚠️ /root/workspace/mindyolo/visdrone_COCO_format/train/images/000000000427.jpg: 1 duplicate labels removed Scanning '/root/workspace/mindyolo/visdrone_COCO_format/train.cache' images and labels... 1261 found, 0 missing, 0 empty, 0 corrupted: 19%|████████████▋ | 1261/6471 [00:00<00:01, 4238.38it/s]WARNING ⚠️ /root/workspace/mindyolo/visdrone_COCO_format/train/images/000000001492.jpg: 1 duplicate labels removed Scanning '/root/workspace/mindyolo/visdrone_COCO_format/train.cache' images and labels... 3866 found, 0 missing, 0 empty, 0 corrupted: 60%|██████████████████████████████████████▊ | 3866/6471 [00:00<00:00, 4332.85it/s]WARNING ⚠️ /root/workspace/mindyolo/visdrone_COCO_format/train/images/000000003868.jpg: 1 duplicate labels removed Scanning '/root/workspace/mindyolo/visdrone_COCO_format/train.cache' images and labels... 5607 found, 0 missing, 0 empty, 0 corrupted: 87%|████████████████████████████████████████████████████████▎ | 5607/6471 [00:01<00:00, 4337.04it/s]WARNING ⚠️ /root/workspace/mindyolo/visdrone_COCO_format/train/images/000000005742.jpg: 1 duplicate labels removed Scanning '/root/workspace/mindyolo/visdrone_COCO_format/train.cache' images and labels... 6471 found, 0 missing, 0 empty, 0 corrupted: 100%|█████████████████████████████████████████████████████████████████| 6471/6471 [00:01<00:00, 4307.45it/s] 2025-10-23 14:48:32,028 [INFO] New cache created: /root/workspace/mindyolo/visdrone_COCO_format/train.cache.npy 2025-10-23 14:48:32,029 [INFO] Dataset caching success. 2025-10-23 14:48:32,051 [INFO] Dataloader num parallel workers: [8] 2025-10-23 14:48:32,135 [INFO] Dataset Cache file hash/version check success. 2025-10-23 14:48:32,135 [INFO] Load dataset cache from [/root/workspace/mindyolo/visdrone_COCO_format/train.cache.npy] success. Scanning '/root/workspace/mindyolo/visdrone_COCO_format/train.cache.npy' images and labels... 6471 found, 0 missing, 0 empty, 0 corrupted: 100%|███████████████████████████████████████████████████████████████████████| 6471/6471 [00:00<?, ?it/s] 2025-10-23 14:48:32,157 [INFO] Dataloader num parallel workers: [8] 2025-10-23 14:48:32,273 [INFO] Registry(name=callback, total=4) 2025-10-23 14:48:32,273 [INFO] (0): YoloxSwitchTrain in mindyolo/utils/callback.py 2025-10-23 14:48:32,273 [INFO] (1): EvalWhileTrain in mindyolo/utils/callback.py 2025-10-23 14:48:32,273 [INFO] (2): SummaryCallback in mindyolo/utils/callback.py 2025-10-23 14:48:32,273 [INFO] (3): ProfilerCallback in mindyolo/utils/callback.py 2025-10-23 14:48:32,273 [INFO] 2025-10-23 14:48:32,276 [INFO] got 1 active callback as follows: 2025-10-23 14:48:32,276 [INFO] SummaryCallback() 2025-10-23 14:48:32,276 [WARNING] The first epoch will be compiled for the graph, which may take a long time; You can come back later :). albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success [INFO] albumentations load success [INFO] albumentations load success [INFO] albumentations load success [INFO] albumentations load success [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success albumentations: Blur(p=0.01, blur_limit=(3, 7)), MedianBlur(p=0.01, blur_limit=(3, 7)), ToGray(p=0.01), CLAHE(p=0.01, clip_limit=(1, 4.0), tile_grid_size=(8, 8)) [INFO] albumentations load success .........2025-10-23 14:52:54,293 [INFO] Epoch 1/10, Step 100/404, imgsize (1024, 1024), loss: 5.5585, lbox: 3.2052, lcls: 0.4855, dfl: 1.8678, cur_lr: 0.09257426112890244 2025-10-23 14:52:55,203 [INFO] Epoch 1/10, Step 100/404, step time: 2629.27 ms 2025-10-23 14:55:40,115 [INFO] Epoch 1/10, Step 200/404, imgsize (1024, 1024), loss: 4.5693, lbox: 2.5884, lcls: 0.4230, dfl: 1.5578, cur_lr: 0.08514851331710815 2025-10-23 14:55:40,138 [INFO] Epoch 1/10, Step 200/404, step time: 1649.36 ms 2025-10-23 14:58:25,055 [INFO] Epoch 1/10, Step 300/404, imgsize (1024, 1024), loss: 3.9681, lbox: 2.1428, lcls: 0.3853, dfl: 1.4400, cur_lr: 0.07772277295589447 2025-10-23 14:58:25,078 [INFO] Epoch 1/10, Step 300/404, step time: 1649.39 ms 2025-10-23 15:01:10,020 [INFO] Epoch 1/10, Step 400/404, imgsize (1024, 1024), loss: 3.6795, lbox: 2.0528, lcls: 0.3339, dfl: 1.2929, cur_lr: 0.07029703259468079 2025-10-23 15:01:10,044 [INFO] Epoch 1/10, Step 400/404, step time: 1649.65 ms 2025-10-23 15:01:17,111 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-1_404.ckpt 2025-10-23 15:01:17,111 [INFO] Epoch 1/10, epoch time: 12.75 min. 2025-10-23 15:04:02,010 [INFO] Epoch 2/10, Step 100/404, imgsize (1024, 1024), loss: 3.5361, lbox: 1.9678, lcls: 0.3183, dfl: 1.2500, cur_lr: 0.062162574380636215 2025-10-23 15:04:02,018 [INFO] Epoch 2/10, Step 100/404, step time: 1649.07 ms 2025-10-23 15:06:46,939 [INFO] Epoch 2/10, Step 200/404, imgsize (1024, 1024), loss: 3.3767, lbox: 1.8395, lcls: 0.3042, dfl: 1.2329, cur_lr: 0.05465514957904816 2025-10-23 15:06:46,947 [INFO] Epoch 2/10, Step 200/404, step time: 1649.28 ms 2025-10-23 15:09:31,885 [INFO] Epoch 2/10, Step 300/404, imgsize (1024, 1024), loss: 3.3604, lbox: 1.8753, lcls: 0.3134, dfl: 1.1718, cur_lr: 0.0471477210521698 2025-10-23 15:09:31,894 [INFO] Epoch 2/10, Step 300/404, step time: 1649.46 ms 2025-10-23 15:12:16,806 [INFO] Epoch 2/10, Step 400/404, imgsize (1024, 1024), loss: 3.2902, lbox: 1.8262, lcls: 0.2795, dfl: 1.1846, cur_lr: 0.03964029625058174 2025-10-23 15:12:16,814 [INFO] Epoch 2/10, Step 400/404, step time: 1649.20 ms 2025-10-23 15:12:23,860 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-2_404.ckpt 2025-10-23 15:12:23,860 [INFO] Epoch 2/10, epoch time: 11.11 min. 2025-10-23 15:15:08,782 [INFO] Epoch 3/10, Step 100/404, imgsize (1024, 1024), loss: 3.3220, lbox: 1.7991, lcls: 0.3124, dfl: 1.2106, cur_lr: 0.031090890988707542 2025-10-23 15:15:08,791 [INFO] Epoch 3/10, Step 100/404, step time: 1649.30 ms 2025-10-23 15:17:53,703 [INFO] Epoch 3/10, Step 200/404, imgsize (1024, 1024), loss: 3.1162, lbox: 1.6879, lcls: 0.2824, dfl: 1.1460, cur_lr: 0.02350178174674511 2025-10-23 15:17:53,711 [INFO] Epoch 3/10, Step 200/404, step time: 1649.20 ms 2025-10-23 15:20:38,631 [INFO] Epoch 3/10, Step 300/404, imgsize (1024, 1024), loss: 3.0332, lbox: 1.6024, lcls: 0.2703, dfl: 1.1605, cur_lr: 0.015912672504782677 2025-10-23 15:20:38,639 [INFO] Epoch 3/10, Step 300/404, step time: 1649.28 ms 2025-10-23 15:23:23,580 [INFO] Epoch 3/10, Step 400/404, imgsize (1024, 1024), loss: 3.1371, lbox: 1.7095, lcls: 0.2808, dfl: 1.1469, cur_lr: 0.008323564194142818 2025-10-23 15:23:23,589 [INFO] Epoch 3/10, Step 400/404, step time: 1649.49 ms 2025-10-23 15:23:30,617 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-3_404.ckpt 2025-10-23 15:23:30,617 [INFO] Epoch 3/10, epoch time: 11.11 min. 2025-10-23 15:26:15,527 [INFO] Epoch 4/10, Step 100/404, imgsize (1024, 1024), loss: 3.2965, lbox: 1.8179, lcls: 0.2614, dfl: 1.2172, cur_lr: 0.007029999978840351 2025-10-23 15:26:15,535 [INFO] Epoch 4/10, Step 100/404, step time: 1649.18 ms 2025-10-23 15:29:00,451 [INFO] Epoch 4/10, Step 200/404, imgsize (1024, 1024), loss: 3.1855, lbox: 1.7697, lcls: 0.2504, dfl: 1.1654, cur_lr: 0.007029999978840351 2025-10-23 15:29:00,459 [INFO] Epoch 4/10, Step 200/404, step time: 1649.24 ms 2025-10-23 15:31:45,369 [INFO] Epoch 4/10, Step 300/404, imgsize (1024, 1024), loss: 2.9900, lbox: 1.6270, lcls: 0.2307, dfl: 1.1323, cur_lr: 0.007029999978840351 2025-10-23 15:31:45,378 [INFO] Epoch 4/10, Step 300/404, step time: 1649.18 ms 2025-10-23 15:34:30,277 [INFO] Epoch 4/10, Step 400/404, imgsize (1024, 1024), loss: 3.1742, lbox: 1.7506, lcls: 0.2590, dfl: 1.1646, cur_lr: 0.007029999978840351 2025-10-23 15:34:30,285 [INFO] Epoch 4/10, Step 400/404, step time: 1649.07 ms 2025-10-23 15:34:37,315 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-4_404.ckpt 2025-10-23 15:34:37,316 [INFO] Epoch 4/10, epoch time: 11.11 min. 2025-10-23 15:37:22,195 [INFO] Epoch 5/10, Step 100/404, imgsize (1024, 1024), loss: 2.9632, lbox: 1.6123, lcls: 0.2424, dfl: 1.1085, cur_lr: 0.006039999891072512 2025-10-23 15:37:22,204 [INFO] Epoch 5/10, Step 100/404, step time: 1648.88 ms 2025-10-23 15:40:07,094 [INFO] Epoch 5/10, Step 200/404, imgsize (1024, 1024), loss: 2.7776, lbox: 1.4777, lcls: 0.2025, dfl: 1.0975, cur_lr: 0.006039999891072512 2025-10-23 15:40:07,103 [INFO] Epoch 5/10, Step 200/404, step time: 1648.99 ms 2025-10-23 15:42:52,021 [INFO] Epoch 5/10, Step 300/404, imgsize (1024, 1024), loss: 2.7209, lbox: 1.4253, lcls: 0.2130, dfl: 1.0826, cur_lr: 0.006039999891072512 2025-10-23 15:42:52,029 [INFO] Epoch 5/10, Step 300/404, step time: 1649.26 ms 2025-10-23 15:45:36,965 [INFO] Epoch 5/10, Step 400/404, imgsize (1024, 1024), loss: 2.7360, lbox: 1.4817, lcls: 0.2157, dfl: 1.0387, cur_lr: 0.006039999891072512 2025-10-23 15:45:36,973 [INFO] Epoch 5/10, Step 400/404, step time: 1649.44 ms 2025-10-23 15:45:44,037 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-5_404.ckpt 2025-10-23 15:45:44,037 [INFO] Epoch 5/10, epoch time: 11.11 min. 2025-10-23 15:48:28,914 [INFO] Epoch 6/10, Step 100/404, imgsize (1024, 1024), loss: 2.6675, lbox: 1.4472, lcls: 0.2042, dfl: 1.0161, cur_lr: 0.005049999803304672 2025-10-23 15:48:28,923 [INFO] Epoch 6/10, Step 100/404, step time: 1648.85 ms 2025-10-23 15:51:13,798 [INFO] Epoch 6/10, Step 200/404, imgsize (1024, 1024), loss: 2.7114, lbox: 1.4235, lcls: 0.1986, dfl: 1.0893, cur_lr: 0.005049999803304672 2025-10-23 15:51:13,807 [INFO] Epoch 6/10, Step 200/404, step time: 1648.84 ms 2025-10-23 15:53:58,688 [INFO] Epoch 6/10, Step 300/404, imgsize (1024, 1024), loss: 2.6783, lbox: 1.4169, lcls: 0.1985, dfl: 1.0629, cur_lr: 0.005049999803304672 2025-10-23 15:53:58,697 [INFO] Epoch 6/10, Step 300/404, step time: 1648.90 ms 2025-10-23 15:56:43,578 [INFO] Epoch 6/10, Step 400/404, imgsize (1024, 1024), loss: 2.7539, lbox: 1.4734, lcls: 0.2037, dfl: 1.0768, cur_lr: 0.005049999803304672 2025-10-23 15:56:43,586 [INFO] Epoch 6/10, Step 400/404, step time: 1648.89 ms 2025-10-23 15:56:50,613 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-6_404.ckpt 2025-10-23 15:56:50,613 [INFO] Epoch 6/10, epoch time: 11.11 min. 2025-10-23 15:59:35,561 [INFO] Epoch 7/10, Step 100/404, imgsize (1024, 1024), loss: 2.9109, lbox: 1.6203, lcls: 0.2210, dfl: 1.0696, cur_lr: 0.00406000018119812 2025-10-23 15:59:35,569 [INFO] Epoch 7/10, Step 100/404, step time: 1649.56 ms 2025-10-23 16:02:20,470 [INFO] Epoch 7/10, Step 200/404, imgsize (1024, 1024), loss: 2.6941, lbox: 1.4727, lcls: 0.2068, dfl: 1.0147, cur_lr: 0.00406000018119812 2025-10-23 16:02:20,479 [INFO] Epoch 7/10, Step 200/404, step time: 1649.10 ms 2025-10-23 16:05:05,384 [INFO] Epoch 7/10, Step 300/404, imgsize (1024, 1024), loss: 2.8098, lbox: 1.4810, lcls: 0.2188, dfl: 1.1101, cur_lr: 0.00406000018119812 2025-10-23 16:05:05,391 [INFO] Epoch 7/10, Step 300/404, step time: 1649.12 ms 2025-10-23 16:07:50,302 [INFO] Epoch 7/10, Step 400/404, imgsize (1024, 1024), loss: 2.8426, lbox: 1.5529, lcls: 0.2108, dfl: 1.0788, cur_lr: 0.00406000018119812 2025-10-23 16:07:50,310 [INFO] Epoch 7/10, Step 400/404, step time: 1649.18 ms 2025-10-23 16:07:57,341 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-7_404.ckpt 2025-10-23 16:07:57,342 [INFO] Epoch 7/10, epoch time: 11.11 min. 2025-10-23 16:10:42,225 [INFO] Epoch 8/10, Step 100/404, imgsize (1024, 1024), loss: 2.4095, lbox: 1.2257, lcls: 0.1704, dfl: 1.0134, cur_lr: 0.0030700000934302807 2025-10-23 16:10:42,233 [INFO] Epoch 8/10, Step 100/404, step time: 1648.92 ms 2025-10-23 16:13:27,126 [INFO] Epoch 8/10, Step 200/404, imgsize (1024, 1024), loss: 2.6034, lbox: 1.3788, lcls: 0.1872, dfl: 1.0374, cur_lr: 0.0030700000934302807 2025-10-23 16:13:27,134 [INFO] Epoch 8/10, Step 200/404, step time: 1649.00 ms 2025-10-23 16:16:12,032 [INFO] Epoch 8/10, Step 300/404, imgsize (1024, 1024), loss: 2.6074, lbox: 1.3916, lcls: 0.1787, dfl: 1.0371, cur_lr: 0.0030700000934302807 2025-10-23 16:16:12,041 [INFO] Epoch 8/10, Step 300/404, step time: 1649.07 ms 2025-10-23 16:18:56,946 [INFO] Epoch 8/10, Step 400/404, imgsize (1024, 1024), loss: 2.8867, lbox: 1.4981, lcls: 0.2189, dfl: 1.1697, cur_lr: 0.0030700000934302807 2025-10-23 16:18:56,954 [INFO] Epoch 8/10, Step 400/404, step time: 1649.13 ms 2025-10-23 16:19:03,973 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-8_404.ckpt 2025-10-23 16:19:03,973 [INFO] Epoch 8/10, epoch time: 11.11 min. 2025-10-23 16:21:48,883 [INFO] Epoch 9/10, Step 100/404, imgsize (1024, 1024), loss: 2.8544, lbox: 1.6248, lcls: 0.2181, dfl: 1.0115, cur_lr: 0.0020800000056624413 2025-10-23 16:21:48,891 [INFO] Epoch 9/10, Step 100/404, step time: 1649.18 ms 2025-10-23 16:24:33,791 [INFO] Epoch 9/10, Step 200/404, imgsize (1024, 1024), loss: 2.9393, lbox: 1.6026, lcls: 0.2223, dfl: 1.1145, cur_lr: 0.0020800000056624413 2025-10-23 16:24:33,799 [INFO] Epoch 9/10, Step 200/404, step time: 1649.08 ms 2025-10-23 16:27:18,695 [INFO] Epoch 9/10, Step 300/404, imgsize (1024, 1024), loss: 2.4632, lbox: 1.2884, lcls: 0.1701, dfl: 1.0047, cur_lr: 0.0020800000056624413 2025-10-23 16:27:18,703 [INFO] Epoch 9/10, Step 300/404, step time: 1649.04 ms 2025-10-23 16:30:03,567 [INFO] Epoch 9/10, Step 400/404, imgsize (1024, 1024), loss: 2.7216, lbox: 1.4867, lcls: 0.2002, dfl: 1.0346, cur_lr: 0.0020800000056624413 2025-10-23 16:30:03,575 [INFO] Epoch 9/10, Step 400/404, step time: 1648.72 ms 2025-10-23 16:30:10,627 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-9_404.ckpt 2025-10-23 16:30:10,627 [INFO] Epoch 9/10, epoch time: 11.11 min. 2025-10-23 16:32:55,537 [INFO] Epoch 10/10, Step 100/404, imgsize (1024, 1024), loss: 2.5899, lbox: 1.4239, lcls: 0.1668, dfl: 0.9992, cur_lr: 0.0010900000343099236 2025-10-23 16:32:55,545 [INFO] Epoch 10/10, Step 100/404, step time: 1649.18 ms 2025-10-23 16:35:40,433 [INFO] Epoch 10/10, Step 200/404, imgsize (1024, 1024), loss: 2.5535, lbox: 1.3745, lcls: 0.1813, dfl: 0.9976, cur_lr: 0.0010900000343099236 2025-10-23 16:35:40,441 [INFO] Epoch 10/10, Step 200/404, step time: 1648.95 ms 2025-10-23 16:38:25,358 [INFO] Epoch 10/10, Step 300/404, imgsize (1024, 1024), loss: 2.4509, lbox: 1.2441, lcls: 0.1717, dfl: 1.0351, cur_lr: 0.0010900000343099236 2025-10-23 16:38:25,366 [INFO] Epoch 10/10, Step 300/404, step time: 1649.25 ms 2025-10-23 16:41:10,260 [INFO] Epoch 10/10, Step 400/404, imgsize (1024, 1024), loss: 2.6832, lbox: 1.4217, lcls: 0.1896, dfl: 1.0719, cur_lr: 0.0010900000343099236 2025-10-23 16:41:10,268 [INFO] Epoch 10/10, Step 400/404, step time: 1649.02 ms 2025-10-23 16:41:17,324 [INFO] Saving model to ./runs/2025.10.23-14.48.02/weights/yolov8s-10_404.ckpt 2025-10-23 16:41:17,324 [INFO] Epoch 10/10, epoch time: 11.11 min. 2025-10-23 16:41:17,742 [INFO] End Train. 2025-10-23 16:41:18,446 [INFO] Training completed.平均每个epoch耗时约10min左右，在训练过程中我们也可以查看AI Core的利用率以及内存的占用情况：npu-smi info+--------------------------------------------------------------------------------------------------------+ | npu-smi v1.0 Version: 24.1.rc4.b999 | +-------------------------------+-----------------+------------------------------------------------------+ | NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) | | Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) | +===============================+=================+======================================================+ | 30208 310P1 | OK | NA 52 11372 / 11372 | | 0 0 | 0000:77:00.0 | 99 24288/ 89608 | +-------------------------------+-----------------+------------------------------------------------------+ | 30208 310P1 | OK | NA 42 0 / 0 | | 1 1 | 0000:77:00.0 | 0 1576 / 89085 | +===============================+=================+======================================================+ +-------------------------------+-----------------+------------------------------------------------------+ | NPU Chip | Process id | Process name | Process memory(MB) | +===============================+=================+======================================================+ | 30208 0 | 336686 | python3 | 22835 | +===============================+=================+======================================================+四、模型验证这里我们仅训练了10个epoch进行模型的验证，可以看到模型的精度和召回率如下：python3 test.py --config ./configs/yolov8/yolov8s.yaml --device_target Ascend --weight ./runs/2025.10.23-14.48.02/weights/yolov8s-10_404.ckpt2025-10-23 16:46:18,824 [INFO] parse_args: 2025-10-23 16:46:18,824 [INFO] task detect 2025-10-23 16:46:18,824 [INFO] device_target Ascend 2025-10-23 16:46:18,824 [INFO] ms_mode 0 2025-10-23 16:46:18,824 [INFO] ms_amp_level O0 2025-10-23 16:46:18,824 [INFO] ms_enable_graph_kernel False 2025-10-23 16:46:18,824 [INFO] precision_mode None 2025-10-23 16:46:18,824 [INFO] weight ./runs/2025.10.23-14.48.02/weights/yolov8s-10_404.ckpt 2025-10-23 16:46:18,824 [INFO] per_batch_size 16 2025-10-23 16:46:18,824 [INFO] img_size 1024 2025-10-23 16:46:18,824 [INFO] single_cls False 2025-10-23 16:46:18,824 [INFO] rect False 2025-10-23 16:46:18,824 [INFO] exec_nms True 2025-10-23 16:46:18,824 [INFO] nms_time_limit 60.0 2025-10-23 16:46:18,824 [INFO] conf_thres 0.001 2025-10-23 16:46:18,824 [INFO] iou_thres 0.7 2025-10-23 16:46:18,824 [INFO] conf_free True 2025-10-23 16:46:18,824 [INFO] seed 2 2025-10-23 16:46:18,824 [INFO] log_level INFO 2025-10-23 16:46:18,824 [INFO] save_dir ./runs_test/2025.10.23-16.46.18 2025-10-23 16:46:18,824 [INFO] enable_modelarts False 2025-10-23 16:46:18,824 [INFO] data_url 2025-10-23 16:46:18,824 [INFO] ckpt_url 2025-10-23 16:46:18,824 [INFO] train_url 2025-10-23 16:46:18,824 [INFO] data_dir /cache/data/ 2025-10-23 16:46:18,824 [INFO] is_parallel False 2025-10-23 16:46:18,824 [INFO] ckpt_dir /cache/pretrain_ckpt/ 2025-10-23 16:46:18,824 [INFO] data.dataset_name result 2025-10-23 16:46:18,824 [INFO] data.train_set /root/workspace/mindyolo/visdrone_COCO_format/train.txt 2025-10-23 16:46:18,824 [INFO] data.val_set /root/workspace/mindyolo/visdrone_COCO_format/val.txt 2025-10-23 16:46:18,824 [INFO] data.test_set /root/workspace/mindyolo/visdrone_COCO_format/val.txt 2025-10-23 16:46:18,824 [INFO] data.nc 12 2025-10-23 16:46:18,824 [INFO] data.names ['ignored regions', 'pedestrian', 'people', 'bicycle', 'car', 'van', 'truck', 'tricycle', 'awning-tricycle', 'bus', 'motor', 'others'] 2025-10-23 16:46:18,824 [INFO] train_transforms.stage_epochs [5, 5] 2025-10-23 16:46:18,824 [INFO] train_transforms.trans_list [[{'func_name': 'mosaic', 'prob': 1.0}, {'func_name': 'resample_segments'}, {'func_name': 'random_perspective', 'prob': 1.0, 'degrees': 0.0, 'translate': 0.1, 'scale': 0.5, 'shear': 0.0}, {'func_name': 'albumentations'}, {'func_name': 'hsv_augment', 'prob': 1.0, 'hgain': 0.015, 'sgain': 0.7, 'vgain': 0.4}, {'func_name': 'fliplr', 'prob': 0.5}, {'func_name': 'label_norm', 'xyxy2xywh_': True}, {'func_name': 'label_pad', 'padding_size': 160, 'padding_value': -1}, {'func_name': 'image_norm', 'scale': 255.0}, {'func_name': 'image_transpose', 'bgr2rgb': True, 'hwc2chw': True}], [{'func_name': 'letterbox', 'scaleup': True}, {'func_name': 'resample_segments'}, {'func_name': 'random_perspective', 'prob': 1.0, 'degrees': 0.0, 'translate': 0.1, 'scale': 0.5, 'shear': 0.0}, {'func_name': 'albumentations'}, {'func_name': 'hsv_augment', 'prob': 1.0, 'hgain': 0.015, 'sgain': 0.7, 'vgain': 0.4}, {'func_name': 'fliplr', 'prob': 0.5}, {'func_name': 'label_norm', 'xyxy2xywh_': True}, {'func_name': 'label_pad', 'padding_size': 160, 'padding_value': -1}, {'func_name': 'image_norm', 'scale': 255.0}, {'func_name': 'image_transpose', 'bgr2rgb': True, 'hwc2chw': True}]] 2025-10-23 16:46:18,824 [INFO] data.test_transforms [{'func_name': 'letterbox', 'scaleup': False, 'only_image': True}, {'func_name': 'image_norm', 'scale': 255.0}, {'func_name': 'image_transpose', 'bgr2rgb': True, 'hwc2chw': True}] 2025-10-23 16:46:18,824 [INFO] data.num_parallel_workers 8 2025-10-23 16:46:18,824 [INFO] optimizer.optimizer momentum 2025-10-23 16:46:18,824 [INFO] optimizer.lr_init 0.01 2025-10-23 16:46:18,824 [INFO] optimizer.momentum 0.937 2025-10-23 16:46:18,824 [INFO] optimizer.nesterov True 2025-10-23 16:46:18,824 [INFO] optimizer.loss_scale 1.0 2025-10-23 16:46:18,824 [INFO] optimizer.warmup_epochs 3 2025-10-23 16:46:18,824 [INFO] optimizer.warmup_momentum 0.8 2025-10-23 16:46:18,824 [INFO] optimizer.warmup_bias_lr 0.1 2025-10-23 16:46:18,824 [INFO] optimizer.min_warmup_step 1000 2025-10-23 16:46:18,824 [INFO] optimizer.group_param yolov8 2025-10-23 16:46:18,824 [INFO] optimizer.gp_weight_decay 0.0005 2025-10-23 16:46:18,824 [INFO] optimizer.start_factor 1.0 2025-10-23 16:46:18,824 [INFO] optimizer.end_factor 0.01 2025-10-23 16:46:18,824 [INFO] loss.name YOLOv8Loss 2025-10-23 16:46:18,824 [INFO] loss.box 7.5 2025-10-23 16:46:18,824 [INFO] loss.cls 0.5 2025-10-23 16:46:18,824 [INFO] loss.dfl 1.5 2025-10-23 16:46:18,824 [INFO] loss.reg_max 16 2025-10-23 16:46:18,824 [INFO] epochs 10 2025-10-23 16:46:18,824 [INFO] sync_bn True 2025-10-23 16:46:18,824 [INFO] anchor_base False 2025-10-23 16:46:18,824 [INFO] opencv_threads_num 0 2025-10-23 16:46:18,824 [INFO] network.model_name yolov8 2025-10-23 16:46:18,824 [INFO] network.nc 80 2025-10-23 16:46:18,824 [INFO] network.reg_max 16 2025-10-23 16:46:18,824 [INFO] network.stride [8, 16, 32] 2025-10-23 16:46:18,824 [INFO] network.backbone [[-1, 1, 'ConvNormAct', [64, 3, 2]], [-1, 1, 'ConvNormAct', [128, 3, 2]], [-1, 3, 'C2f', [128, True]], [-1, 1, 'ConvNormAct', [256, 3, 2]], [-1, 6, 'C2f', [256, True]], [-1, 1, 'ConvNormAct', [512, 3, 2]], [-1, 6, 'C2f', [512, True]], [-1, 1, 'ConvNormAct', [1024, 3, 2]], [-1, 3, 'C2f', [1024, True]], [-1, 1, 'SPPF', [1024, 5]]] 2025-10-23 16:46:18,824 [INFO] network.head [[-1, 1, 'Upsample', ['None', 2, 'nearest']], [[-1, 6], 1, 'Concat', [1]], [-1, 3, 'C2f', [512]], [-1, 1, 'Upsample', ['None', 2, 'nearest']], [[-1, 4], 1, 'Concat', [1]], [-1, 3, 'C2f', [256]], [-1, 1, 'ConvNormAct', [256, 3, 2]], [[-1, 12], 1, 'Concat', [1]], [-1, 3, 'C2f', [512]], [-1, 1, 'ConvNormAct', [512, 3, 2]], [[-1, 9], 1, 'Concat', [1]], [-1, 3, 'C2f', [1024]], [[15, 18, 21], 1, 'YOLOv8Head', ['nc', 'reg_max', 'stride']]] 2025-10-23 16:46:18,824 [INFO] network.depth_multiple 0.33 2025-10-23 16:46:18,824 [INFO] network.width_multiple 0.5 2025-10-23 16:46:18,824 [INFO] network.max_channels 1024 2025-10-23 16:46:18,824 [INFO] overflow_still_update False 2025-10-23 16:46:18,824 [INFO] config ./configs/yolov8/yolov8s.yaml 2025-10-23 16:46:18,824 [INFO] rank 0 2025-10-23 16:46:18,824 [INFO] rank_size 1 2025-10-23 16:46:18,824 [INFO] 2025-10-23 16:46:18,898 [WARNING] Parse Model, args: nearest, keep str type 2025-10-23 16:46:18,909 [WARNING] Parse Model, args: nearest, keep str type 2025-10-23 16:46:18,984 [INFO] number of network params, total: 11.160279M, trainable: 11.140228M [WARNING] GE_ADPT(540183,7efcd8e26740,python3):2025-10-23-16:46:22.493.658 [mindspore/ops/kernel/ascend/acl_ir/op_api_exec.cc:169] GetAscendDefaultCustomPath] Checking whether the so exists or if permission to access it is available: /usr/local/Ascend/ascend-toolkit/latest/opp/vendors/customize_vision/op_api/lib/libcust_opapi.so 2025-10-23 16:46:23,434 [INFO] Load checkpoint from [./runs/2025.10.23-14.48.02/weights/yolov8s-10_404.ckpt] success. 2025-10-23 16:46:23,437 [INFO] No dataset cache available, caching now... Scanning '/root/workspace/mindyolo/visdrone_COCO_format/val.cache' images and labels... 548 found, 0 missing, 0 empty, 0 corrupted: 100%|█████████████████████████████████████████████████████████████████████████████| 548/548 [00:00<00:00, 3754.44it/s] 2025-10-23 16:46:23,595 [INFO] New cache created: /root/workspace/mindyolo/visdrone_COCO_format/val.cache.npy 2025-10-23 16:46:23,595 [INFO] Dataset caching success. 2025-10-23 16:46:23,597 [INFO] Dataloader num parallel workers: [8] 2025-10-23 16:46:23,607 [WARNING] unable to load fast_coco_eval api, use normal one instead Warning: tiling offset out of range, index: 32 ..2025-10-23 16:46:55,297 [INFO] Sample 35/1, time cost: 30512.14 ms. 2025-10-23 16:46:57,108 [INFO] Sample 35/2, time cost: 1722.38 ms. 2025-10-23 16:46:58,628 [INFO] Sample 35/3, time cost: 1420.95 ms. 2025-10-23 16:47:00,538 [INFO] Sample 35/4, time cost: 1809.91 ms. 2025-10-23 16:47:02,502 [INFO] Sample 35/5, time cost: 1865.30 ms. 2025-10-23 16:47:04,321 [INFO] Sample 35/6, time cost: 1718.46 ms. 2025-10-23 16:47:06,724 [INFO] Sample 35/7, time cost: 2303.35 ms. 2025-10-23 16:47:08,940 [INFO] Sample 35/8, time cost: 2117.25 ms. 2025-10-23 16:47:11,018 [INFO] Sample 35/9, time cost: 1978.46 ms. 2025-10-23 16:47:13,101 [INFO] Sample 35/10, time cost: 1982.41 ms. 2025-10-23 16:47:14,871 [INFO] Sample 35/11, time cost: 1671.05 ms. 2025-10-23 16:47:17,112 [INFO] Sample 35/12, time cost: 2140.79 ms. 2025-10-23 16:47:19,142 [INFO] Sample 35/13, time cost: 1930.53 ms. 2025-10-23 16:47:20,984 [INFO] Sample 35/14, time cost: 1741.35 ms. 2025-10-23 16:47:23,393 [INFO] Sample 35/15, time cost: 2307.50 ms. 2025-10-23 16:47:25,557 [INFO] Sample 35/16, time cost: 2060.89 ms. 2025-10-23 16:47:27,324 [INFO] Sample 35/17, time cost: 1664.00 ms. 2025-10-23 16:47:29,254 [INFO] Sample 35/18, time cost: 1824.31 ms. 2025-10-23 16:47:31,281 [INFO] Sample 35/19, time cost: 1921.78 ms. 2025-10-23 16:47:33,331 [INFO] Sample 35/20, time cost: 1942.85 ms. 2025-10-23 16:47:35,806 [INFO] Sample 35/21, time cost: 2368.87 ms. 2025-10-23 16:47:38,165 [INFO] Sample 35/22, time cost: 2255.00 ms. 2025-10-23 16:47:40,453 [INFO] Sample 35/23, time cost: 2182.96 ms. 2025-10-23 16:47:42,588 [INFO] Sample 35/24, time cost: 2029.14 ms. 2025-10-23 16:47:44,490 [INFO] Sample 35/25, time cost: 1796.02 ms. 2025-10-23 16:47:46,804 [INFO] Sample 35/26, time cost: 2207.91 ms. 2025-10-23 16:47:49,181 [INFO] Sample 35/27, time cost: 2270.69 ms. 2025-10-23 16:47:50,926 [INFO] Sample 35/28, time cost: 1638.70 ms. 2025-10-23 16:47:53,079 [INFO] Sample 35/29, time cost: 2046.37 ms. 2025-10-23 16:47:55,061 [INFO] Sample 35/30, time cost: 1875.28 ms. 2025-10-23 16:47:57,140 [INFO] Sample 35/31, time cost: 1972.00 ms. 2025-10-23 16:47:59,895 [INFO] Sample 35/32, time cost: 2647.24 ms. 2025-10-23 16:48:02,196 [INFO] Sample 35/33, time cost: 2191.50 ms. 2025-10-23 16:48:04,739 [INFO] Sample 35/34, time cost: 2434.77 ms. ..2025-10-23 16:48:20,509 [INFO] Sample 35/35, time cost: 15723.18 ms. 2025-10-23 16:48:20,509 [INFO] loading annotations into memory... 2025-10-23 16:48:20,639 [INFO] Done (t=0.13s) 2025-10-23 16:48:20,639 [INFO] creating index... 2025-10-23 16:48:20,650 [INFO] index created! 2025-10-23 16:48:20,650 [INFO] Loading and preparing results... 2025-10-23 16:48:21,106 [INFO] DONE (t=0.46s) 2025-10-23 16:48:21,106 [INFO] creating index... 2025-10-23 16:48:21,134 [INFO] index created! 2025-10-23 16:48:21,135 [INFO] Running per image evaluation... 2025-10-23 16:48:21,135 [INFO] Evaluate annotation type *bbox* 2025-10-23 16:48:31,087 [INFO] DONE (t=9.95s). 2025-10-23 16:48:31,087 [INFO] Accumulating evaluation results... 2025-10-23 16:48:31,996 [INFO] DONE (t=0.91s). 2025-10-23 16:48:31,996 [INFO] Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.019 2025-10-23 16:48:31,996 [INFO] Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.036 2025-10-23 16:48:31,996 [INFO] Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.019 2025-10-23 16:48:31,996 [INFO] Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.016 2025-10-23 16:48:31,997 [INFO] Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.024 2025-10-23 16:48:31,997 [INFO] Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.056 2025-10-23 16:48:31,997 [INFO] Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.009 2025-10-23 16:48:31,997 [INFO] Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.049 2025-10-23 16:48:31,997 [INFO] Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.076 2025-10-23 16:48:31,997 [INFO] Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.057 2025-10-23 16:48:31,997 [INFO] Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.106 2025-10-23 16:48:31,997 [INFO] Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.161 2025-10-23 16:48:31,997 [INFO] Speed: 99.0/100.3/199.3 ms inference/NMS/total per 1024x1024 image at batch-size 16; 2025-10-23 16:48:31,997 [INFO] Testing completed, cost 133.18s.使用predict.py测试训练模型参数的结果并进行可视化推理，运行方式如下：python3 examples/finetune_visdrone/predict.py --config ./configs/yolov8/yolov8s.yaml --weight=./runs/2025.10.23-14.48.02/weights/yolov8s-120_404.ckpt --image_path ./visdrone_COCO_format/val/images/000000000001.jpg训练120个epoch后，模型的推理效果如下：五、小结本文详细阐述了在OrangePi AI Studio Pro上基于昇腾310P使用MindYolo框架实现YOLOv8模型训练与验证的完整流程，涵盖环境准备、数据集格式转换、模型训练参数配置及性能评估。

【朝推夜训】Ascend310p YOLOv8 NPU 训练和推理在华为昇思MindSpore框架的加持下，我们在OrangePi AI Studio Pro开发板上实现YOLOv8m模型的完整训练流程。在单块NPU上训练YOLOv8m模型，每轮训练7000张图像仅需6.92分钟，10轮训练总耗时约69分钟。从训练日志可以看出，模型损失值loss从第一轮的6.45逐步下降到最后一轮的2.58左右，表明模型训练效果良好。训练过程中，NPU的AICore利用率和内存占用情况都保持在合理水平，证明了Ascend 310P芯片在目标检测任务中的优异表现，其性能可与NVIDIA GPU相媲美，为开发者提供了另一种高效的AI计算平台选择。通过mindyolo开源仓库，其他开发者也可以复现这一成果并进行进一步的开发和优化。我们在昇腾310AI加速卡上使用昇思MindSpore把YOLOv8模型的NPU训练和推理给跑通了，性能不输于NVIDIA的GPU。OrangePi AI Stuido Pro与Atlas 300V Pro视频解析卡搭载是同款Ascend 310p芯片，总共是两块，每块有96G的内存，可以提供176TFlops的训练算力和352Tops的推理算力。上图是在单块NPU上训练yolov8m模型的AICore的利用率以及内存的占用情况，总共7000张图像每轮训练时长仅需6.92分钟：2025-09-24 16:47:11,931 [INFO] 2025-09-24 16:47:11,931 [INFO] Please check the above information for the configurations 2025-09-24 16:47:12,050 [WARNING] Parse Model, args: nearest, keep str type 2025-09-24 16:47:12,069 [WARNING] Parse Model, args: nearest, keep str type 2025-09-24 16:47:12,184 [INFO] number of network params, total: 25.896391M, trainable: 25.863252M 2025-09-24 16:47:16,786 [WARNING] Parse Model, args: nearest, keep str type 2025-09-24 16:47:16,807 [WARNING] Parse Model, args: nearest, keep str type 2025-09-24 16:47:16,920 [INFO] number of network params, total: 25.896391M, trainable: 25.863252M 2025-09-24 16:47:31,011 [INFO] ema_weight not exist, default pretrain weight is currently used. 2025-09-24 16:47:31,118 [INFO] Dataset Cache file hash/version check success. 2025-09-24 16:47:31,118 [INFO] Load dataset cache from [/home/orangepi/workspace/mindyolo/examples/finetune_visdrone/train.cache.npy] success. 2025-09-24 16:47:31,142 [INFO] Dataloader num parallel workers: [8] 2025-09-24 16:47:31,240 [INFO] Dataset Cache file hash/version check success. 2025-09-24 16:47:31,240 [INFO] Load dataset cache from [/home/orangepi/workspace/mindyolo/examples/finetune_visdrone/train.cache.npy] success. 2025-09-24 16:47:31,264 [INFO] Dataloader num parallel workers: [8] 2025-09-24 16:47:31,438 [INFO] 2025-09-24 16:47:31,445 [INFO] got 1 active callback as follows: 2025-09-24 16:47:31,445 [INFO] SummaryCallback() 2025-09-24 16:47:31,445 [WARNING] The first epoch will be compiled for the graph, which may take a long time; You can come back later :). 2025-09-24 16:50:38,076 [INFO] Epoch 1/10, Step 100/404, imgsize (640, 640), loss: 6.4507, lbox: 3.8446, lcls: 0.5687, dfl: 2.0375, cur_lr: 0.09257426112890244 2025-09-24 16:50:38,970 [INFO] Epoch 1/10, Step 100/404, step time: 1875.26 ms 2025-09-24 16:52:21,629 [INFO] Epoch 1/10, Step 200/404, imgsize (640, 640), loss: 4.8078, lbox: 3.0080, lcls: 0.4118, dfl: 1.3880, cur_lr: 0.08514851331710815 2025-09-24 16:52:21,653 [INFO] Epoch 1/10, Step 200/404, step time: 1026.83 ms 2025-09-24 16:54:04,347 [INFO] Epoch 1/10, Step 300/404, imgsize (640, 640), loss: 4.0795, lbox: 2.4281, lcls: 0.3466, dfl: 1.3048, cur_lr: 0.07772277295589447 2025-09-24 16:54:04,371 [INFO] Epoch 1/10, Step 300/404, step time: 1027.18 ms 2025-09-24 16:55:47,067 [INFO] Epoch 1/10, Step 400/404, imgsize (640, 640), loss: 3.8245, lbox: 2.1755, lcls: 0.3567, dfl: 1.2923, cur_lr: 0.07029703259468079 2025-09-24 16:55:47,091 [INFO] Epoch 1/10, Step 400/404, step time: 1027.19 ms 2025-09-24 16:55:52,087 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-1_404.ckpt 2025-09-24 16:55:52,087 [INFO] Epoch 1/10, epoch time: 8.34 min. 2025-09-24 16:57:34,759 [INFO] Epoch 2/10, Step 100/404, imgsize (640, 640), loss: 3.8083, lbox: 2.2584, lcls: 0.3404, dfl: 1.2095, cur_lr: 0.062162574380636215 2025-09-24 16:57:34,768 [INFO] Epoch 2/10, Step 100/404, step time: 1026.80 ms 2025-09-24 16:59:17,441 [INFO] Epoch 2/10, Step 200/404, imgsize (640, 640), loss: 3.7835, lbox: 2.2670, lcls: 0.3574, dfl: 1.1592, cur_lr: 0.05465514957904816 2025-09-24 16:59:17,450 [INFO] Epoch 2/10, Step 200/404, step time: 1026.82 ms 2025-09-24 17:01:00,127 [INFO] Epoch 2/10, Step 300/404, imgsize (640, 640), loss: 3.5251, lbox: 2.0144, lcls: 0.3210, dfl: 1.1898, cur_lr: 0.0471477210521698 2025-09-24 17:01:00,136 [INFO] Epoch 2/10, Step 300/404, step time: 1026.85 ms 2025-09-24 17:02:42,826 [INFO] Epoch 2/10, Step 400/404, imgsize (640, 640), loss: 3.5596, lbox: 2.0947, lcls: 0.3086, dfl: 1.1563, cur_lr: 0.03964029625058174 2025-09-24 17:02:42,835 [INFO] Epoch 2/10, Step 400/404, step time: 1026.99 ms 2025-09-24 17:02:47,745 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-2_404.ckpt 2025-09-24 17:02:47,745 [INFO] Epoch 2/10, epoch time: 6.93 min. 2025-09-24 17:04:30,489 [INFO] Epoch 3/10, Step 100/404, imgsize (640, 640), loss: 3.5524, lbox: 2.1004, lcls: 0.2938, dfl: 1.1582, cur_lr: 0.031090890988707542 2025-09-24 17:04:30,497 [INFO] Epoch 3/10, Step 100/404, step time: 1027.52 ms 2025-09-24 17:06:13,196 [INFO] Epoch 3/10, Step 200/404, imgsize (640, 640), loss: 3.8549, lbox: 2.2845, lcls: 0.3526, dfl: 1.2178, cur_lr: 0.02350178174674511 2025-09-24 17:06:13,205 [INFO] Epoch 3/10, Step 200/404, step time: 1027.07 ms 2025-09-24 17:07:55,875 [INFO] Epoch 3/10, Step 300/404, imgsize (640, 640), loss: 3.6236, lbox: 2.1016, lcls: 0.3113, dfl: 1.2106, cur_lr: 0.015912672504782677 2025-09-24 17:07:55,883 [INFO] Epoch 3/10, Step 300/404, step time: 1026.78 ms 2025-09-24 17:09:38,572 [INFO] Epoch 3/10, Step 400/404, imgsize (640, 640), loss: 3.5586, lbox: 2.0730, lcls: 0.3314, dfl: 1.1542, cur_lr: 0.008323564194142818 2025-09-24 17:09:38,581 [INFO] Epoch 3/10, Step 400/404, step time: 1026.97 ms 2025-09-24 17:09:43,528 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-3_404.ckpt 2025-09-24 17:09:43,529 [INFO] Epoch 3/10, epoch time: 6.93 min. 2025-09-24 17:11:26,211 [INFO] Epoch 4/10, Step 100/404, imgsize (640, 640), loss: 3.3767, lbox: 1.9760, lcls: 0.2928, dfl: 1.1079, cur_lr: 0.007029999978840351 2025-09-24 17:11:26,218 [INFO] Epoch 4/10, Step 100/404, step time: 1026.90 ms 2025-09-24 17:13:08,899 [INFO] Epoch 4/10, Step 200/404, imgsize (640, 640), loss: 3.4213, lbox: 1.9382, lcls: 0.3052, dfl: 1.1779, cur_lr: 0.007029999978840351 2025-09-24 17:13:08,908 [INFO] Epoch 4/10, Step 200/404, step time: 1026.89 ms 2025-09-24 17:14:51,583 [INFO] Epoch 4/10, Step 300/404, imgsize (640, 640), loss: 2.8313, lbox: 1.5666, lcls: 0.2380, dfl: 1.0267, cur_lr: 0.007029999978840351 2025-09-24 17:14:51,591 [INFO] Epoch 4/10, Step 300/404, step time: 1026.83 ms 2025-09-24 17:16:34,277 [INFO] Epoch 4/10, Step 400/404, imgsize (640, 640), loss: 3.2905, lbox: 1.9274, lcls: 0.2889, dfl: 1.0741, cur_lr: 0.007029999978840351 2025-09-24 17:16:34,285 [INFO] Epoch 4/10, Step 400/404, step time: 1026.94 ms 2025-09-24 17:16:39,232 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-4_404.ckpt 2025-09-24 17:16:39,232 [INFO] Epoch 4/10, epoch time: 6.93 min. 2025-09-24 17:18:21,892 [INFO] Epoch 5/10, Step 100/404, imgsize (640, 640), loss: 3.1534, lbox: 1.7844, lcls: 0.2581, dfl: 1.1109, cur_lr: 0.006039999891072512 2025-09-24 17:18:21,900 [INFO] Epoch 5/10, Step 100/404, step time: 1026.67 ms 2025-09-24 17:20:04,596 [INFO] Epoch 5/10, Step 200/404, imgsize (640, 640), loss: 3.1152, lbox: 1.7685, lcls: 0.2518, dfl: 1.0949, cur_lr: 0.006039999891072512 2025-09-24 17:20:04,604 [INFO] Epoch 5/10, Step 200/404, step time: 1027.04 ms 2025-09-24 17:21:47,284 [INFO] Epoch 5/10, Step 300/404, imgsize (640, 640), loss: 3.3179, lbox: 1.8412, lcls: 0.2888, dfl: 1.1880, cur_lr: 0.006039999891072512 2025-09-24 17:21:47,292 [INFO] Epoch 5/10, Step 300/404, step time: 1026.88 ms 2025-09-24 17:23:29,968 [INFO] Epoch 5/10, Step 400/404, imgsize (640, 640), loss: 3.2193, lbox: 1.8366, lcls: 0.2620, dfl: 1.1207, cur_lr: 0.006039999891072512 2025-09-24 17:23:29,976 [INFO] Epoch 5/10, Step 400/404, step time: 1026.84 ms 2025-09-24 17:23:34,954 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-5_404.ckpt 2025-09-24 17:23:34,954 [INFO] Epoch 5/10, epoch time: 6.93 min. 2025-09-24 17:25:17,530 [INFO] Epoch 6/10, Step 100/404, imgsize (640, 640), loss: 2.7642, lbox: 1.5834, lcls: 0.2164, dfl: 0.9643, cur_lr: 0.005049999803304672 2025-09-24 17:25:17,538 [INFO] Epoch 6/10, Step 100/404, step time: 1025.84 ms 2025-09-24 17:27:00,125 [INFO] Epoch 6/10, Step 200/404, imgsize (640, 640), loss: 2.6854, lbox: 1.4272, lcls: 0.2080, dfl: 1.0502, cur_lr: 0.005049999803304672 2025-09-24 17:27:00,134 [INFO] Epoch 6/10, Step 200/404, step time: 1025.96 ms 2025-09-24 17:28:42,720 [INFO] Epoch 6/10, Step 300/404, imgsize (640, 640), loss: 2.7541, lbox: 1.5028, lcls: 0.2171, dfl: 1.0342, cur_lr: 0.005049999803304672 2025-09-24 17:28:42,728 [INFO] Epoch 6/10, Step 300/404, step time: 1025.94 ms 2025-09-24 17:30:25,315 [INFO] Epoch 6/10, Step 400/404, imgsize (640, 640), loss: 2.8092, lbox: 1.5545, lcls: 0.2121, dfl: 1.0427, cur_lr: 0.005049999803304672 2025-09-24 17:30:25,323 [INFO] Epoch 6/10, Step 400/404, step time: 1025.95 ms 2025-09-24 17:30:30,293 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-6_404.ckpt 2025-09-24 17:30:30,294 [INFO] Epoch 6/10, epoch time: 6.92 min. 2025-09-24 17:32:12,881 [INFO] Epoch 7/10, Step 100/404, imgsize (640, 640), loss: 3.0997, lbox: 1.8226, lcls: 0.2402, dfl: 1.0369, cur_lr: 0.00406000018119812 2025-09-24 17:32:12,890 [INFO] Epoch 7/10, Step 100/404, step time: 1025.96 ms 2025-09-24 17:33:55,477 [INFO] Epoch 7/10, Step 200/404, imgsize (640, 640), loss: 2.8140, lbox: 1.5979, lcls: 0.2143, dfl: 1.0018, cur_lr: 0.00406000018119812 2025-09-24 17:33:55,485 [INFO] Epoch 7/10, Step 200/404, step time: 1025.96 ms 2025-09-24 17:35:38,072 [INFO] Epoch 7/10, Step 300/404, imgsize (640, 640), loss: 3.0294, lbox: 1.6439, lcls: 0.2544, dfl: 1.1310, cur_lr: 0.00406000018119812 2025-09-24 17:35:38,081 [INFO] Epoch 7/10, Step 300/404, step time: 1025.95 ms 2025-09-24 17:37:20,660 [INFO] Epoch 7/10, Step 400/404, imgsize (640, 640), loss: 2.8015, lbox: 1.5686, lcls: 0.2252, dfl: 1.0077, cur_lr: 0.00406000018119812 2025-09-24 17:37:20,669 [INFO] Epoch 7/10, Step 400/404, step time: 1025.88 ms 2025-09-24 17:37:25,643 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-7_404.ckpt 2025-09-24 17:37:25,644 [INFO] Epoch 7/10, epoch time: 6.92 min. 2025-09-24 17:39:08,227 [INFO] Epoch 8/10, Step 100/404, imgsize (640, 640), loss: 2.5091, lbox: 1.3373, lcls: 0.1711, dfl: 1.0007, cur_lr: 0.0030700000934302807 2025-09-24 17:39:08,236 [INFO] Epoch 8/10, Step 100/404, step time: 1025.92 ms 2025-09-24 17:40:50,818 [INFO] Epoch 8/10, Step 200/404, imgsize (640, 640), loss: 2.5926, lbox: 1.4141, lcls: 0.1923, dfl: 0.9863, cur_lr: 0.0030700000934302807 2025-09-24 17:40:50,826 [INFO] Epoch 8/10, Step 200/404, step time: 1025.91 ms 2025-09-24 17:42:33,392 [INFO] Epoch 8/10, Step 300/404, imgsize (640, 640), loss: 2.5341, lbox: 1.3811, lcls: 0.1869, dfl: 0.9660, cur_lr: 0.0030700000934302807 2025-09-24 17:42:33,400 [INFO] Epoch 8/10, Step 300/404, step time: 1025.74 ms 2025-09-24 17:44:15,994 [INFO] Epoch 8/10, Step 400/404, imgsize (640, 640), loss: 3.0024, lbox: 1.6379, lcls: 0.2284, dfl: 1.1361, cur_lr: 0.0030700000934302807 2025-09-24 17:44:16,002 [INFO] Epoch 8/10, Step 400/404, step time: 1026.02 ms 2025-09-24 17:44:20,974 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-8_404.ckpt 2025-09-24 17:44:20,975 [INFO] Epoch 8/10, epoch time: 6.92 min. 2025-09-24 17:46:03,561 [INFO] Epoch 9/10, Step 100/404, imgsize (640, 640), loss: 3.0890, lbox: 1.8395, lcls: 0.2321, dfl: 1.0174, cur_lr: 0.0020800000056624413 2025-09-24 17:46:03,569 [INFO] Epoch 9/10, Step 100/404, step time: 1025.94 ms 2025-09-24 17:47:46,157 [INFO] Epoch 9/10, Step 200/404, imgsize (640, 640), loss: 2.9621, lbox: 1.6608, lcls: 0.2360, dfl: 1.0652, cur_lr: 0.0020800000056624413 2025-09-24 17:47:46,166 [INFO] Epoch 9/10, Step 200/404, step time: 1025.96 ms 2025-09-24 17:49:28,755 [INFO] Epoch 9/10, Step 300/404, imgsize (640, 640), loss: 2.4801, lbox: 1.3320, lcls: 0.1753, dfl: 0.9728, cur_lr: 0.0020800000056624413 2025-09-24 17:49:28,763 [INFO] Epoch 9/10, Step 300/404, step time: 1025.97 ms 2025-09-24 17:51:11,359 [INFO] Epoch 9/10, Step 400/404, imgsize (640, 640), loss: 2.8075, lbox: 1.5971, lcls: 0.1995, dfl: 1.0109, cur_lr: 0.0020800000056624413 2025-09-24 17:51:11,367 [INFO] Epoch 9/10, Step 400/404, step time: 1026.03 ms 2025-09-24 17:51:16,330 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-9_404.ckpt 2025-09-24 17:51:16,331 [INFO] Epoch 9/10, epoch time: 6.92 min. 2025-09-24 17:52:58,913 [INFO] Epoch 10/10, Step 100/404, imgsize (640, 640), loss: 2.6278, lbox: 1.4529, lcls: 0.1860, dfl: 0.9889, cur_lr: 0.0010900000343099236 2025-09-24 17:52:58,921 [INFO] Epoch 10/10, Step 100/404, step time: 1025.90 ms 2025-09-24 17:54:41,521 [INFO] Epoch 10/10, Step 200/404, imgsize (640, 640), loss: 2.7550, lbox: 1.5724, lcls: 0.2083, dfl: 0.9742, cur_lr: 0.0010900000343099236 2025-09-24 17:54:41,529 [INFO] Epoch 10/10, Step 200/404, step time: 1026.08 ms 2025-09-24 17:56:24,125 [INFO] Epoch 10/10, Step 300/404, imgsize (640, 640), loss: 2.4470, lbox: 1.2448, lcls: 0.1758, dfl: 1.0263, cur_lr: 0.0010900000343099236 2025-09-24 17:56:24,133 [INFO] Epoch 10/10, Step 300/404, step time: 1026.03 ms 2025-09-24 17:58:06,727 [INFO] Epoch 10/10, Step 400/404, imgsize (640, 640), loss: 2.5783, lbox: 1.3733, lcls: 0.1848, dfl: 1.0202, cur_lr: 0.0010900000343099236 2025-09-24 17:58:06,736 [INFO] Epoch 10/10, Step 400/404, step time: 1026.02 ms 2025-09-24 17:58:11,744 [INFO] Saving model to ./runs/2025.09.24-16.47.11/weights/yolov8m-10_404.ckpt 2025-09-24 17:58:11,745 [INFO] Epoch 10/10, epoch time: 6.92 min. 2025-09-24 17:58:12,149 [INFO] End Train. 2025-09-24 17:58:12,561 [INFO] Training completed.以下是模型训练了10个epoch的使用NPU在测试集图片上的推理结果：2025-09-24 18:13:24,511 [WARNING] Parse Model, args: nearest, keep str type 2025-09-24 18:13:24,532 [WARNING] Parse Model, args: nearest, keep str type 2025-09-24 18:13:24,639 [INFO] number of network params, total: 25.896391M, trainable: 25.863252M 2025-09-24 18:13:29,405 [INFO] Load checkpoint from [/home/orangepi/workspace/mindyolo/runs/2025.09.24-16.47.11/weights/yolov8m-10_404.ckpt] success. 2025-09-24 18:13:53,915 [INFO] Predict result is: {'category_id': [4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 1, 4, 4, 5, 10, 4, 1, 4, 2, 4, 1, 5, 10, 4, 2, 4, 1], 'bbox': [[866.402, 359.922, 125.209, 179.961], [619.836, 379.246, 140.848, 229.434], [704.238, 192.678, 102.631, 112.359], [572.588, 189.689, 108.707, 103.76], [80.484, 471.75, 334.953, 243.844], [739.99, 15.987, 60.305, 60.944], [1179.242, 68.017, 143.637, 56.163], [1220.215, 154.843, 138.523, 76.782], [1217.559, 108.026, 140.516, 63.733], [822.475, 15.34, 56.744, 75.039], [621.438, 70.781, 19.938, 55.292], [1106.859, 128.463, 79.986, 95.99], [773.168, 90.047, 71.42, 95.293], [773.467, 88.951, 70.988, 95.924], [1122.158, 371.145, 48.12, 90.512], [1168.982, 2.274, 83.141, 77.081], [723.45, 65.277, 21.877, 51.017], [1145.906, 0.556, 76.467, 46.708], [672.513, 71.818, 25.857, 46.933], [488.816, 350.559, 107.844, 117.605], [672.778, 71.918, 26.172, 48.194], [1106.826, 128.612, 79.621, 96.239], [1058.831, 319.314, 35.087, 75.056], [1146.62, 0.365, 54.586, 48.643], [1124.963, 370.945, 42.359, 66.473], [1148.197, 1.046, 92.537, 51.581], [526.153, 87.349, 29.123, 37.91]], 'score': [0.93223, 0.92336, 0.90671, 0.90539, 0.84414, 0.83682, 0.83292, 0.75641, 0.74857, 0.74295, 0.72221, 0.63341, 0.62439, 0.5829, 0.50411, 0.48259, 0.42391, 0.42188, 0.42185, 0.36533, 0.29963, 0.29451, 0.29264, 0.28265, 0.26525, 0.2585, 0.25038]} 2025-09-24 18:13:53,915 [INFO] Speed: 24481.6/5.7/24487.3 ms inference/NMS/total per 640x640 image at batch-size 1; 2025-09-24 18:13:53,915 [INFO] Detect a image success. 2025-09-24 18:13:53,924 [INFO] Infer completed.模型训练和推理代码可以从mindyolo仓库上下载：https://github.com/mindspore-lab/mindyolo

华为云开发者空间基于昇腾NPU实现CT肺炎影像分割模型训练与推理本案例将介绍如何在华为云开发者空间的AI Notebook环境中，利用昇腾NPU 910B4硬件资源训练和推理一个用于CT肺炎影像分割的深度学习模型，涵盖从数据准备、预处理、模型构建、训练到推理可视化的完整深度学习工作流。首先，从OBS存储桶下载并解压了COVID-19 CT扫描数据集，该数据集包含原始CT扫描图像、肺部掩码、感染区域掩码以及肺部和感染区域的组合掩码。import os import zipfile # 下载数据集 if not os.path.exists('Covid-19.zip'): os.system('wget -q https://orangepi-ai-studio.obs.cn-north-4.myhuaweicloud.com/Covid-19.zip') # 解压数据集 if not os.path.exists('Covid-19'): zip_file = zipfile.ZipFile('Covid-19.zip') zip_file.extractall() zip_file.close() 读取数据集中的元数据并显示前5行：import pandas as pd data = pd.read_csv('Covid-19/metadata.csv') data.head() ct_scanlung_maskinfection_masklung_and_infection_mask0…/input/covid19-ct-scans/ct_scans/coronacases……/input/covid19-ct-scans/lung_mask/coronacase……/input/covid19-ct-scans/infection_mask/coron……/input/covid19-ct-scans/lung_and_infection_m…1…/input/covid19-ct-scans/ct_scans/coronacases……/input/covid19-ct-scans/lung_mask/coronacase……/input/covid19-ct-scans/infection_mask/coron……/input/covid19-ct-scans/lung_and_infection_m…2…/input/covid19-ct-scans/ct_scans/coronacases……/input/covid19-ct-scans/lung_mask/coronacase……/input/covid19-ct-scans/infection_mask/coron……/input/covid19-ct-scans/lung_and_infection_m…3…/input/covid19-ct-scans/ct_scans/coronacases……/input/covid19-ct-scans/lung_mask/coronacase……/input/covid19-ct-scans/infection_mask/coron……/input/covid19-ct-scans/lung_and_infection_m…4…/input/covid19-ct-scans/ct_scans/coronacases……/input/covid19-ct-scans/lung_mask/coronacase……/input/covid19-ct-scans/infection_mask/coron……/input/covid19-ct-scans/lung_and_infection_m…我们分别获取原始图像、肺部mask、感染mask、肺部和感染mask的文件路径：# 原始图像 ct_scan_sample_file = data.loc[0,'ct_scan'].replace('../input/covid19-ct-scans','Covid-19') # 肺部mask lung_mask_sample_file = data.loc[0,'lung_mask'].replace('../input/covid19-ct-scans','Covid-19') # 感染mask infection_mask_sample_file = data.loc[0,'infection_mask'].replace('../input/covid19-ct-scans','Covid-19') # 肺部和感染mask lung_and_infection_mask_sample_file = data.loc[0,'lung_and_infection_mask'].replace('../input/covid19-ct-scans','Covid-19') 安装nibabel库读取NIfTI格式的医学影像文件，使用matplotlib库进行可视化展示：!pip install nibabelimport numpy as np import nibabel as nib # 读取nifti文件 def read_nii_file(fileName): img = nib.load(fileName) img_data = img.get_fdata() img_data = np.rot90(np.array(img_data)) return img_data # 读取 ct_scan_imgs = read_nii_file(ct_scan_sample_file) lung_mas_imgs = read_nii_file(lung_mask_sample_file) infection_mask_imgs = read_nii_file(infection_mask_sample_file) lung_and_infection_mas_imgs = read_nii_file(lung_and_infection_mask_sample_file) # 查看大小 print(ct_scan_imgs.shape) print(lung_mas_imgs.shape) (512, 512, 301) (512, 512, 301) # 绘制 import matplotlib.pyplot as plt %matplotlib inline color_map = 'spring' layer_index = 180 fig = plt.figure(figsize=(20, 4)) plt.subplot(1, 4, 1) plt.imshow(ct_scan_imgs[:,:,layer_index], cmap='bone') plt.title('Original Image') plt.axis('off') plt.subplot(1,4,2) plt.imshow(ct_scan_imgs[:,:,layer_index], cmap='bone') mask_ = np.ma.masked_where(lung_mas_imgs[:,:,layer_index]== 0, lung_mas_imgs[:,:,layer_index]) plt.imshow(mask_, alpha=0.8, cmap=color_map) plt.title('Lung Mask') plt.axis('off') plt.subplot(1,4,3) plt.imshow(ct_scan_imgs[:,:,layer_index], cmap='bone') mask_ = np.ma.masked_where(infection_mask_imgs[:,:,layer_index]== 0, infection_mask_imgs[:,:,layer_index]) plt.imshow(mask_, alpha=0.8, cmap=color_map) plt.title('Infection Mask') plt.axis('off') plt.subplot(1,4,4) plt.imshow(ct_scan_imgs[:,:,layer_index], cmap='bone') mask_ = np.ma.masked_where(lung_and_infection_mas_imgs[:,:,layer_index]== 0, lung_and_infection_mas_imgs[:,:,layer_index]) plt.imshow(mask_, alpha=0.8, cmap=color_map) plt.title('Lung and Infection Mask') plt.axis('off') plt.show() 之后对数据进行标准化和归一化，划分训练集和测试集，并统一缩放到256x256的大小保存为npy文件。标准化x′=x−mean(x)σx'= \frac{x-mean(x)}{\sigma} x′=σx−mean(x)​归一化x′=x−min(x)max(x)−min(x)x'= \frac{x-min(x)}{max(x)-min(x)} x′=max(x)−min(x)x−min(x)​# 标准化 def standardize(data): # 计算均值 mean = data.mean() # 计算标准差 std = np.std(data) # 计算结果 standardized = (data - mean) / std return standardized # 归一化 def normalize(data): # 计算最大最小值 max_val = data.max() min_val = data.min() normalized = (data - min_val) / (max_val - min_val) return normalized std = standardize(ct_scan_imgs) normalize(std).max(),normalize(std).min() (1.0, 0.0) # 处理所有文件 import cv2 import glob train_file_list =[file_path.replace('../input/covid19-ct-scans','Covid-19') for file_path in data.loc[:,'ct_scan']] train_label_list = [file_path.replace('../input/covid19-ct-scans','Covid-19') for file_path in data.loc[:,'infection_mask']] train_file_list[:5], len(train_label_list), train_label_list[:5], len(train_file_list) (['Covid-19/ct_scans/coronacases_org_001.nii', 'Covid-19/ct_scans/coronacases_org_002.nii', 'Covid-19/ct_scans/coronacases_org_003.nii', 'Covid-19/ct_scans/coronacases_org_004.nii', 'Covid-19/ct_scans/coronacases_org_005.nii'], 20, ['Covid-19/infection_mask/coronacases_001.nii', 'Covid-19/infection_mask/coronacases_002.nii', 'Covid-19/infection_mask/coronacases_003.nii', 'Covid-19/infection_mask/coronacases_004.nii', 'Covid-19/infection_mask/coronacases_005.nii'], 20) from tqdm import tqdm for index in tqdm(range(len(train_file_list))): # 读取 img = nib.load(train_file_list[index]) mask = nib.load(train_label_list[index]) img_data = img.get_fdata() mask_data = mask.get_fdata().astype(np.uint8) # 标准化和归一化 std = standardize(img_data) normalized = normalize(std) # 分为训练数据和测试数据 if index < 17: save_dir = 'processed/train/' else: save_dir = 'processed/test/' # 遍历所有层，分层存入文件夹，存储路径格式：'processed/train/0/img_0.npy'，'processed/train/0/label_0.npy'， layer_num = normalized.shape[-1] for i in range(layer_num): layer = normalized[:,:,i] mask = mask_data[:,:,i] # 缩放 layer = cv2.resize(layer, (256, 256)) mask = cv2.resize(mask, (256, 256), interpolation=cv2.INTER_NEAREST) # 创建文件夹 img_dir = save_dir + str(index) if not os.path.exists(img_dir): os.makedirs(img_dir) # 保存为npy文件 np.save(img_dir+'/img_'+str(i), layer) np.save(img_dir+'/label_'+str(i), mask) 100%|██████████| 20/20 [01:08<00:00, 3.44s/it] 同时采用imgaug库进行数据增强，包括图像的缩放、旋转和弹性变换等操作，以提升模型的泛化能力。!pip install imgaugimport imgaug as ia import imgaug.augmenters as iaa from torch.utils.data import Dataset from imgaug.augmentables.segmaps import SegmentationMapsOnImage class SegmentDataset(Dataset): def __init__(self,where='train',seq=None): # 获取数据 self.img_list = glob.glob('processed/{}/*/img_*'.format(where)) self.mask_list = glob.glob('processed/{}/*/img_*') # 数据增强pipeline self.seq = seq def __len__(self): # 返回数据大小 return len(self.img_list) def __getitem__(self, idx): # 获取具体每一个数据 # 获取图片 img_file = self.img_list[idx] mask_file = img_file.replace('img','label') img = np.load(img_file) # 获取mask mask = np.load(mask_file) # 如果需要数据增强 if self.seq: segmap = SegmentationMapsOnImage(mask, shape=mask.shape) img,mask = seq(image=img, segmentation_maps=segmap) # 直接获取数组内容 mask = mask.get_arr() # 灰度图扩张维度成张量 return np.expand_dims(img,0) , np.expand_dims(mask,0) # 数据增强处理流程 seq = iaa.Sequential([ iaa.Affine(scale=(0.8, 1.2), # 缩放 rotate=(-45, 45)), # 旋转 iaa.ElasticTransformation() # 变换 ]) 创建dataloader，开启8个线程一次加载16张图片进行处理。import torch import torch_npu from torch_npu.contrib import transfer_to_npu # 使用dataloader加载 batch_size = 16 num_workers = 8 train_dataset = SegmentDataset('train', seq) test_dataset = SegmentDataset('test', None) train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, num_workers=num_workers, shuffle=True) test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, num_workers=num_workers, shuffle=False) 我们可以查看单张图像增强后的效果：# 对同一个图片显示多次 fig = plt.figure(figsize=(12, 12)) for i in range(16): plt.subplot(4, 4, i+1) img , mask = train_dataset[101] plt.imshow(img[0], cmap='bone') mask_ = np.ma.masked_where(mask[0]== 0, mask[0]) plt.imshow(mask_, alpha=0.8, cmap="spring") plt.axis('off') plt.show() 数据准备完成后，我们开始从头构建Unet的网络结构。U-Net是一种经典的编码器-解码器结构，特别适用于医学图像分割任务。网络包含四个编码层和四个解码层，通过跳跃连接将编码器的特征图与解码器对应层进行融合，保留了丰富的空间信息。# 定义两次卷积操作 class ConvBlock(torch.nn.Module): def __init__(self,in_channels,out_channels): super().__init__() self.step = torch.nn.Sequential( # 第一次卷积 torch.nn.Conv2d(in_channels=in_channels,out_channels=out_channels,kernel_size=3,padding=1,stride=1), # ReLU torch.nn.ReLU(), # 第二次卷积 torch.nn.Conv2d(in_channels=out_channels,out_channels=out_channels,kernel_size=3,padding=1,stride=1), # ReLU torch.nn.ReLU() ) def forward(self,x): return self.step(x) class UNet(torch.nn.Module): def __init__(self): super().__init__() # 定义左侧编码器的操作 self.layer1 = ConvBlock(1,64) self.layer2 = ConvBlock(64,128) self.layer3 = ConvBlock(128,256) self.layer4 = ConvBlock(256,512) # 定义右侧解码器的操作 self.layer5 = ConvBlock(256+512,256) self.layer6 = ConvBlock(128+256,128) self.layer7 = ConvBlock(64+128,64) #最后一个卷积 self.layer8 = torch.nn.Conv2d(in_channels=64,out_channels=1,kernel_size=1,padding=0,stride=1) # 定一些其他操作 # 池化 self.maxpool = torch.nn.MaxPool2d(kernel_size=2) #上采样 self.upsample = torch.nn.Upsample(scale_factor=2,mode='bilinear') # sigmoid self.sigmoid = torch.nn.Sigmoid() def forward(self,x): # 对输入数据进行处理 # 定义下采样部分 # input:1X256x256, output: 64x256x256 x1 = self.layer1(x) # input:64x256x256, output: 64 x 128 x 128 x1_p = self.maxpool(x1) # input: 64 x 128 x 128 , output: 128 x 128 x 128 x2 = self.layer2(x1_p) # input:128 x 128 x 128 , output: 128 x 64 x 64 x2_p = self.maxpool(x2) # input: 128 x 64 x 64, output: 256 x 64 x 64 x3 = self.layer3(x2_p) #input:256 x 64 x 64, output: 256 x 32 x 32 x3_p = self.maxpool(x3) #input: 256 x 32 x 32, output: 512 x 32 x 32 x4 = self.layer4(x3_p) # 定义上采样 # input: 512 x 32 x 32，output: 512 x 64 x 64 x5 = self.upsample(x4) # 拼接,output: 768x 64 x 64 x5 = torch.cat([x5,x3],dim=1) # input: 768x 64 x 64,output: 256 x 64 x 64 x5 = self.layer5(x5) # input: 256 x 64 x 64,output: 256 x 128 x 128 x6 = self.upsample(x5) # 拼接,output: 384 x 128 x 128 x6 = torch.cat([x6,x2],dim=1) # input: 384 x 128 x 128, output: 128 x 128 x 128 x6 = self.layer6(x6) # input:128 x 128 x 128, output: 128 x 256 x 256 x7 = self.upsample(x6) # 拼接, output: 192 x 256 x256 x7 = torch.cat([x7,x1],dim=1) # input: 192 x 256 x256, output: 64 x 256 x 256 x7 = self.layer7(x7) # 最后一次卷积,input: 64 x 256 x 256, output: 1 x 256 x 256 x8 = self.layer8(x7) #sigmoid # x9= self.sigmoid(x8) return x8网络定义完成后我们可以安装torchsummary库将搭建好的模型可视化。!pip install torchsummary# 模型架构可视化 from torchsummary import summary # device device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") model = UNet().to(device) summary(model,(1, 256, 256)) [W compiler_depend.ts:623] Warning: expandable_segments currently defaults to false. You can enable this feature by `export PYTORCH_NPU_ALLOC_CONF = expandable_segments:True`. (function operator()) ---------------------------------------------------------------- Layer (type) Output Shape Param # ================================================================ Conv2d-1 [-1, 64, 256, 256] 640 ReLU-2 [-1, 64, 256, 256] 0 Conv2d-3 [-1, 64, 256, 256] 36,928 ReLU-4 [-1, 64, 256, 256] 0 ConvBlock-5 [-1, 64, 256, 256] 0 MaxPool2d-6 [-1, 64, 128, 128] 0 Conv2d-7 [-1, 128, 128, 128] 73,856 ReLU-8 [-1, 128, 128, 128] 0 Conv2d-9 [-1, 128, 128, 128] 147,584 ReLU-10 [-1, 128, 128, 128] 0 ConvBlock-11 [-1, 128, 128, 128] 0 MaxPool2d-12 [-1, 128, 64, 64] 0 Conv2d-13 [-1, 256, 64, 64] 295,168 ReLU-14 [-1, 256, 64, 64] 0 Conv2d-15 [-1, 256, 64, 64] 590,080 ReLU-16 [-1, 256, 64, 64] 0 ConvBlock-17 [-1, 256, 64, 64] 0 MaxPool2d-18 [-1, 256, 32, 32] 0 Conv2d-19 [-1, 512, 32, 32] 1,180,160 ReLU-20 [-1, 512, 32, 32] 0 Conv2d-21 [-1, 512, 32, 32] 2,359,808 ReLU-22 [-1, 512, 32, 32] 0 ConvBlock-23 [-1, 512, 32, 32] 0 Upsample-24 [-1, 512, 64, 64] 0 Conv2d-25 [-1, 256, 64, 64] 1,769,728 ReLU-26 [-1, 256, 64, 64] 0 Conv2d-27 [-1, 256, 64, 64] 590,080 ReLU-28 [-1, 256, 64, 64] 0 ConvBlock-29 [-1, 256, 64, 64] 0 Upsample-30 [-1, 256, 128, 128] 0 Conv2d-31 [-1, 128, 128, 128] 442,496 ReLU-32 [-1, 128, 128, 128] 0 Conv2d-33 [-1, 128, 128, 128] 147,584 ReLU-34 [-1, 128, 128, 128] 0 ConvBlock-35 [-1, 128, 128, 128] 0 Upsample-36 [-1, 128, 256, 256] 0 Conv2d-37 [-1, 64, 256, 256] 110,656 ReLU-38 [-1, 64, 256, 256] 0 Conv2d-39 [-1, 64, 256, 256] 36,928 ReLU-40 [-1, 64, 256, 256] 0 ConvBlock-41 [-1, 64, 256, 256] 0 Conv2d-42 [-1, 1, 256, 256] 65 ================================================================ Total params: 7,781,761 Trainable params: 7,781,761 Non-trainable params: 0 ---------------------------------------------------------------- Input size (MB): 0.25 Forward/backward pass size (MB): 706.50 Params size (MB): 29.69 Estimated Total Size (MB): 736.44 ----------------------------------------------------------------random_input = torch.randn(1, 1, 256, 256).to(device) output = model(random_input) output.shapetorch.Size([1, 1, 256, 256]) 最后定义损失函数和优化器，编写模型的训练代码，在昇腾NPU上训练50轮，使用Adam优化器和BCEWithLogitsLoss损失函数，并通过ReduceLROnPlateau调度器动态调整模型的学习率，每轮训练结束后保存模型的最优权重。import time from torch.optim.lr_scheduler import ReduceLROnPlateau # 定义损失 loss_fn = torch.nn.BCEWithLogitsLoss() # 定义优化器 optimizer = torch.optim.Adam(model.parameters(), lr=1e-4) # 动态减少LR scheduler = ReduceLROnPlateau(optimizer, 'min') # 计算测试集的loss def check_test_loss(loader,model): loss = 0 # 不记录梯度 with torch.no_grad(): for i, (x, y) in enumerate(loader): # 图片 x = x.to(device,dtype=torch.float32) # 标签 y = y.to(device,dtype=torch.float32) # 预测值 y_pred = model(x) #计算损失 loss_batch = loss_fn(y_pred, y) loss += loss_batch return loss / len(loader) !pip install tensorboard progressbar# 使用tensorboard记录参数 from torch.utils.tensorboard import SummaryWriter # 使用progressbar打印进度 from progressbar import ProgressBar, Percentage, Bar, Timer, ETA, FileTransferSpeed # 记录变量 writer = SummaryWriter(log_dir='./log') # 打印进度 widgets = ['Progress: ', Percentage(), ' ', Bar('#'), ' ', Timer(), ' ', ETA(), ' ', FileTransferSpeed()] # 训练100个epoch EPOCH_NUM = 50 # 记录最好的测试acc best_test_loss = 10 # 保存训练结果 train_loss_results = [] test_loss_results = [] for epoch in range(EPOCH_NUM): # 获取批次图像 print('Epoch:{}'.format(epoch+1)) start_time = time.time() loss = 0 progress = ProgressBar(widgets=widgets) for (x, y) in progress(train_loader): # ！！！每次update前清空梯度 model.zero_grad() # 获取数据 # 图片 x = x.to(device,dtype=torch.float32) # 标签 y = y.to(device,dtype=torch.float32) # 预测值 y_pred = model(x) #计算损失 loss_batch = loss_fn(y_pred, y) # 计算梯度 loss_batch.backward() optimizer.step() optimizer.zero_grad() # 记录每个batch的train loss loss_batch = loss_batch.detach().cpu() loss += loss_batch # 每个epoch的loss loss = loss / len(train_loader) # 如果降低LR：如果loss连续10个epoch不再下降，就减少LR scheduler.step(loss) # 计算测试集的loss test_loss = check_test_loss(test_loader, model) # tensorboard 记录 Loss/train writer.add_scalar('Loss/train', loss, epoch) # tensorboard 记录 Loss/test writer.add_scalar('Loss/test', test_loss, epoch) #保存信息 train_loss_results.append(loss.item()) test_loss_results.append(test_loss.item()) # 保存最新模型 torch.save(model.state_dict(), 'unet_latest.pt') # 记录最好的测试loss，并保存模型 if best_test_loss > test_loss: print('test loss improved from {:.4f} to {:.4f}'.format(best_test_loss, test_loss.item())) best_test_loss = test_loss # 保存模型 torch.save(model.state_dict(), 'unet_best.pt') Epoch:1 Progress: 0% | | Elapsed Time: 0:00:00 ETA: --:--:-- 0.00 B/s . Progress: 100% |##############| Elapsed Time: 0:00:29 Time: 0:00:29 7.17 B/s test loss improved from 50.0000 to 0.1825 Epoch:2 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.90 B/s test loss improved from 0.1825 to 0.1356 Epoch:3 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.88 B/s test loss improved from 0.1356 to 0.1190 Epoch:4 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.88 B/s test loss improved from 0.1190 to 0.1071 Epoch:5 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.99 B/s test loss improved from 0.1071 to 0.0826 Epoch:6 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.98 B/s test loss improved from 0.0826 to 0.0783 Epoch:7 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.00 B/s Epoch:8 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.01 B/s test loss improved from 0.0783 to 0.0564 Epoch:9 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.01 B/s Epoch:10 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.02 B/s Epoch:11 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.96 B/s Epoch:12 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.00 B/s Epoch:13 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.99 B/s test loss improved from 0.0564 to 0.0546 Epoch:14 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.99 B/s Epoch:15 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.89 B/s Epoch:16 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.99 B/s Epoch:17 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.01 B/s Epoch:18 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.01 B/s test loss improved from 0.0546 to 0.0502 Epoch:19 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.02 B/s Epoch:20 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.00 B/s Epoch:21 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.98 B/s Epoch:22 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.00 B/s test loss improved from 0.0502 to 0.0434 Epoch:23 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.00 B/s Epoch:24 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.98 B/s Epoch:25 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.99 B/s Epoch:26 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.94 B/s Epoch:27 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.84 B/s Epoch:28 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.93 B/s Epoch:29 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.89 B/s Epoch:30 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.96 B/s Epoch:31 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.94 B/s Epoch:32 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.92 B/s Epoch:33 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.88 B/s Epoch:34 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.93 B/s Epoch:35 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.97 B/s Epoch:36 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.92 B/s Epoch:37 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.91 B/s Epoch:38 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.93 B/s Epoch:39 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.91 B/s Epoch:40 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.89 B/s Epoch:41 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.87 B/s Epoch:42 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.96 B/s Epoch:43 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.97 B/s Epoch:44 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.97 B/s test loss improved from 0.0434 to 0.0378 Epoch:45 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.96 B/s Epoch:46 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.91 B/s Epoch:47 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.00 B/s Epoch:48 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.99 B/s Epoch:49 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 7.98 B/s Epoch:50 Progress: 100% |##############| Elapsed Time: 0:00:26 Time: 0:00:26 8.00 B/s 在模型的训练过程中，我们可以在另一个终端中查看NPU的利用率：训练结束后我们使用matplotlib绘制模型的损失曲线，观察在训练过程中loss的收敛情况。plt.plot(range(1, len(train_loss_results)+1), train_loss_results, label='train_loss') plt.plot(range(1, len(test_loss_results)+1), test_loss_results, label='test_loss') plt.title('Model Loss') plt.legend() 这里我们加载模型在训练过程中的最优权重在测试集上评估模型的训练效果：model.load_state_dict(torch.load('unet_best.pt')) model.eval() UNet( (layer1): ConvBlock( (step): Sequential( (0): Conv2d(1, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU() (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (3): ReLU() ) ) (layer2): ConvBlock( (step): Sequential( (0): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU() (2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (3): ReLU() ) ) (layer3): ConvBlock( (step): Sequential( (0): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU() (2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (3): ReLU() ) ) (layer4): ConvBlock( (step): Sequential( (0): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU() (2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (3): ReLU() ) ) (layer5): ConvBlock( (step): Sequential( (0): Conv2d(768, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU() (2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (3): ReLU() ) ) (layer6): ConvBlock( (step): Sequential( (0): Conv2d(384, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU() (2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (3): ReLU() ) ) (layer7): ConvBlock( (step): Sequential( (0): Conv2d(192, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (1): ReLU() (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1)) (3): ReLU() ) ) (layer8): Conv2d(64, 1, kernel_size=(1, 1), stride=(1, 1)) (maxpool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False) (upsample): Upsample(scale_factor=2.0, mode='bilinear') (sigmoid): Sigmoid() ) class SegmentDataset(Dataset): def __init__(self,where='train',seq=None): # 获取数据 self.img_list =natsorted(glob.glob('processed/{}/*/img_*'.format(where))) self.mask_list =natsorted( glob.glob('processed/{}/*/img_*') ) # 数据增强pipeline self.seq = seq def __len__(self): # 返回数据大小 return len(self.img_list) def __getitem__(self, idx): # 获取具体每一个数据 # 获取图片 img_file = self.img_list[idx] mask_file = img_file.replace('img','label') img = np.load(img_file) # 获取mask mask = np.load(mask_file) # 如果需要数据增强 if self.seq: segmap = SegmentationMapsOnImage(mask, shape=mask.shape) img,mask = seq(image=img, segmentation_maps=segmap) # 直接获取数组内容 mask = mask.get_arr() # 灰度图扩张维度成张量 return np.expand_dims(img,0) , np.expand_dims(mask,0) !pip install natsortfrom natsort import natsorted # 使用dataloader加载 batch_size = 12 num_workers = 8 test_dataset = SegmentDataset('test',None) test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, num_workers=num_workers, shuffle=False) !pip install celluloidfrom tqdm import tqdm from celluloid import Camera from IPython.display import Image # 将每层画面制作成视频 fig = plt.figure(figsize=(10, 10)) camera = Camera(fig) # 遍历所有数据 index = 0 for x, y in tqdm(test_dataset): # 输出输入 input = torch.tensor([x]).to(device,dtype=torch.float32) # 推理 y_pred = model(input) # 获取mask mask_data = (y_pred.detach().cpu().numpy()[0][0] > 0.5) plt.subplot(1, 2, 1) plt.imshow(x[0], cmap='bone') mask_ = np.ma.masked_where(y[0] == 0, y[0]) plt.imshow(mask_, alpha=0.8, cmap="spring") plt.title('truth') plt.axis('off') plt.subplot(1, 2, 2) plt.imshow(x[0], cmap='bone') mask_ = np.ma.masked_where(mask_data == 0, mask_data) plt.imshow(mask_, alpha=0.8, cmap="spring") plt.title('prediction') plt.axis('off') camera.snap() index +=1 if index > 500: break animation = camera.animate() 0%| | 0/177 [00:00<?, ?it/s]/home/service/.local/lib/python3.10/site-packages/torch_npu/contrib/transfer_to_npu.py:151: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at /pytorch/torch/csrc/utils/tensor_new.cpp:261.) return fn(*args, **kwargs) 100%|██████████| 177/177 [00:06<00:00, 29.07it/s] # convert the animation to a video animation.save('animation.gif', writer='imagemagick') Image(open('animation.gif','rb').read()) 可以看到，模型准确地分割出肺炎的病变区域，为医疗诊断提供决策支持。本案例充分体现了华为云昇腾AI平台在医学图像处理领域的强大计算能力和易用性，为医疗AI应用的开发提供了完整的实践范例。

破解目标检测中的“隐形杀手”：样本失衡难题攻克之道在目标检测领域，样本失衡是困扰工程师和研究者的“隐形杀手”。它普遍存在于现实场景中：背景区域远多于目标区域、常见类别目标（如行人、车辆）的样本量远超稀有类别（如交通锥、特定动物）。这种失衡会导致模型严重偏向多数类，对稀有目标的检测性能急剧下降，直接影响模型的实用价值。那么，如何有效攻克这一难题？以下是一些关键策略：数据层面：开源与增流重采样： 核心思路是调整训练数据的分布。过采样： 复制稀有目标样本或应用高级技巧（如SMOTE及其变种）在特征空间合成新样本。需警惕单纯复制导致的过拟合。欠采样： 随机或有策略地减少多数类样本（如背景或大类别）。优点是加速训练，但可能损失重要信息。通常结合其他方法使用。数据增强： 对少数类样本应用更积极、多样化的增强（旋转、缩放、裁剪、色彩变换、MixUp、CutMix等），显著增加其“曝光度”和多样性，提高模型鲁棒性。算法层面：加权与重构损失函数改造：Focal Loss： 这是针对样本失衡的里程碑式解决方案。它通过降低易分类样本（通常是多数类）的权重，让模型更关注难分类样本（通常是少数类或困难正样本），有效缓解失衡问题。代价敏感学习： 在损失函数中为不同类别的错误预测分配不同的惩罚权重。通常对误检稀有目标的惩罚远大于误检背景，迫使模型重视小类。模型结构调整： 针对高度失衡问题（如背景占绝对主导），可考虑两阶段策略（先粗筛疑似目标区域，再精细分类）或引入在线困难样本挖掘机制，让模型在训练中自动聚焦于那些分类困难的样本（常包含小目标）。评估与调优：洞察全局选用合适的评估指标： 单纯追求高mAP可能掩盖问题。需特别关注小类、稀有类的精确率、召回率以及F1-Score。mAP@0.5:0.95、P-R曲线能更全面反映模型在类别不平衡下的性能。消融实验： 严格测试不同策略（单独或组合）的效果，记录它们对各类别性能的影响，找到最优解。实用技巧：知己知彼： 开始前，务必可视化数据集，精确掌握每个类别的样本数量和分布特点。组合拳出击： 单一方法往往效果有限。结合使用数据增强（针对小类）、Focal Loss（或加权损失） 是目前最常见有效的组合。持续监控： 在验证集和测试集上，持续跟踪各个类别的单独表现，特别是那些你关心的关键少数类。拥抱新技术： Few-Shot Learning、自监督预训练、域适应等技术在缓解小样本问题方面展现出潜力，值得持续关注。小结：攻克目标检测样本失衡，本质上是对数据进行深度理解和智慧干预。需要在数据的源头（采样、增强）、模型学习的核心（损失函数）以及最终的评估环节（指标选择）进行系统性优化。没有“银弹”，但通过科学组合上述策略，并辅以细致的分析调优，我们完全能够显著提升模型在真实、复杂场景下的均衡检测能力，让“隐形杀手”无所遁形。

视觉多模态模型切分检测和边缘推理一、NanoOWL + SAHINanoOWL（边缘实时开放词汇目标检测模型）基于Vision Transformer架构，结合CLIP的图文对齐能力，可通过文本查询在图像中检测任意类别目标。我们可以结合SAHI框架使用“文本提示+图像切分”在Jetson Orin等嵌入式设备上实现低空目标的多模态检测和TensorRT推理。二、代码实现首先我们拉取官方代码仓库https://github.com/dusty-nv/jetson-containers并运行安装命令install.sh：git clone https://github.com/dusty-nv/jetson-containers bash jetson-containers/install.sh之后使用jetson-containers run和autotag命令自动提取并构建兼容的容器：jetson-containers run --workdir /opt/nanoowl $(autotag nanoowl) 我们在终端中查看容器的ID并将容器中nanoowl拷贝到自定义目录下/home/vsuav/workspace/vit：sudo docker ps -asudo docker cp af063d738879:/opt/nanoowl /home/vsuav/workspace/vit将/home/vsuav/workspace/vit/nanoowl目录及其内部所有文件和子目录的所有者和所属组都设置为当前登录用户，从而确保我们可以正常访问和修改这些文件。sudo chown -R $(whoami):$(whoami) /home/vsuav/workspace/vit/nanoowl运行jetson-containers run命令并指定--workdir参数，将nanoowl目录挂载到容器中，设置容器的名称为NanoOWL：jetson-containers run -v /home/vsuav/workspace/vit/nanoowl:/opt/nanoowl --name NanoOWL --workdir /opt/nanoowl $(autotag nanoowl) 运行docker start NanoOWL启动容器并进入容器的终端：sudo docker start NanoOWL sudo docker exec -it NanoOWL bash 我们在容器内部使用pip3安装python库sahi并创建main.pypip3 install sahi -i https://pypi.tuna.tsinghua.edu.cn/simple我们在examples/owl_predict.py基础上添加SAHI切分检测的逻辑，将无人机拍摄的高清大图切分为640x640和1280x1280的子图并叠加原图和类别名称的编码信息送入模型进行预测，最后把推理结果映射到原图上使用GreedyNMM进行合并后处理，完整代码如下：import os import cv2 import PIL.Image import numpy as np from sahi.slicing import get_slice_bboxes from nanoowl.owl_predictor import OwlPredictor from sahi.postprocess.utils import ObjectPrediction from sahi.postprocess.combine import GreedyNMMPostprocess class OWL_SAHI: def __init__(self, model_path, label_list, OBJ_THRESH, NMS_THRESH, overlap_ratio, slice, slice_scales): self.model_path = model_path if label_list != []: self.label_list = label_list else: self.label_list = [""] self.OBJ_THRESH = OBJ_THRESH self.NMS_THRESH = NMS_THRESH self.overlap_ratio = overlap_ratio self.slice = slice self.slice_scales = slice_scales self.predictor = OwlPredictor( "google/owlvit-base-patch32", image_encoder_engine = self.model_path ) self.text_encodings = self.predictor.encode_text(self.label_list) self.postprocess = GreedyNMMPostprocess( match_threshold = self.NMS_THRESH, match_metric = "IOS", class_agnostic = False, ) def getImageSlices(self, image): img_height, img_width = image.shape[:2] slice_bboxes = [] if self.slice: for slice_scale in self.slice_scales: slice_bboxe = get_slice_bboxes( image_height = img_height, image_width = img_width, auto_slice_resolution = True, slice_height = slice_scale[1], slice_width = slice_scale[0], overlap_height_ratio = self.overlap_ratio, overlap_width_ratio = self.overlap_ratio, ) slice_bboxes.extend(slice_bboxe) slice_bboxes.append([0, 0, img_width, img_height]) else: slice_bboxes = [[0, 0, img_width, img_height]] img_batch = [] for bbox in slice_bboxes: l, t, r, b = bbox img_batch.append(image[t:b, l:r]) return img_batch, slice_bboxes def predict(self, image_path): image = cv2.imread(image_path) image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) img_batch, slice_bboxes = self.getImageSlices(image) all_boxes = [] for img, slice_bbox in zip(img_batch, slice_bboxes): img_pil = PIL.Image.fromarray(img) output = self.predictor.predict( image = img_pil, text = self.label_list, text_encodings = self.text_encodings, threshold = self.OBJ_THRESH, pad_square = True ) boxes = output.boxes.cpu().numpy() if boxes.shape[0] > 0: boxes[:, 0] = boxes[:, 0] + slice_bbox[0] boxes[:, 1] = boxes[:, 1] + slice_bbox[1] boxes[:, 2] = boxes[:, 2] + slice_bbox[0] boxes[:, 3] = boxes[:, 3] + slice_bbox[1] boxes = boxes.astype(np.int32).tolist() for i in range(len(boxes)): obj_item = ObjectPrediction( bbox = boxes[i], score = float(output.scores[i]), category_id = int(output.labels[i]) ) all_boxes.append(obj_item) if len(all_boxes) > 0: all_boxes = self.postprocess(all_boxes) return all_boxes if __name__ == "__main__": model_path = "./data/owl_image_encoder_patch32.engine" label_list = ["car", "tower"] OBJ_THRESH = 0.1 NMS_THRESH = 0.5 overlap_ratio = 0.25 slice = True slice_scales = [[640, 640], [1280, 1280]] owl_sahi = OWL_SAHI(model_path, label_list, OBJ_THRESH, NMS_THRESH, overlap_ratio, slice, slice_scales) images = os.listdir("images") for image_file in images: print(image_file) image_path = os.path.join("images", image_file) all_boxes_processed = owl_sahi.predict(image_path) image = cv2.imread(image_path) for box in all_boxes_processed: xmin, ymin, xmax, ymax = box.bbox.to_xyxy() score = box.score.value clsse = box.category.id cv2.rectangle(image, (xmin, ymin), (xmax, ymax), (0, 255, 0), 4) cv2.putText(image, '{0} {1:.2f}'.format(label_list[clsse], score), (xmin, ymin - 10), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 0, 255), 4, cv2.LINE_AA) cv2.imwrite(f"output/{image_file}", image) 三、小结本文介绍了基于NanoOWL和SAHI框架的视觉多模态模型切分检测方案，通过结合文本提示与图像切分技术，在Jetson Orin等边缘设备上实现开放词汇目标检测。该方案利用Vision Transformer架构和CLIP图文对齐能力，支持任意类别目标检测，并通过SAHI进行图像切片处理与后处理合并，提升检测精度与效率，适用于低空无人机目标检测等边缘推理场景。

【朝推夜训】松材线虫病高清图片切分检测我们以 Jetson Orin Nano 为例，介绍如何使用Python在资源受限的嵌入式设备上实现高清大图切分检测。一、模型导出1. 安装 Cmake，创建 24G swap 空间模型导出时依赖更高版本的Cmake，这里我们直接编译安装：sudo apt update sudo apt install libssl-dev git clone -b v3.25.1 https://github.com/Kitware/CMake.git cd CMake ./bootstrap && make && sudo make install cmake --version交换空间是操作系统用来拓展可用内存的一种机制，可以在内存不足的情况下继续运行，避免程序崩溃或者系统卡死，但是交换空间的访问速度远低于物理内存！禁用Jetson设备上的ZRAM交换配置：ZRAM会将内存页面压缩并存储在内存中，以减少对磁盘的依赖。sudo systemctl disable nvzramconfig使用fallocate创建一个24GB大小的文件，位于/var/24GB.swap路径。sudo fallocate -l 24G /var/24GB.swap设置交换空间格式sudo mkswap /var/24GB.swap启用交换空间sudo swapon /var/24GB.swap永久自启交换空间echo "/var/24GB.swap none swap sw 0 0" | sudo tee -a /etc/fstab重启系统后，系统交换空间增加至24GB：sudo reboot 2. 安装 ultralytics，创建 TensorRT 软连接pip install ultralytics pip install tqdm pandas pip install onnx==1.12.0 onnxslim==0.1.65 protobuf==3.20.1 pip install onnx-simplifier==0.3.10 pip install /home/vsuav/Downloads/onnxruntime_gpu-1.12.1-cp38-cp38-linux_aarch64.whl导出TensorRT模型时依赖其Python安装包，一般在系统Python目录下，以Jetson Orin Nano为例，我们可以建立软连接指向TensorRT安装路径：sudo ln -s /usr/lib/python3.8/dist-packages/tensorrt* /home/vsuav/miniconda3/envs/py38/lib/python3.8/site-packages3. 导出 TensorRT FP16 精度的模型from ultralytics import YOLO model = YOLO("yolov8-1_640x640_amd64_fp32.pt") model.export( format="engine", workspace=4, imgsz=640, half=True, device=0, batch=1 ) 二、切分检测1. 安装 SAHI 库pip install sahi==0.11.18这里我们使用0.11.18版本，修改/home/vsuav/miniconda3/envs/py38/lib/python3.8/site-packages/sahi/models/yolov8.py文件，注释掉第33行代码使其能够加载导出的Engine模型。class Yolov8DetectionModel(DetectionModel): def check_dependencies(self) -> None: check_requirements(["ultralytics"]) def load_model(self): """ Detection model is initialized and set to self.model. """ from ultralytics import YOLO try: model = YOLO(self.model_path) # model.to(self.device) self.set_model(model) except Exception as e: raise TypeError("model_path is not a valid yolov8 model path: ", e) 2. 运行检测代码加载模型import cv2 import numpy as np import matplotlib.pyplot as plt from sahi import AutoDetectionModel from sahi.predict import get_sliced_prediction detection_model = AutoDetectionModel.from_pretrained( model_type='yolov8', model_path="yolov8-1_640x640_amd64_fp16.engine", confidence_threshold=0.45 ) WARNING ⚠️ Unable to automatically guess model task, assuming 'task=detect'. Explicitly define task for your model, i.e. 'task=detect', 'segment', 'classify','pose' or 'obb'. Loading yolov8-1_640x640_amd64_fp16.engine for TensorRT inference... [09/04/2025-11:27:15] [TRT] [I] Loaded engine size: 51 MiB [09/04/2025-11:27:17] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +616, GPU +757, now: CPU 1052, GPU 5537 (MiB) [09/04/2025-11:27:17] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +49, now: CPU 0, GPU 49 (MiB) [09/04/2025-11:27:17] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +29, now: CPU 1001, GPU 5519 (MiB) [09/04/2025-11:27:18] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +27, now: CPU 0, GPU 76 (MiB) 图片切分检测image_path = "28c56c0b-3ff7-4997-88b3-5a8330f7ea88.jpeg" result = get_sliced_prediction( image_path, detection_model, slice_height = 640, slice_width = 640, overlap_height_ratio = 0.2, overlap_width_ratio = 0.2, perform_standard_pred = True, postprocess_class_agnostic = True, postprocess_match_threshold = 0.55, ) Performing prediction on 6 slices. Loading yolov8-1_640x640_amd64_fp16.engine for TensorRT inference... [09/04/2025-11:27:20] [TRT] [I] The logger passed into createInferRuntime differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value. [09/04/2025-11:27:20] [TRT] [I] Loaded engine size: 51 MiB [09/04/2025-11:27:20] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +32, now: CPU 1581, GPU 6577 (MiB) [09/04/2025-11:27:20] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +50, now: CPU 0, GPU 126 (MiB) [09/04/2025-11:27:20] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +6, now: CPU 1530, GPU 6537 (MiB) [09/04/2025-11:27:20] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +1, GPU +27, now: CPU 1, GPU 153 (MiB) 导出检测结果result.export_visuals(export_dir="output/", file_name="sliced_result") result_img_split = cv2.imread("output/sliced_result.png") plt.imshow(result_img_split[:, :, ::-1]) plt.axis('off') plt.show() 三、小结该方案成功在Jetson Orin Nano设备上运行，能够有效处理高清大图的松材线虫病检测任务，在保证检测精度的同时充分利用了嵌入式设备的硬件资源,为林业病虫害防治提供了实用的技术方案。

C++ TensorRT YOLOv8-SAHI 高性能部署指南项目介绍本项目将介绍如何在Jetson等嵌入式设备上实现YOLOv8-SAHI的高性能部署，特别是使用Int8引擎的优化方案。在Jetson Orin Nano (8GB)设备上，图像切片和批量推理的测试时间消耗小于0.05秒，对1080p视频进行切分检测和bytetrack跟踪性能接近15FPS。# 代码仓库： https://github.com/HouYanSong/tensorrtx-yolov8-sahi导出 YOLOv8 Int8 量化模型我们固定输入图像尺寸1440x1080进行切分，其中每张切分子图的大小为640x640重叠度>20%，加上原始图像一次推理对8张图像进行检测，导出Int8量化后BatchSize=8的模型。从yolov8.pt生成yolov8s.wts权重文件pip install ultralytics python gen_wts.py从yolov8s.wts导出yolov8s.engine引擎文件，BatchSize大小为8sudo apt install libeigen3-devrm -fr build cmake -S . -B build cmake --build build cd build ./yolov8_sahi -s ../weights/yolov8s.wts ../weights/yolov8s.engine s模型的参数配置模型的配置文件为include/config.h，这里我们使用yolov8s官方预训练模型，模型的输入大小为640x640总共有80个类别，并且设置模型的kBatchSize = 8，一次最多可8以推理8张图像，指定量化图片的路径导出Int8量化后的模型。#ifndef CONFIG_H #define CONFIG_H // #define USE_FP16 #define USE_INT8 #include <string> #include <vector> const static char *kInputTensorName = "images"; const static char *kOutputTensorName = "output"; const static int kNumClass = 80; const static int kBatchSize = 8; const static int kGpuId = 0; const static int kInputH = 640; const static int kInputW = 640; const static float kNmsThresh = 0.55f; const static float kConfThresh = 0.45f; const static int kMaxInputImageSize = 3000 * 3000; const static int kMaxNumOutputBbox = 1000; const std::string trtFile = "../weights/yolov8s.engine"; const std::string cacheFile = "./int8calib.table"; const std::string calibrationDataPath = "../images/"; // 存放用于 int8 量化校准的图像 const std::vector<std::string> vClassNames { "person", "bicycle", "car", "motorcycle", "airplane", "bus", "train", "truck", "boat", "traffic light", "fire hydrant", "stop sign", "parking meter", "bench", "bird", "cat", "dog", "horse", "sheep", "cow", "elephant", "bear", "zebra", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite", "baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife", "spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza", "donut", "cake", "chair", "couch", "potted plant", "bed", "dining table", "toilet", "tv", "laptop", "mouse", "remote", "keyboard", "cell phone", "microwave", "oven", "toaster", "sink", "refrigerator", "book", "clock", "vase", "scissors", "teddy bear", "hair drier", "toothbrush" }; #endif // CONFIG_H YOLOv8-SAHI 切分检测为了验证量化后模型精度以及Batch推理的性能，这里我们使用Int8量化后的模型直接对量化图片进行切分检测，推理命令如下：cd build ./yolov8_sahi -d ../weights/yolov8s.engine ../images/在Jetson Orin Nano (8GB)上使用Int8引擎的YOLOv8-SAHI性能表现如下：sample0102.png YOLOv8-SAHI: 1775ms sample0206.png YOLOv8-SAHI: 46ms sample0121.png YOLOv8-SAHI: 44ms sample0058.png YOLOv8-SAHI: 44ms sample0070.png YOLOv8-SAHI: 44ms sample0324.png YOLOv8-SAHI: 43ms sample0122.png YOLOv8-SAHI: 44ms sample0086.png YOLOv8-SAHI: 45ms sample0124.png YOLOv8-SAHI: 45ms sample0230.png YOLOv8-SAHI: 45ms ...可以看到模型对单张图片的推理时间小于0.5毫秒，可以达到实时检测的要求。YOLOv8-SAHI-ByteTrack 视频跟踪我们可以结合ByteTrack跟踪算法对视频文件进行实时的切分检测和跟踪，在build目录下执行：cd build ./yolov8_sahi_track ../media/c3_1080.mp4 在Jetson Orin Nano (8GB)上YOLOv8-SAHI-ByteTrack性能表现如下：Total frames: 341 Init ByteTrack! Processing frame 20 (8 fps) Processing frame 40 (11 fps) Processing frame 60 (12 fps) Processing frame 80 (12 fps) Processing frame 100 (13 fps) Processing frame 120 (13 fps) Processing frame 140 (13 fps) Processing frame 160 (14 fps) Processing frame 180 (14 fps) Processing frame 200 (14 fps) Processing frame 220 (14 fps) Processing frame 240 (14 fps) Processing frame 260 (14 fps) Processing frame 280 (14 fps) Processing frame 300 (14 fps) Processing frame 320 (14 fps) Processing frame 340 (15 fps) FPS: 15 可以看到模型在1080p的视频上切分检测的帧率接近15FPS，并且ByteTrack的跟踪效果非常优秀。小结通过本项目，开发者可以在资源受限的嵌入式设备上实现高效的YOLOv8切分检测和跟踪，特别适用于需要实时处理的边缘计算场景。

猫脸关键点检测（ModelBox）一、模型训练与转换ResNet50V2是改进版的深度卷积神经网络，基于 ResNet 架构发展而来。它采用前置激活（将 BN 和 ReLU 移至卷积前）与身份映射，优化了信息传播和模型训练性能。作为 50 层深度的网络，ResNet50V2 广泛应用于图像分类、目标检测等任务，支持迁移学习，适合快速适配新数据集，具有良好的泛化能力和较高准确率。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、应用开发1. 创建工程在ModelBox sdk目录下使用create.bat创建ResNet50V2工程：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t server -n ResNet50V2 ... success: create ResNet50V2 in D:\modelbox-win10-x64-1.5.3\workspacecreate.bat工具的参数中，-t参数，表示所创建实例的类型，包括server（ModelBox工程）、python（Python功能单元）、c++（C++功能单元）、infer（推理功能单元）等；-n参数，表示所创建实例的名称；-s参数，表示将使用后面参数值代表的模板创建工程，而不是创建空的工程。2. 创建推理功能单元在ModelBox sdk目录下使用create.bat创建resnet50v2_infer推理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t infer -n resnet50_infer -p ResNet50V2 ... success: create infer resnet50_infer in D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2/model/resnet50_infercreate.bat工具使用时，-t infer即表示创建的是推理功能单元；-n xxx_infer表示创建的功能单元名称为xxx_infer；-p表示所创建的功能单元属于ResNet50V2应用。下载转换好的ResNet50V2.onnx模型到ResNet50V2\model目录下，修改推理功能单元resnet50v2_infer.toml模型的配置文件：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [base] name = "resnet50_infer" device = "cpu" version = "1.0.0" description = "your description" entry = "./ResNet50V2.onnx" # model file path, use relative path type = "inference" virtual_type = "onnx" # inference engine type: win10 now only support onnx group_type = "Inference" # flowunit group attribution, do not change # Input ports description [input] [input.input1] # input port number, Format is input.input[N] name = "Input" # input port name type = "float" # input port data type ,e.g. float or uint8 device = "cpu" # input buffer type: cpu, win10 now copy input from cpu # Output ports description [output] [output.output1] # output port number, Format is output.output[N] name = "Output" # output port name type = "float" # output port data type ,e.g. float or uint83. 创建后处理功能单元在ModelBox sdk目录下使用create.bat创建resnet50v2_post后处理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t python -n resnet50v2_post -p ResNet50V2 ... success: create python resnet50v2_post in D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2/etc/flowunit/resnet50v2_postcreate.bat工具使用时，-t python即表示创建的是通用功能单元；-n xxx_post表示创建的功能单元名称为xxx_post；-p表示所创建的功能单元属于ResNet50V2应用。a. 修改配置文件我们的模型有一个输入和输出，总共包含猫脸的9个关键点：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. # Basic config [base] name = "resnet50v2_post" # The FlowUnit name device = "cpu" # The flowunit runs on cpu version = "1.0.0" # The version of the flowunit type = "python" # Fixed value, do not change description = "description" # The description of the flowunit entry = "resnet50v2_post@resnet50v2_postFlowUnit" # Python flowunit entry function group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ... # Flowunit Type stream = false # Whether the flowunit is a stream flowunit condition = false # Whether the flowunit is a condition flowunit collapse = false # Whether the flowunit is a collapse flowunit collapse_all = false # Whether the flowunit will collapse all the data expand = false # Whether the flowunit is a expand flowunit # The default Flowunit config [config] keypoints = 9 # Input ports description [input] [input.input1] # Input port number, the format is input.input[N] name = "in_feat" # Input port name type = "float" # Input port type # Output ports description [output] [output.output1] # Output port number, the format is output.output[N] name = "out_data" # Output port name type = "string" # Output port typeb. 修改逻辑代码# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. #!/usr/bin/env python # -*- coding: utf-8 -*- import _flowunit as modelbox import numpy as np import json class resnet50v2_postFlowUnit(modelbox.FlowUnit): # Derived from modelbox.FlowUnit def __init__(self): super().__init__() def open(self, config): # Open the flowunit to obtain configuration information self.params = {} self.params['keypoints'] = config.get_int('keypoints') return modelbox.Status.StatusCode.STATUS_SUCCESS def process(self, data_context): # Process the data in_data = data_context.input("in_feat") out_data = data_context.output("out_data") # resnet50v2_post process code. # Remove the following code and add your own code here. for buffer_feat in in_data: feat_data = np.array(buffer_feat.as_object(), copy=False) keypoints = feat_data.reshape(-1, 2).tolist() result = {"keypoints": keypoints} result_str = json.dumps(result) out_buffer = modelbox.Buffer(self.get_bind_device(), result_str) out_data.push_back(out_buffer) return modelbox.Status.StatusCode.STATUS_SUCCESS def close(self): # Close the flowunit return modelbox.Status() def data_pre(self, data_context): # Before streaming data starts return modelbox.Status() def data_post(self, data_context): # After streaming data ends return modelbox.Status() def data_group_pre(self, data_context): # Before all streaming data starts return modelbox.Status() def data_group_post(self, data_context): # After all streaming data ends return modelbox.Status() 4. 修改应用的流程图ResNet50V2工程graph目录下存放流程图，默认的流程图ResNet50V2.toml与工程同名：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [driver] dir = ["${HILENS_APP_ROOT}/etc/flowunit", "${HILENS_APP_ROOT}/etc/flowunit/cpp", "${HILENS_APP_ROOT}/model", "${HILENS_MB_SDK_PATH}/flowunit"] skip-default = true [profile] profile=false trace=false dir="${HILENS_DATA_DIR}/mb_profile" [graph] format = "graphviz" graphconf = """digraph ResNet50V2 { node [shape=Mrecord] queue_size = 4 batch_size = 1 input1[type=input,flowunit=input,device=cpu,deviceid=0] httpserver_sync_receive[type=flowunit, flowunit=httpserver_sync_receive_v2, device=cpu, deviceid=0, time_out_ms=5000, endpoint="http://0.0.0.0:1234/v1/ResNet50V2", max_requests=100] image_decoder[type=flowunit, flowunit=image_decoder, device=cpu, key="image_base64", queue_size=4] image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=224, image_height=224] normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="0.003921568627450,0.003921568627450,0.003921568627450"] resnet50v2_infer[type=flowunit, flowunit=resnet50v2_infer, device=cpu, deviceid=0, batch_size=1] resnet50v2_post[type=flowunit, flowunit=resnet50v2_post, device=cpu, deviceid=0] httpserver_sync_reply[type=flowunit, flowunit=httpserver_sync_reply_v2, device=cpu, deviceid=0] input1:input -> httpserver_sync_receive:in_url httpserver_sync_receive:out_request_info -> image_decoder:in_encoded_image image_decoder:out_image -> image_resize:in_image image_resize:out_image -> normalize:in_data normalize:out_data -> resnet50v2_infer:Input resnet50v2_infer:Output -> resnet50v2_post:in_feat resnet50v2_post:out_data -> httpserver_sync_reply:in_reply_info }""" [flow] desc = "ResNet50V2 run in modelbox-win10-x64" 在命令行中运行.\create.bat -t editor即可打开ModelBox图编排界面，可以实时修改并查看项目的流程图：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t editor5. 运行应用在ResNet50V2工程目录下执行.\bin\main.bat运行应用：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2 PS D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2> .\bin\main.bat在ResNet50V2工程data目录下新建test_http.py测试脚本：#!/usr/bin/env python # -*- coding: utf-8 -*- # Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. import os import cv2 import json import base64 import http.client class HttpConfig: '''http调用的参数配置''' def __init__(self, host_ip, port, url, img_base64_str): self.hostIP = host_ip self.Port = port self.httpMethod = "POST" self.requstURL = url self.headerdata = { "Content-Type": "application/json" } self.test_data = { "image_base64": img_base64_str } self.body = json.dumps(self.test_data) def read_image(img_path): '''读取图片数据并转为base64编码的字符串''' img_data = cv2.imread(img_path) img_data = cv2.cvtColor(img_data, cv2.COLOR_BGR2RGB) img_str = cv2.imencode('.jpg', img_data)[1].tobytes() img_bin = base64.b64encode(img_str) img_base64_str = str(img_bin, encoding='utf8') return img_data, img_base64_str def test_image(img_path, ip, port, url): '''单张图片测试''' img_data, img_base64_str = read_image(img_path) http_config = HttpConfig(ip, port, url, img_base64_str) conn = http.client.HTTPConnection(host=http_config.hostIP, port=http_config.Port) conn.request(method=http_config.httpMethod, url=http_config.requstURL, body=http_config.body, headers=http_config.headerdata) response = conn.getresponse().read().decode() print('response: ', response) result = json.loads(response) w, h = img_data.shape[1], img_data.shape[0] for x, y in result["keypoints"]: if x > 0 and y > 0: cv2.circle(img_data, (int(x * w), int(y * h)), 5, (0, 255, 0), -1) cv2.imwrite('./result-' + os.path.basename(img_path), img_data[..., ::-1]) if __name__ == "__main__": port = 1234 ip = "127.0.0.1" url = "/v1/ResNet50V2" img_folder = './test_imgs' file_list = os.listdir(img_folder) for img_file in file_list: print("\n================ {} ================".format(img_file)) img_path = os.path.join(img_folder, img_file) test_image(img_path, ip, port, url) 在ResNet50V2工程data目录下新建test_imgs文件夹存放测试图片：在另一个终端中进入ResNet50V2工程目录data文件夹下运行test_http.py脚本发起HTTP请求测试：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2\data PS D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe .\test_http.py ================ 2256.jpg ================ response: {"keypoints": [[0.19147011637687683, 0.26770520210266113], [0.29639703035354614, 0.26533427834510803], [0.24554343521595, 0.35762542486190796], [0.11009970307350159, 0.2090619057416916], [0.08408773690462112, 0.09547536075115204], [0.17451311647891998, 0.169035404920578], [0.2880205512046814, 0.168979212641716], [0.3739408254623413, 0.0717596635222435], [0.34669068455696106, 0.20229394733905792]]} ================ 6899.jpg ================ response: {"keypoints": [[0.3829421401023865, 0.41393953561782837], [0.47102952003479004, 0.42683106660842896], [0.4321300983428955, 0.5082458853721619], [0.3185971677303314, 0.36286458373069763], [0.33502572774887085, 0.2243150770664215], [0.3852037489414215, 0.29658034443855286], [0.4819968640804291, 0.30954840779304504], [0.5504774451255798, 0.2711380124092102], [0.5290539264678955, 0.3962092399597168]]} 在ResNet50V2工程data目录下即可查看测试图片的推理结果：三、小结本节介绍了如何使用ModelArts和ModelBox训练开发一个ResNet50V2猫脸关键点检测的AI应用，我们只需要准备模型文件以及简单的配置即可创建一个HTTP服务。同时我们可以了解到ResNet50V2网络的基本结构、数据处理和模型训练方法，以及对应推理应用的逻辑。----转自博客：https://bbs.huaweicloud.com/blogs/451999

果蔬病虫害分割（ModelBox）一、模型训练与转换FCN（全卷积网络，Fully Convolutional Networks）是用于语义分割任务的一种深度学习模型架构，引入了跳跃结构（Skip Architecture），通过融合浅层和深层的特征图，保留更多的细节信息，提升分割精度。此外，FCN还利用多尺度上下文聚合，捕捉不同层级的特征，增强了对不同大小目标的识别能力。FCN的成功推动了语义分割领域的发展，成为后续许多先进模型的基础。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、应用开发1. 创建工程在ModelBox sdk目录下使用create.bat创建FCN工程：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t server -n FCN ... success: create FCN in D:\modelbox-win10-x64-1.5.3\workspacecreate.bat工具的参数中，-t参数，表示所创建实例的类型，包括server（ModelBox工程）、python（Python功能单元）、c++（C++功能单元）、infer（推理功能单元）等；-n参数，表示所创建实例的名称；-s参数，表示将使用后面参数值代表的模板创建工程，而不是创建空的工程。2. 创建推理功能单元在ModelBox sdk目录下使用create.bat创建fcn_infer推理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t infer -n fcn_infer -p FCN ... success: create infer fcn_infer in D:\modelbox-win10-x64-1.5.3\workspace\FCN/model/fcn_infercreate.bat工具使用时，-t infer即表示创建的是推理功能单元；-n xxx_infer表示创建的功能单元名称为xxx_infer；-p表示所创建的功能单元属于FCN应用。下载转换好的FCN.onnx模型到FCN\model目录下，修改推理功能单元fcn_infer.toml模型的配置文件：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [base] name = "fcn_infer" device = "cpu" version = "1.0.0" description = "your description" entry = "./FCN.onnx" # model file path, use relative path type = "inference" virtual_type = "onnx" # inference engine type: win10 now only support onnx group_type = "Inference" # flowunit group attribution, do not change # Input ports description [input] [input.input1] # input port number, Format is input.input[N] name = "Input" # input port name type = "float" # input port data type ,e.g. float or uint8 device = "cpu" # input buffer type: cpu, win10 now copy input from cpu # Output ports description [output] [output.output1] # output port number, Format is output.output[N] name = "Output" # output port name type = "float" # output port data type ,e.g. float or uint83. 创建后处理功能单元在ModelBox sdk目录下使用create.bat创建fcn_post后处理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t python -n fcn_post -p FCN ... success: create python fcn_post in D:\modelbox-win10-x64-1.5.3\workspace\FCN/etc/flowunit/fcn_postcreate.bat工具使用时，-t python即表示创建的是通用功能单元；-n xxx_post表示创建的功能单元名称为xxx_post；-p表示所创建的功能单元属于FCN应用。a. 修改配置文件我们的模型有一个输入和输出，对116种果蔬病虫害进行分割，加上背景总共是117类：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. # Basic config [base] name = "fcn_post" # The FlowUnit name device = "cpu" # The flowunit runs on cpu version = "1.0.0" # The version of the flowunit type = "python" # Fixed value, do not change description = "description" # The description of the flowunit entry = "fcn_post@fcn_postFlowUnit" # Python flowunit entry function group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ... # Flowunit Type stream = false # Whether the flowunit is a stream flowunit condition = false # Whether the flowunit is a condition flowunit collapse = false # Whether the flowunit is a collapse flowunit collapse_all = false # Whether the flowunit will collapse all the data expand = false # Whether the flowunit is a expand flowunit # The default Flowunit config [config] num_classes = 117 net_w = 224 net_h = 224 # Input ports description [input] [input.input1] # Input port number, the format is input.input[N] name = "in_image" # Input port name type = "uint8" # Input port type [input.input2] # Input port number, the format is input.input[N] name = "in_feat" # Input port name type = "float" # Input port type # Output ports description [output] [output.output1] # Output port number, the format is output.output[N] name = "out_image" # Output port name type = "uint8" # Output port typeb. 修改逻辑代码# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. #!/usr/bin/env python # -*- coding: utf-8 -*- import _flowunit as modelbox import numpy as np import cv2 class fcn_postFlowUnit(modelbox.FlowUnit): # Derived from modelbox.FlowUnit def __init__(self): super().__init__() def open(self, config): # Open the flowunit to obtain configuration information self.params = {} self.params['num_classes'] = config.get_int('num_classes') self.params['net_w'] = config.get_int('net_w') self.params['net_h'] = config.get_int('net_h') return modelbox.Status.StatusCode.STATUS_SUCCESS def process(self, data_context): # Process the data in_image = data_context.input("in_image") in_feat = data_context.input("in_feat") out_image = data_context.output("out_image") # fcn_post process code. # Remove the following code and add your own code here. for buffer_image, buffer_feat in zip(in_image, in_feat): channel = buffer_image.get('channel') width = buffer_image.get('width') height = buffer_image.get('height') image = np.array(buffer_image.as_object(), dtype=np.uint8, copy=False) image = image.reshape(height, width, channel) feat = np.array(buffer_feat.as_object(), dtype=np.float32, copy=False) feat = feat.reshape(self.params['net_h'], self.params['net_w'], self.params['num_classes']) mask = np.argmax(feat, axis=-1).astype(np.uint8) mask = cv2.resize(mask, (width, height), interpolation=cv2.INTER_NEAREST) overlay = np.zeros_like(image) for i in range(1, self.params['num_classes']): color = np.random.randint(0, 255, (3,)).tolist() overlay[mask==i] = color result_image = cv2.addWeighted(image[..., ::-1], 0.5, overlay, 0.5, 0) add_buffer = modelbox.Buffer(self.get_bind_device(), result_image) add_buffer.copy_meta(buffer_image) out_image.push_back(add_buffer) return modelbox.Status.StatusCode.STATUS_SUCCESS def close(self): # Close the flowunit return modelbox.Status() def data_pre(self, data_context): # Before streaming data starts return modelbox.Status() def data_post(self, data_context): # After streaming data ends return modelbox.Status() def data_group_pre(self, data_context): # Before all streaming data starts return modelbox.Status() def data_group_post(self, data_context): # After all streaming data ends return modelbox.Status() 4. 修改应用的流程图FCN工程graph目录下存放流程图，默认的流程图FCN.toml与工程同名：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [driver] dir = ["${HILENS_APP_ROOT}/etc/flowunit", "${HILENS_APP_ROOT}/etc/flowunit/cpp", "${HILENS_APP_ROOT}/model", "${HILENS_MB_SDK_PATH}/flowunit"] skip-default = true [profile] profile=false trace=false dir="${HILENS_DATA_DIR}/mb_profile" [graph] format = "graphviz" graphconf = """digraph FCN { node [shape=Mrecord] queue_size = 4 batch_size = 1 input1[type=input,flowunit=input,device=cpu,deviceid=0] httpserver_sync_receive[type=flowunit, flowunit=httpserver_sync_receive_v2, device=cpu, deviceid=0, time_out_ms=5000, endpoint="http://0.0.0.0:1234/v1/FCN", max_requests=100] image_decoder[type=flowunit, flowunit=image_decoder, device=cpu, key="image_base64", queue_size=4] image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=224, image_height=224] normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="0.003921568627450,0.003921568627450,0.003921568627450"] fcn_infer[type=flowunit, flowunit=fcn_infer, device=cpu, deviceid=0, batch_size=1] fcn_post[type=flowunit, flowunit=fcn_post, device=cpu, deviceid=0] httpserver_sync_reply[type=flowunit, flowunit=httpserver_sync_reply_v2, device=cpu, deviceid=0] input1:input -> httpserver_sync_receive:in_url httpserver_sync_receive:out_request_info -> image_decoder:in_encoded_image image_decoder:out_image -> image_resize:in_image image_resize:out_image -> normalize:in_data normalize:out_data -> fcn_infer:Input image_decoder:out_image -> fcn_post:in_image fcn_infer:Output -> fcn_post:in_feat fcn_post:out_image -> httpserver_sync_reply:in_reply_info }""" [flow] desc = "FCN run in modelbox-win10-x64" 在命令行中运行.\create.bat -t editor即可打开ModelBox图编排界面，可以实时修改并查看项目的流程图：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t editor5. 运行应用在FCN工程目录下执行.\bin\main.bat运行应用：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\FCN PS D:\modelbox-win10-x64-1.5.3\workspace\FCN> .\bin\main.bat在FCN工程data目录下新建test_http.py测试脚本：import cv2 import json import base64 import requests import numpy as np if __name__ == "__main__": port = 1234 ip = "127.0.0.1" url = "/v1/FCN" img_path = "apple_black_rot_google_0056.jpg" img_data = cv2.imread(img_path) img_data = cv2.cvtColor(img_data, cv2.COLOR_BGR2RGB) img_str = cv2.imencode('.jpg', img_data)[1].tobytes() img = base64.b64encode(img_str) img_base64_str = str(img, encoding='utf8') params = {"image_base64": img_base64_str} response = requests.post(f'http://{ip}:{port}{url}', data=json.dumps(params), headers={"Content-Type": "application/json"}) h, w, c = img_data.shape img_array = np.frombuffer(response.content, np.uint8) img_array = img_array.reshape((h, -1, c)) cv2.imwrite("res.jpg", img_array) 在FCN工程data目录下存放测试图片：在另一个终端中进入FCN工程目录data文件夹下：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\FCN\data首先安装requests依赖包：PS D:\modelbox-win10-x64-1.5.3\workspace\FCN\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe -m pip install requests然后运行test_http.py脚本发起HTTP请求测试：PS D:\modelbox-win10-x64-1.5.3\workspace\FCN\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe .\test_http.py测试图片的分割结果res.jpg将保存在FCN工程data目录下：三、小结本节介绍了如何使用ModelArts和ModelBox训练开发一个FCN果蔬病虫害分割的AI应用，我们只需要准备模型文件以及简单的配置即可创建一个HTTP服务。同时我们可以了解到FCN网络的基本结构、数据处理和模型训练方法，以及对应推理应用的逻辑。----转自博客：https://bbs.huaweicloud.com/blogs/449045

深海鱼类检测（ModelBox）一、模型训练和转换YOLOX是YOLO系列的优化版本，引入了解耦头、数据增强、无锚点以及标签分类等目标检测领域的优秀进展，拥有较好的精度表现，同时对工程部署友好。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、ModelBox 应用开发1. 创建工程在ModelBox sdk目录下使用create.bat创建fish_det工程：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t server -n fish_det -s car_det ... success: create fish_det in D:\modelbox-win10-x64-1.5.3\workspacecreate.bat工具的参数中，-t参数，表示所创建实例的类型，包括server（ModelBox工程）、python（Python功能单元）、c++（C++功能单元）、infer（推理功能单元）等；-n参数，表示所创建实例的名称；-s参数，表示将使用后面参数值代表的模板创建工程，而不是创建空的工程。2. 修改推理功能单元下载转换好的yolox_fish.onnx模型到fish_det\model目录下，修改推理功能单元yolox_infer.toml模型的配置文件：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. [base] name = "yolox_infer" device = "cpu" version = "1.0.0" description = "fish detection" entry = "./yolox_fish.onnx" # model file path, use relative path type = "inference" virtual_type = "onnx" # inference engine type: win10 now only support onnx group_type = "Inference" # flowunit group attribution, do not change # input port description, suporrt multiple input ports [input] [input.input1] name = "input" type = "float" device = "cpu" # output port description, suporrt multiple output ports [output] [output.output1] name = "output" type = "float" 3. 修改后处理功能单元我们的模型的输入大小为320，类别数量是1，修改fish_det\etc\flowunit\yolox_post目录下的yolox_post.toml配置文件：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. # Basic config [base] name = "yolox_post" # The FlowUnit name device = "cpu" # The device the flowunit runs on，cpu，cuda，ascend。 version = "1.0.0" # The version of the flowunit description = "description" # The description of the flowunit entry = "yolox_post@yolox_postFlowUnit" # Python flowunit entry function type = "python" # Fixed value group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ... # Flowunit Type stream = false # Whether the flowunit is a stream flowunit condition = false # Whether the flowunit is a condition flowunit collapse = false # Whether the flowunit is a collapse flowunit collapse_all = false # Whether the flowunit will collapse all the data expand = false # Whether the flowunit is a expand flowunit [config] net_h = 320 net_w = 320 num_classes = 1 strides = ['8', '16', '32'] conf_threshold = 0.25 iou_threshold = 0.45 [input] [input.input1] name = "in_feat" type = "float" [output] [output.output1] name = "out_data" type = "string" 4. 修改绘图功能单元我们这里只有一个类别，所以修改coco_car_labels = [0]只检测鱼这个类别：... def decode_car_bboxes(self, bbox_str, input_shape): try: coco_car_labels = [0] # fish det_result = json.loads(bbox_str)['det_result'] if (det_result == "None"): return [] bboxes = json.loads(det_result) car_bboxes = list(filter(lambda x: int(x[5]) in coco_car_labels, bboxes)) except Exception as ex: modelbox.error(str(ex)) return [] else: for bbox in car_bboxes: bbox[0] = int(bbox[0] * input_shape[1]) bbox[1] = int(bbox[1] * input_shape[0]) bbox[2] = int(bbox[2] * input_shape[1]) bbox[3] = int(bbox[3] * input_shape[0]) return car_bboxes ... 5. 修改应用的流程图修改image_resize图像预处理功能单元参数image_width=320, image_height=320与模型的输入大小保持一致：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. [driver] dir = ["${HILENS_APP_ROOT}/etc/flowunit", "${HILENS_APP_ROOT}/etc/flowunit/cpp", "${HILENS_APP_ROOT}/model", "${HILENS_MB_SDK_PATH}/flowunit"] skip-default = true [profile] profile=false trace=false dir="${HILENS_DATA_DIR}/mb_profile" [graph] format = "graphviz" graphconf = """digraph fish_det { node [shape=Mrecord] queue_size = 1 batch_size = 1 input1[type=input,flowunit=input,device=cpu,deviceid=0] data_source_parser[type=flowunit, flowunit=data_source_parser, device=cpu, deviceid=0] video_demuxer[type=flowunit, flowunit=video_demuxer, device=cpu, deviceid=0] video_decoder[type=flowunit, flowunit=video_decoder, device=cpu, deviceid=0, pix_fmt=bgr] image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=320, image_height=320] image_transpose[type=flowunit, flowunit=packed_planar_transpose, device=cpu, deviceid=0] normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="1,1,1"] car_detection[type=flowunit, flowunit=yolox_infer, device=cpu, deviceid=0, batch_size = 1] yolox_post[type=flowunit, flowunit=yolox_post, device=cpu, deviceid=0] draw_car_bbox[type=flowunit, flowunit=draw_car_bbox, device=cpu, deviceid=0] video_out[type=flowunit, flowunit=video_out, device=cpu, deviceid=0] input1:input -> data_source_parser:in_data data_source_parser:out_video_url -> video_demuxer:in_video_url video_demuxer:out_video_packet -> video_decoder:in_video_packet video_decoder:out_video_frame -> image_resize:in_image image_resize:out_image -> image_transpose:in_image image_transpose:out_image -> normalize:in_data normalize:out_data -> car_detection:input car_detection:output -> yolox_post:in_feat video_decoder:out_video_frame -> draw_car_bbox:in_image yolox_post:out_data -> draw_car_bbox:in_bbox draw_car_bbox:out_image -> video_out:in_video_frame }""" [flow] desc = "fish_det run in modelbox-win10-x64" 在命令行中运行.\create.bat -t editor即可打开ModelBox图编排界面，可以实时修改并查看项目的流程图：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t editor6. 配置应用的输入输出下载测试视频到fish_det\data目录下，修改应用fish_det\bin\mock_task.toml配置文件：# 用于本地mock文件读取任务，脚本中已经配置了IVA_SVC_CONFIG环境变量, 添加了此文件路径 ########### 请确定使用linux的路径类型，比如在windows上要用 D:/xxx/xxx 不能用D:\xxx\xxx ########### # 任务的参数为一个压缩并转义后的json字符串 # 直接写需要转义双引号， 也可以用 content_file 添加一个json文件 [common] content = "{\"param_str\":\"string param\",\"param_int\":10,\"param_float\":10.5}" # 任务输入配置，mock模拟目前仅支持一路rtsp或者本地url, 当前支持以下几种输入方式： # 1. rtsp摄像头或rtsp视频流：type="rtsp", url="rtsp://xxx.xxx" (type为rtsp的时候，支持视频中断自动重连) # 2. 设备自带摄像头或者USB摄像头：type="url"，url="摄像头编号,比如 0 或者 1 等" (需配合local_camera功能单元使用) # 3. 本地视频文件：type="url"，url="视频文件路径" (可以是相对路径 -- 相对这个mock_task.toml文件, 也支持从环境变量${HILENS_APP_ROOT}所在目录文件输入) # 4. http服务：type="url", url="http://xxx.xxx"(指的是任务作为http服务启动，此处需填写对外暴露的http服务地址，需配合httpserver类的功能单元使用) [input] type = "url" url = "${HILENS_APP_ROOT}/data/Test_ROV_video_h264_decim.mp4" # 任务输出配置，当前支持以下几种输出方式： # 1. rtsp视频流：type="local", url="rtsp://xxx.xxx" # 2. 本地屏幕：type="local", url="0:xxx" (设备需要接显示器，系统需要安装桌面) # 3. 本地视频文件：type="local"，url="视频文件路径" (可以是相对路径——相对这个mock_task.toml文件, 也支持输出到环境变量${HILENS_DATA_DIR}所在目录或子目录) # 4. http服务：type="webhook", url="http://xxx.xxx" (指的是任务产生的数据上报给某个http服务，此处需填写上传的http服务地址) [output] type = "local" url = "0" 7. 运行应用在fish_det工程目录下执行.\bin\main.bat运行应用，本地屏幕上会自动弹出鱼群的实时检测画面：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\fish_det PS D:\modelbox-win10-x64-1.5.3\workspace\fish_det> .\bin\main.bat三、小结本节介绍了如何使用ModelArts和ModelBox训练开发一个YOLOX鱼类目标检测的AI应用，我们只需要准备模型并配置对应的toml文件，即可快速实现模型的高效推理和部署。 ----转自博客：https://bbs.huaweicloud.com/blogs/449038

动物分类（ModelBox）一、模型训练与转换Inception V3，GoogLeNet的改进版本，采用InceptionModule和全局平均池化层，v3一个最重要的改进是分解（Factorization），将7x7分解成两个一维的卷积（1x7,7x1），3x3也是一样（1x3,3x1），这样的好处，既可以加速计算（多余的计算能力可以用来加深网络），又可以将1个conv拆成2个conv，使得网络深度进一步增加，增加了网络的非线性。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、ModelBox 应用开发1. 创建工程在ModelBox sdk目录下使用create.bat创建InceptionV3工程：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t server -n InceptionV3 ... success: create InceptionV3 in D:\modelbox-win10-x64-1.5.3\workspacecreate.bat工具的参数中，-t参数，表示所创建实例的类型，包括server（ModelBox工程）、python（Python功能单元）、c++（C++功能单元）、infer（推理功能单元）等；-n参数，表示所创建实例的名称；-s参数，表示将使用后面参数值代表的模板创建工程，而不是创建空的工程。2. 创建推理功能单元在ModelBox sdk目录下使用create.bat创建inceptionv3_infer推理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t infer -n inceptionv3_infer -p InceptionV3 ... success: create infer inceptionv3_infer in D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3/model/inceptionv3_infercreate.bat工具使用时，-t infer即表示创建的是推理功能单元；-n xxx_infer表示创建的功能单元名称为xxx_infer；-p表示所创建的功能单元属于InceptionV3应用。下载转换好的InceptionV3.onnx模型到InceptionV3\model目录下，修改推理功能单元inceptionv3_infer.toml模型的配置文件：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [base] name = "inceptionv3_infer" device = "cpu" version = "1.0.0" description = "your description" entry = "./InceptionV3.onnx" # model file path, use relative path type = "inference" virtual_type = "onnx" # inference engine type: win10 now only support onnx group_type = "Inference" # flowunit group attribution, do not change # Input ports description [input] [input.input1] # input port number, Format is input.input[N] name = "Input" # input port name type = "float" # input port data type ,e.g. float or uint8 device = "cpu" # input buffer type: cpu, win10 now copy input from cpu # Output ports description [output] [output.output1] # output port number, Format is output.output[N] name = "Output" # output port name type = "float" # output port data type ,e.g. float or uint83. 创建后处理功能单元在ModelBox sdk目录下使用create.bat创建inceptionv3_post后处理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t python -n inceptionv3_post -p InceptionV3 ... success: create python inceptionv3_post in D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3/etc/flowunit/inceptionv3_postcreate.bat工具使用时，-t python即表示创建的是通用功能单元；-n xxx_post表示创建的功能单元名称为xxx_post；-p表示所创建的功能单元属于InceptionV3应用。a. 修改配置文件我们的模型有一个输入和输出，总共包含90种动物类别：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. # Basic config [base] name = "inceptionv3_post" # The FlowUnit name device = "cpu" # The flowunit runs on cpu version = "1.0.0" # The version of the flowunit type = "python" # Fixed value, do not change description = "description" # The description of the flowunit entry = "inceptionv3_post@inceptionv3_postFlowUnit" # Python flowunit entry function group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ... # Flowunit Type stream = false # Whether the flowunit is a stream flowunit condition = false # Whether the flowunit is a condition flowunit collapse = false # Whether the flowunit is a collapse flowunit collapse_all = false # Whether the flowunit will collapse all the data expand = false # Whether the flowunit is a expand flowunit # The default Flowunit config [config] num_classes = 90 # Input ports description [input] [input.input1] # Input port number, the format is input.input[N] name = "in_feat" # Input port name type = "float" # Input port type # Output ports description [output] [output.output1] # Output port number, the format is output.output[N] name = "out_data" # Output port name type = "string" # Output port typeb. 修改逻辑代码# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. #!/usr/bin/env python # -*- coding: utf-8 -*- import _flowunit as modelbox import numpy as np import json class inceptionv3_postFlowUnit(modelbox.FlowUnit): # Derived from modelbox.FlowUnit def __init__(self): super().__init__() def open(self, config): # Open the flowunit to obtain configuration information self.params = {} self.params['num_classes'] = config.get_int('num_classes') return modelbox.Status.StatusCode.STATUS_SUCCESS def process(self, data_context): # Process the data in_feat = data_context.input("in_feat") out_data = data_context.output("out_data") # inceptionv3_post process code. # Remove the following code and add your own code here. for buffer_feat in in_feat: feat_data = np.array(buffer_feat.as_object(), copy=False) clsse = np.argmax(feat_data).astype(np.int32).item() score = feat_data[clsse].astype(np.float32).item() result = {"clsse": clsse, "score":score} result_str = json.dumps(result) out_buffer = modelbox.Buffer(self.get_bind_device(), result_str) out_data.push_back(out_buffer) return modelbox.Status.StatusCode.STATUS_SUCCESS def close(self): # Close the flowunit return modelbox.Status() def data_pre(self, data_context): # Before streaming data starts return modelbox.Status() def data_post(self, data_context): # After streaming data ends return modelbox.Status() def data_group_pre(self, data_context): # Before all streaming data starts return modelbox.Status() def data_group_post(self, data_context): # After all streaming data ends return modelbox.Status() 4. 修改应用的流程图InceptionV3工程graph目录下存放流程图，默认的流程图InceptionV3.toml与工程同名：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [driver] dir = ["${HILENS_APP_ROOT}/etc/flowunit", "${HILENS_APP_ROOT}/etc/flowunit/cpp", "${HILENS_APP_ROOT}/model", "${HILENS_MB_SDK_PATH}/flowunit"] skip-default = true [profile] profile=false trace=false dir="${HILENS_DATA_DIR}/mb_profile" [graph] format = "graphviz" graphconf = """digraph InceptionV3 { node [shape=Mrecord] queue_size = 4 batch_size = 1 input1[type=input,flowunit=input,device=cpu,deviceid=0] httpserver_sync_receive[type=flowunit, flowunit=httpserver_sync_receive_v2, device=cpu, deviceid=0, time_out_ms=5000, endpoint="http://0.0.0.0:1234/v1/InceptionV3", max_requests=100] image_decoder[type=flowunit, flowunit=image_decoder, device=cpu, key="image_base64", queue_size=4] image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=224, image_height=224] normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="0.003921568627450,0.003921568627450,0.003921568627450"] inceptionv3_infer[type=flowunit, flowunit=inceptionv3_infer, device=cpu, deviceid=0, batch_size=1] inceptionv3_post[type=flowunit, flowunit=inceptionv3_post, device=cpu, deviceid=0] httpserver_sync_reply[type=flowunit, flowunit=httpserver_sync_reply_v2, device=cpu, deviceid=0] input1:input -> httpserver_sync_receive:in_url httpserver_sync_receive:out_request_info -> image_decoder:in_encoded_image image_decoder:out_image -> image_resize:in_image image_resize:out_image -> normalize:in_data normalize:out_data -> inceptionv3_infer:Input inceptionv3_infer:Output -> inceptionv3_post:in_feat inceptionv3_post:out_data -> httpserver_sync_reply:in_reply_info }""" [flow] desc = "InceptionV3 run in modelbox-win10-x64" 在命令行中运行.\create.bat -t editor即可打开ModelBox图编排界面，可以实时修改并查看项目的流程图：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t editor5. 运行应用在InceptionV3工程目录下执行.\bin\main.bat运行应用：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3 PS D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3> .\bin\main.bat在InceptionV3工程data目录下新建test_http.py测试脚本：#!/usr/bin/env python # -*- coding: utf-8 -*- # Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. import os import cv2 import json import base64 import http.client class HttpConfig: '''http调用的参数配置''' def __init__(self, host_ip, port, url, img_base64_str): self.hostIP = host_ip self.Port = port self.httpMethod = "POST" self.requstURL = url self.headerdata = { "Content-Type": "application/json" } self.test_data = { "image_base64": img_base64_str } self.body = json.dumps(self.test_data) def read_image(img_path): '''读取图片数据并转为base64编码的字符串''' img_data = cv2.imread(img_path) img_data = cv2.cvtColor(img_data, cv2.COLOR_BGR2RGB) img_str = cv2.imencode('.jpg', img_data)[1].tobytes() img_bin = base64.b64encode(img_str) img_base64_str = str(img_bin, encoding='utf8') return img_data, img_base64_str def decode_result_str(result_str): try: result = json.loads(result_str) except Exception as ex: print(str(ex)) return [] else: return result labels = ['antelope', 'badger', 'bat', 'bear', 'bee', 'beetle', 'bison', 'boar', 'butterfly', 'cat', 'caterpillar', 'chimpanzee', 'cockroach', 'cow', 'coyote', 'crab', 'crow', 'deer', 'dog', 'dolphin', 'donkey', 'dragonfly', 'duck', 'eagle', 'elephant', 'flamingo', 'fly', 'fox', 'goat', 'goldfish', 'goose', 'gorilla', 'grasshopper', 'hamster', 'hare', 'hedgehog', 'hippopotamus', 'hornbill', 'horse', 'hummingbird', 'hyena', 'jellyfish', 'kangaroo', 'koala', 'ladybugs', 'leopard', 'lion', 'lizard', 'lobster', 'mosquito', 'moth', 'mouse', 'octopus', 'okapi', 'orangutan', 'otter', 'owl', 'ox', 'oyster', 'panda', 'parrot', 'pelecaniformes', 'penguin', 'pig', 'pigeon', 'porcupine', 'possum', 'raccoon', 'rat', 'reindeer', 'rhinoceros', 'sandpiper', 'seahorse', 'seal', 'shark', 'sheep', 'snake', 'sparrow', 'squid', 'squirrel', 'starfish', 'swan', 'tiger', 'turkey', 'turtle', 'whale', 'wolf', 'wombat', 'woodpecker', 'zebra'] def test_image(img_path, ip, port, url): '''单张图片测试''' img_data, img_base64_str = read_image(img_path) http_config = HttpConfig(ip, port, url, img_base64_str) conn = http.client.HTTPConnection(host=http_config.hostIP, port=http_config.Port) conn.request(method=http_config.httpMethod, url=http_config.requstURL, body=http_config.body, headers=http_config.headerdata) response = conn.getresponse().read().decode() print('response: ', response) result = decode_result_str(response) clsse, score = result["clsse"], result["score"] result_str = f"{labels[clsse]}:{round(score, 2)}" cv2.putText(img_data, result_str, (0, 100), cv2.FONT_HERSHEY_TRIPLEX, 4, (0, 255, 0), 2) cv2.imwrite('./result-' + os.path.basename(img_path), img_data[..., ::-1]) if __name__ == "__main__": port = 1234 ip = "127.0.0.1" url = "/v1/InceptionV3" img_folder = './test_imgs' file_list = os.listdir(img_folder) for img_file in file_list: print("\n================ {} ================".format(img_file)) img_path = os.path.join(img_folder, img_file) test_image(img_path, ip, port, url) 在InceptionV3工程data目录下新建test_imgs文件夹存放测试图片：在另一个终端中进入InceptionV3工程目录data文件夹下运行test_http.py脚本发起HTTP请求测试：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3\data PS D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe .\test_http.py ================ 61cf5127ce.jpg ================ response: {"clsse": 63, "score": 0.9996486902236938} ================ 7e2a453559.jpg ================ response: {"clsse": 81, "score": 0.999880313873291} 在InceptionV3工程data目录下即可查看测试图片的推理结果：三、小结本节介绍了如何使用ModelArts和ModelBox训练开发一个InceptionV3动物图片分类的AI应用，我们只需要准备模型文件以及简单的配置即可创建一个HTTP服务。同时我们可以了解到InceptionV3网络的基本结构、数据处理和模型训练方法，以及对应推理应用的逻辑。----转自博客：https://bbs.huaweicloud.com/blogs/449036

小样本分类实现度量学习方法1、Prototypical Networks为每个类别计算原型向量（类中心）通过计算样本与原型的距离进行分类import torch import torch.nn as nn class PrototypicalNetwork(nn.Module): def __init__(self, encoder): super().__init__() self.encoder = encoder def forward(self, support_images, support_labels, query_images): # 编码支持集和查询集 support_embeddings = self.encoder(support_images) query_embeddings = self.encoder(query_images) # 计算类原型 prototypes = compute_prototypes(support_embeddings, support_labels) # 计算距离并分类 distances = compute_distances(query_embeddings, prototypes) return -distances2、Siamese Networks学习样本对之间的相似性度量通过比较查询样本与支持样本的相似度进行分类核心思想Siamese Networks采用孪生网络结构，通过比较两个输入样本的特征表示来判断它们的相似性：网络结构：两个完全相同的子网络共享权重输入方式：同时输入一对样本（正样本对或负样本对）学习目标：学习一个距离度量函数最佳实践可以参考AI Gallery中的Noetbook示例：

【朝推夜训】如何在边缘设备上搭建深度学习开发环境如何在边缘设备上搭建深度学习的开发环境，我们以Jetson Orin Nano为例，介绍如何在开发板上安装Miniconda并配置conda源，以及如何安装Pytorch和Torchvision。1. 安装 Miniconda首先下载Miniconda最新安装包：wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh运行Miniconda3-latest-Linux-aarch64.sh安装脚本进行安装：bash ~/Miniconda3-latest-Linux-aarch64.sh关闭并重新打开终端窗口以使安装完全生效，或者使用以下命令刷新终端：source ~/.bashrc2. conda 换源首先编辑.condarc文件：vi ~/.condarc我们使用清华源，将文件修改为如下内容，即可添加Anaconda Python免费仓库。channels: - defaults show_channel_urls: true default_channels: - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2 custom_channels: conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud使用下列命令清除索引缓存，并创建Python-3.8开发环境。conda clean -i conda create -n py38 python=3.8 3. pip 换源在用户目录创建.pip目录，并编辑pip.conf文件：cd ~ mkdir .pip cd .pip vi pip.confpip.conf写入以下内容：[global] index-url = https://pypi.tuna.tsinghua.edu.cn/simple/ [install] trusted-host = pypi.tuna.tsinghua.edu.cn保存pip.conf4. 安装 Pytorch 和 Torchvision首先打开PyTorch for Jetson官网根据自己的JetPack版本选择合适安装包进行下载：https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048 cd ~/Downloads wget https://developer.download.nvidia.cn/compute/redist/jp/v512/pytorch/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl我下载版本的是PyTorch v2.1.0，之后激活我们刚刚创建好的conda环境py38并安装Pytorch：conda activate py38 sudo apt-get install python3-pip libopenblas-base libopenmpi-dev libomp-dev pip install ~/Downloads/torch-2.1.0a0+41361538.nv23.06-cp38-cp38-linux_aarch64.whl安装完成后，我们可以验证一下，在命令行中输入：python -c "import torch; print(torch.cuda.is_available())" 结果为True，则说明安装成功。之后安装Torchvision，Pytorch v2.1.0对应torchvision v0.16.1：PyTorch v1.0 - torchvision v0.2.2PyTorch v1.1 - torchvision v0.3.0PyTorch v1.2 - torchvision v0.4.0PyTorch v1.3 - torchvision v0.4.2PyTorch v1.4 - torchvision v0.5.0PyTorch v1.5 - torchvision v0.6.0PyTorch v1.6 - torchvision v0.7.0PyTorch v1.7 - torchvision v0.8.1PyTorch v1.8 - torchvision v0.9.0PyTorch v1.9 - torchvision v0.10.0PyTorch v1.10 - torchvision v0.11.1PyTorch v1.11 - torchvision v0.12.0PyTorch v1.12 - torchvision v0.13.0PyTorch v1.13 - torchvision v0.13.0PyTorch v1.14 - torchvision v0.14.1PyTorch v2.0 - torchvision v0.15.1PyTorch v2.1 - torchvision v0.16.1PyTorch v2.2 - torchvision v0.17.1PyTorch v2.3 - torchvision v0.18.0Torchvision安装命令如下：sudo apt-get install libjpeg-dev zlib1g-dev libpython3-dev libopenblas-dev libavcodec-dev libavformat-dev libswscale-dev git clone --branch v0.16.1 https://github.com/pytorch/vision torchvision cd torchvision conda activate py38 export BUILD_VERSION=0.16.1 pip install numpy==1.23.5 Pillow==9.5.0 requests==2.32.4 python setup.py install --user等待Trochvision安装完成后，我们可以验证一下，在命令行中输入：python -c "import torchvision; print(torchvision.__version__)" 成功打印版本号说明安装成功！

RK3588 AI 应用开发 （ResNet50V2-关键点检测）一、模型训练与转换ResNet50V2 是改进版的深度卷积神经网络，基于 ResNet 架构发展而来。它采用前置激活（将 BN 和 ReLU 移至卷积前）与身份映射，优化了信息传播和模型训练性能。作为 50 层深度的网络，ResNet50V2 广泛应用于图像分类、目标检测等任务，支持迁移学习，适合快速适配新数据集，具有良好的泛化能力和较高准确率。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、应用开发1. 开发 Gradio 界面import cv2 import json import base64 import requests import numpy as np import gradio as gr def test_image(image_path): try: image_bgr = cv2.imread(image_path) image_string = cv2.imencode('.jpg', image_bgr)[1].tobytes() image_base64 = base64.b64encode(image_string).decode('utf-8') params = {"image_base64": image_base64} response = requests.post(f'http://{ip}:{port}{url}', data=json.dumps(params), headers={"Content-Type": "application/json"}) if response.status_code == 200: image_base64 = response.json().get("image_base64") image_binary = base64.b64decode(image_base64) image_array = np.frombuffer(image_binary, dtype=np.uint8) image_rgb = cv2.imdecode(image_array, cv2.IMREAD_COLOR) else: image_rgb = None except Exception as e: return None else: return image_rgb if __name__ == "__main__": port = 8000 ip = "127.0.0.1" url = "/v1/ResNet50V2" demo = gr.Interface(fn=test_image, inputs=gr.Image(type="filepath"), outputs=["image"], title="ResNet50V2 猫脸关键点检测") demo.launch(share=False, server_port=3000) /home/orangepi/miniconda3/envs/python-3.10.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html from .autonotebook import tqdm as notebook_tqdm * Running on local URL: http://127.0.0.1:3000 * To create a public link, set `share=True` in `launch()`. 2. 编写推理代码class ResNet50V2: def __init__(self, model_path): self.rknn_lite = RKNNLite() self.rknn_lite.load_rknn(model_path) self.rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) def preprocess(self, image): image = image[:, :, ::-1] image = cv2.resize(image, (224, 224)) return np.expand_dims(image, axis=0) def rknn_infer(self, data): outputs = self.rknn_lite.inference(inputs=[data]) return outputs[0] def post_process(self, pred): feat = pred.squeeze().reshape(-1, 2) return feat def predict(self, image): # 图像预处理 data = self.preprocess(image) # 模型推理 pred = self.rknn_infer(data) # 模型后处理 keypoints = self.post_process(pred) # 绘制关键点检测结果 h, w, _ = image.shape for x, y in keypoints: cv2.circle(image, (int(x * w), int(y * h)), 5, (0, 255, 0), -1) return image[..., ::-1] def release(self): self.rknn_lite.release() 3. 图片批量预测import os import cv2 import numpy as np import matplotlib.pyplot as plt from rknnlite.api import RKNNLite model = ResNet50V2('model/ResNet50V2.rknn') for image in os.listdir("image"): image = cv2.imread(os.path.join("image", image)) image = model.predict(image) plt.imshow(image) plt.axis('off') plt.show() model.release() 4. 创建 Flask 服务import cv2 import base64 import numpy as np from rknnlite.api import RKNNLite from flask import Flask, request, jsonify from flask_cors import CORS app = Flask(__name__) CORS(app) @app.route('/v1/ResNet50V2', methods=['POST']) def inference(): data = request.get_json() image_base64 = data.get("image_base64") image_binary = base64.b64decode(image_base64) image_array = np.frombuffer(image_binary, dtype=np.uint8) image_bgr = cv2.imdecode(image_array, cv2.IMREAD_COLOR) image_rgb = model.predict(image_bgr) image_string = cv2.imencode('.jpg', image_rgb)[1].tobytes() image_base64 = base64.b64encode(image_string).decode('utf-8') return jsonify({ "image_base64": image_base64 }), 200 if __name__ == '__main__': model = ResNet50V2('model/ResNet50V2.rknn') app.run(host='0.0.0.0', port=8000) model.release() W rknn-toolkit-lite2 version: 2.3.2 * Serving Flask app '__main__' * Debug mode: off WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:8000 * Running on http://192.168.3.50:8000 Press CTRL+C to quit 127.0.0.1 - - [02/May/2025 02:13:40] "POST /v1/ResNet50V2 HTTP/1.1" 200 - 127.0.0.1 - - [02/May/2025 02:13:46] "POST /v1/ResNet50V2 HTTP/1.1" 200 - 5. 上传图片预测三、小结本章介绍了基于 RK3588 的 ResNet50V2 关键点检测应用开发全流程，包括模型训练与转换、Gradio 界面设计、推理代码实现、批量预测处理及 Flask 服务部署，完整实现了从模型到端到端应用的落地。----转自博客：https://bbs.huaweicloud.com/blogs/451999