猫脸关键点检测（ModelBox）一、模型训练与转换ResNet50V2是改进版的深度卷积神经网络，基于 ResNet 架构发展而来。它采用前置激活（将 BN 和 ReLU 移至卷积前）与身份映射，优化了信息传播和模型训练性能。作为 50 层深度的网络，ResNet50V2 广泛应用于图像分类、目标检测等任务，支持迁移学习，适合快速适配新数据集，具有良好的泛化能力和较高准确率。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、应用开发1. 创建工程在ModelBox sdk目录下使用create.bat创建ResNet50V2工程：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t server -n ResNet50V2 ... success: create ResNet50V2 in D:\modelbox-win10-x64-1.5.3\workspacecreate.bat工具的参数中，-t参数，表示所创建实例的类型，包括server（ModelBox工程）、python（Python功能单元）、c++（C++功能单元）、infer（推理功能单元）等；-n参数，表示所创建实例的名称；-s参数，表示将使用后面参数值代表的模板创建工程，而不是创建空的工程。2. 创建推理功能单元在ModelBox sdk目录下使用create.bat创建resnet50v2_infer推理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t infer -n resnet50_infer -p ResNet50V2 ... success: create infer resnet50_infer in D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2/model/resnet50_infercreate.bat工具使用时，-t infer即表示创建的是推理功能单元；-n xxx_infer表示创建的功能单元名称为xxx_infer；-p表示所创建的功能单元属于ResNet50V2应用。下载转换好的ResNet50V2.onnx模型到ResNet50V2\model目录下，修改推理功能单元resnet50v2_infer.toml模型的配置文件：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [base] name = "resnet50_infer" device = "cpu" version = "1.0.0" description = "your description" entry = "./ResNet50V2.onnx" # model file path, use relative path type = "inference" virtual_type = "onnx" # inference engine type: win10 now only support onnx group_type = "Inference" # flowunit group attribution, do not change # Input ports description [input] [input.input1] # input port number, Format is input.input[N] name = "Input" # input port name type = "float" # input port data type ,e.g. float or uint8 device = "cpu" # input buffer type: cpu, win10 now copy input from cpu # Output ports description [output] [output.output1] # output port number, Format is output.output[N] name = "Output" # output port name type = "float" # output port data type ,e.g. float or uint83. 创建后处理功能单元在ModelBox sdk目录下使用create.bat创建resnet50v2_post后处理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t python -n resnet50v2_post -p ResNet50V2 ... success: create python resnet50v2_post in D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2/etc/flowunit/resnet50v2_postcreate.bat工具使用时，-t python即表示创建的是通用功能单元；-n xxx_post表示创建的功能单元名称为xxx_post；-p表示所创建的功能单元属于ResNet50V2应用。a. 修改配置文件我们的模型有一个输入和输出，总共包含猫脸的9个关键点：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. # Basic config [base] name = "resnet50v2_post" # The FlowUnit name device = "cpu" # The flowunit runs on cpu version = "1.0.0" # The version of the flowunit type = "python" # Fixed value, do not change description = "description" # The description of the flowunit entry = "resnet50v2_post@resnet50v2_postFlowUnit" # Python flowunit entry function group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ... # Flowunit Type stream = false # Whether the flowunit is a stream flowunit condition = false # Whether the flowunit is a condition flowunit collapse = false # Whether the flowunit is a collapse flowunit collapse_all = false # Whether the flowunit will collapse all the data expand = false # Whether the flowunit is a expand flowunit # The default Flowunit config [config] keypoints = 9 # Input ports description [input] [input.input1] # Input port number, the format is input.input[N] name = "in_feat" # Input port name type = "float" # Input port type # Output ports description [output] [output.output1] # Output port number, the format is output.output[N] name = "out_data" # Output port name type = "string" # Output port typeb. 修改逻辑代码# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. #!/usr/bin/env python # -*- coding: utf-8 -*- import _flowunit as modelbox import numpy as np import json class resnet50v2_postFlowUnit(modelbox.FlowUnit): # Derived from modelbox.FlowUnit def __init__(self): super().__init__() def open(self, config): # Open the flowunit to obtain configuration information self.params = {} self.params['keypoints'] = config.get_int('keypoints') return modelbox.Status.StatusCode.STATUS_SUCCESS def process(self, data_context): # Process the data in_data = data_context.input("in_feat") out_data = data_context.output("out_data") # resnet50v2_post process code. # Remove the following code and add your own code here. for buffer_feat in in_data: feat_data = np.array(buffer_feat.as_object(), copy=False) keypoints = feat_data.reshape(-1, 2).tolist() result = {"keypoints": keypoints} result_str = json.dumps(result) out_buffer = modelbox.Buffer(self.get_bind_device(), result_str) out_data.push_back(out_buffer) return modelbox.Status.StatusCode.STATUS_SUCCESS def close(self): # Close the flowunit return modelbox.Status() def data_pre(self, data_context): # Before streaming data starts return modelbox.Status() def data_post(self, data_context): # After streaming data ends return modelbox.Status() def data_group_pre(self, data_context): # Before all streaming data starts return modelbox.Status() def data_group_post(self, data_context): # After all streaming data ends return modelbox.Status() 4. 修改应用的流程图ResNet50V2工程graph目录下存放流程图，默认的流程图ResNet50V2.toml与工程同名：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [driver] dir = ["${HILENS_APP_ROOT}/etc/flowunit", "${HILENS_APP_ROOT}/etc/flowunit/cpp", "${HILENS_APP_ROOT}/model", "${HILENS_MB_SDK_PATH}/flowunit"] skip-default = true [profile] profile=false trace=false dir="${HILENS_DATA_DIR}/mb_profile" [graph] format = "graphviz" graphconf = """digraph ResNet50V2 { node [shape=Mrecord] queue_size = 4 batch_size = 1 input1[type=input,flowunit=input,device=cpu,deviceid=0] httpserver_sync_receive[type=flowunit, flowunit=httpserver_sync_receive_v2, device=cpu, deviceid=0, time_out_ms=5000, endpoint="http://0.0.0.0:1234/v1/ResNet50V2", max_requests=100] image_decoder[type=flowunit, flowunit=image_decoder, device=cpu, key="image_base64", queue_size=4] image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=224, image_height=224] normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="0.003921568627450,0.003921568627450,0.003921568627450"] resnet50v2_infer[type=flowunit, flowunit=resnet50v2_infer, device=cpu, deviceid=0, batch_size=1] resnet50v2_post[type=flowunit, flowunit=resnet50v2_post, device=cpu, deviceid=0] httpserver_sync_reply[type=flowunit, flowunit=httpserver_sync_reply_v2, device=cpu, deviceid=0] input1:input -> httpserver_sync_receive:in_url httpserver_sync_receive:out_request_info -> image_decoder:in_encoded_image image_decoder:out_image -> image_resize:in_image image_resize:out_image -> normalize:in_data normalize:out_data -> resnet50v2_infer:Input resnet50v2_infer:Output -> resnet50v2_post:in_feat resnet50v2_post:out_data -> httpserver_sync_reply:in_reply_info }""" [flow] desc = "ResNet50V2 run in modelbox-win10-x64" 在命令行中运行.\create.bat -t editor即可打开ModelBox图编排界面，可以实时修改并查看项目的流程图：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t editor5. 运行应用在ResNet50V2工程目录下执行.\bin\main.bat运行应用：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2 PS D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2> .\bin\main.bat在ResNet50V2工程data目录下新建test_http.py测试脚本：#!/usr/bin/env python # -*- coding: utf-8 -*- # Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. import os import cv2 import json import base64 import http.client class HttpConfig: '''http调用的参数配置''' def __init__(self, host_ip, port, url, img_base64_str): self.hostIP = host_ip self.Port = port self.httpMethod = "POST" self.requstURL = url self.headerdata = { "Content-Type": "application/json" } self.test_data = { "image_base64": img_base64_str } self.body = json.dumps(self.test_data) def read_image(img_path): '''读取图片数据并转为base64编码的字符串''' img_data = cv2.imread(img_path) img_data = cv2.cvtColor(img_data, cv2.COLOR_BGR2RGB) img_str = cv2.imencode('.jpg', img_data)[1].tobytes() img_bin = base64.b64encode(img_str) img_base64_str = str(img_bin, encoding='utf8') return img_data, img_base64_str def test_image(img_path, ip, port, url): '''单张图片测试''' img_data, img_base64_str = read_image(img_path) http_config = HttpConfig(ip, port, url, img_base64_str) conn = http.client.HTTPConnection(host=http_config.hostIP, port=http_config.Port) conn.request(method=http_config.httpMethod, url=http_config.requstURL, body=http_config.body, headers=http_config.headerdata) response = conn.getresponse().read().decode() print('response: ', response) result = json.loads(response) w, h = img_data.shape[1], img_data.shape[0] for x, y in result["keypoints"]: if x > 0 and y > 0: cv2.circle(img_data, (int(x * w), int(y * h)), 5, (0, 255, 0), -1) cv2.imwrite('./result-' + os.path.basename(img_path), img_data[..., ::-1]) if __name__ == "__main__": port = 1234 ip = "127.0.0.1" url = "/v1/ResNet50V2" img_folder = './test_imgs' file_list = os.listdir(img_folder) for img_file in file_list: print("\n================ {} ================".format(img_file)) img_path = os.path.join(img_folder, img_file) test_image(img_path, ip, port, url) 在ResNet50V2工程data目录下新建test_imgs文件夹存放测试图片：在另一个终端中进入ResNet50V2工程目录data文件夹下运行test_http.py脚本发起HTTP请求测试：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2\data PS D:\modelbox-win10-x64-1.5.3\workspace\ResNet50V2\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe .\test_http.py ================ 2256.jpg ================ response: {"keypoints": [[0.19147011637687683, 0.26770520210266113], [0.29639703035354614, 0.26533427834510803], [0.24554343521595, 0.35762542486190796], [0.11009970307350159, 0.2090619057416916], [0.08408773690462112, 0.09547536075115204], [0.17451311647891998, 0.169035404920578], [0.2880205512046814, 0.168979212641716], [0.3739408254623413, 0.0717596635222435], [0.34669068455696106, 0.20229394733905792]]} ================ 6899.jpg ================ response: {"keypoints": [[0.3829421401023865, 0.41393953561782837], [0.47102952003479004, 0.42683106660842896], [0.4321300983428955, 0.5082458853721619], [0.3185971677303314, 0.36286458373069763], [0.33502572774887085, 0.2243150770664215], [0.3852037489414215, 0.29658034443855286], [0.4819968640804291, 0.30954840779304504], [0.5504774451255798, 0.2711380124092102], [0.5290539264678955, 0.3962092399597168]]} 在ResNet50V2工程data目录下即可查看测试图片的推理结果：三、小结本节介绍了如何使用ModelArts和ModelBox训练开发一个ResNet50V2猫脸关键点检测的AI应用，我们只需要准备模型文件以及简单的配置即可创建一个HTTP服务。同时我们可以了解到ResNet50V2网络的基本结构、数据处理和模型训练方法，以及对应推理应用的逻辑。----转自博客：https://bbs.huaweicloud.com/blogs/451999

果蔬病虫害分割（ModelBox）一、模型训练与转换FCN（全卷积网络，Fully Convolutional Networks）是用于语义分割任务的一种深度学习模型架构，引入了跳跃结构（Skip Architecture），通过融合浅层和深层的特征图，保留更多的细节信息，提升分割精度。此外，FCN还利用多尺度上下文聚合，捕捉不同层级的特征，增强了对不同大小目标的识别能力。FCN的成功推动了语义分割领域的发展，成为后续许多先进模型的基础。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、应用开发1. 创建工程在ModelBox sdk目录下使用create.bat创建FCN工程：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t server -n FCN ... success: create FCN in D:\modelbox-win10-x64-1.5.3\workspacecreate.bat工具的参数中，-t参数，表示所创建实例的类型，包括server（ModelBox工程）、python（Python功能单元）、c++（C++功能单元）、infer（推理功能单元）等；-n参数，表示所创建实例的名称；-s参数，表示将使用后面参数值代表的模板创建工程，而不是创建空的工程。2. 创建推理功能单元在ModelBox sdk目录下使用create.bat创建fcn_infer推理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t infer -n fcn_infer -p FCN ... success: create infer fcn_infer in D:\modelbox-win10-x64-1.5.3\workspace\FCN/model/fcn_infercreate.bat工具使用时，-t infer即表示创建的是推理功能单元；-n xxx_infer表示创建的功能单元名称为xxx_infer；-p表示所创建的功能单元属于FCN应用。下载转换好的FCN.onnx模型到FCN\model目录下，修改推理功能单元fcn_infer.toml模型的配置文件：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [base] name = "fcn_infer" device = "cpu" version = "1.0.0" description = "your description" entry = "./FCN.onnx" # model file path, use relative path type = "inference" virtual_type = "onnx" # inference engine type: win10 now only support onnx group_type = "Inference" # flowunit group attribution, do not change # Input ports description [input] [input.input1] # input port number, Format is input.input[N] name = "Input" # input port name type = "float" # input port data type ,e.g. float or uint8 device = "cpu" # input buffer type: cpu, win10 now copy input from cpu # Output ports description [output] [output.output1] # output port number, Format is output.output[N] name = "Output" # output port name type = "float" # output port data type ,e.g. float or uint83. 创建后处理功能单元在ModelBox sdk目录下使用create.bat创建fcn_post后处理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t python -n fcn_post -p FCN ... success: create python fcn_post in D:\modelbox-win10-x64-1.5.3\workspace\FCN/etc/flowunit/fcn_postcreate.bat工具使用时，-t python即表示创建的是通用功能单元；-n xxx_post表示创建的功能单元名称为xxx_post；-p表示所创建的功能单元属于FCN应用。a. 修改配置文件我们的模型有一个输入和输出，对116种果蔬病虫害进行分割，加上背景总共是117类：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. # Basic config [base] name = "fcn_post" # The FlowUnit name device = "cpu" # The flowunit runs on cpu version = "1.0.0" # The version of the flowunit type = "python" # Fixed value, do not change description = "description" # The description of the flowunit entry = "fcn_post@fcn_postFlowUnit" # Python flowunit entry function group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ... # Flowunit Type stream = false # Whether the flowunit is a stream flowunit condition = false # Whether the flowunit is a condition flowunit collapse = false # Whether the flowunit is a collapse flowunit collapse_all = false # Whether the flowunit will collapse all the data expand = false # Whether the flowunit is a expand flowunit # The default Flowunit config [config] num_classes = 117 net_w = 224 net_h = 224 # Input ports description [input] [input.input1] # Input port number, the format is input.input[N] name = "in_image" # Input port name type = "uint8" # Input port type [input.input2] # Input port number, the format is input.input[N] name = "in_feat" # Input port name type = "float" # Input port type # Output ports description [output] [output.output1] # Output port number, the format is output.output[N] name = "out_image" # Output port name type = "uint8" # Output port typeb. 修改逻辑代码# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. #!/usr/bin/env python # -*- coding: utf-8 -*- import _flowunit as modelbox import numpy as np import cv2 class fcn_postFlowUnit(modelbox.FlowUnit): # Derived from modelbox.FlowUnit def __init__(self): super().__init__() def open(self, config): # Open the flowunit to obtain configuration information self.params = {} self.params['num_classes'] = config.get_int('num_classes') self.params['net_w'] = config.get_int('net_w') self.params['net_h'] = config.get_int('net_h') return modelbox.Status.StatusCode.STATUS_SUCCESS def process(self, data_context): # Process the data in_image = data_context.input("in_image") in_feat = data_context.input("in_feat") out_image = data_context.output("out_image") # fcn_post process code. # Remove the following code and add your own code here. for buffer_image, buffer_feat in zip(in_image, in_feat): channel = buffer_image.get('channel') width = buffer_image.get('width') height = buffer_image.get('height') image = np.array(buffer_image.as_object(), dtype=np.uint8, copy=False) image = image.reshape(height, width, channel) feat = np.array(buffer_feat.as_object(), dtype=np.float32, copy=False) feat = feat.reshape(self.params['net_h'], self.params['net_w'], self.params['num_classes']) mask = np.argmax(feat, axis=-1).astype(np.uint8) mask = cv2.resize(mask, (width, height), interpolation=cv2.INTER_NEAREST) overlay = np.zeros_like(image) for i in range(1, self.params['num_classes']): color = np.random.randint(0, 255, (3,)).tolist() overlay[mask==i] = color result_image = cv2.addWeighted(image[..., ::-1], 0.5, overlay, 0.5, 0) add_buffer = modelbox.Buffer(self.get_bind_device(), result_image) add_buffer.copy_meta(buffer_image) out_image.push_back(add_buffer) return modelbox.Status.StatusCode.STATUS_SUCCESS def close(self): # Close the flowunit return modelbox.Status() def data_pre(self, data_context): # Before streaming data starts return modelbox.Status() def data_post(self, data_context): # After streaming data ends return modelbox.Status() def data_group_pre(self, data_context): # Before all streaming data starts return modelbox.Status() def data_group_post(self, data_context): # After all streaming data ends return modelbox.Status() 4. 修改应用的流程图FCN工程graph目录下存放流程图，默认的流程图FCN.toml与工程同名：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [driver] dir = ["${HILENS_APP_ROOT}/etc/flowunit", "${HILENS_APP_ROOT}/etc/flowunit/cpp", "${HILENS_APP_ROOT}/model", "${HILENS_MB_SDK_PATH}/flowunit"] skip-default = true [profile] profile=false trace=false dir="${HILENS_DATA_DIR}/mb_profile" [graph] format = "graphviz" graphconf = """digraph FCN { node [shape=Mrecord] queue_size = 4 batch_size = 1 input1[type=input,flowunit=input,device=cpu,deviceid=0] httpserver_sync_receive[type=flowunit, flowunit=httpserver_sync_receive_v2, device=cpu, deviceid=0, time_out_ms=5000, endpoint="http://0.0.0.0:1234/v1/FCN", max_requests=100] image_decoder[type=flowunit, flowunit=image_decoder, device=cpu, key="image_base64", queue_size=4] image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=224, image_height=224] normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="0.003921568627450,0.003921568627450,0.003921568627450"] fcn_infer[type=flowunit, flowunit=fcn_infer, device=cpu, deviceid=0, batch_size=1] fcn_post[type=flowunit, flowunit=fcn_post, device=cpu, deviceid=0] httpserver_sync_reply[type=flowunit, flowunit=httpserver_sync_reply_v2, device=cpu, deviceid=0] input1:input -> httpserver_sync_receive:in_url httpserver_sync_receive:out_request_info -> image_decoder:in_encoded_image image_decoder:out_image -> image_resize:in_image image_resize:out_image -> normalize:in_data normalize:out_data -> fcn_infer:Input image_decoder:out_image -> fcn_post:in_image fcn_infer:Output -> fcn_post:in_feat fcn_post:out_image -> httpserver_sync_reply:in_reply_info }""" [flow] desc = "FCN run in modelbox-win10-x64" 在命令行中运行.\create.bat -t editor即可打开ModelBox图编排界面，可以实时修改并查看项目的流程图：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t editor5. 运行应用在FCN工程目录下执行.\bin\main.bat运行应用：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\FCN PS D:\modelbox-win10-x64-1.5.3\workspace\FCN> .\bin\main.bat在FCN工程data目录下新建test_http.py测试脚本：import cv2 import json import base64 import requests import numpy as np if __name__ == "__main__": port = 1234 ip = "127.0.0.1" url = "/v1/FCN" img_path = "apple_black_rot_google_0056.jpg" img_data = cv2.imread(img_path) img_data = cv2.cvtColor(img_data, cv2.COLOR_BGR2RGB) img_str = cv2.imencode('.jpg', img_data)[1].tobytes() img = base64.b64encode(img_str) img_base64_str = str(img, encoding='utf8') params = {"image_base64": img_base64_str} response = requests.post(f'http://{ip}:{port}{url}', data=json.dumps(params), headers={"Content-Type": "application/json"}) h, w, c = img_data.shape img_array = np.frombuffer(response.content, np.uint8) img_array = img_array.reshape((h, -1, c)) cv2.imwrite("res.jpg", img_array) 在FCN工程data目录下存放测试图片：在另一个终端中进入FCN工程目录data文件夹下：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\FCN\data首先安装requests依赖包：PS D:\modelbox-win10-x64-1.5.3\workspace\FCN\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe -m pip install requests然后运行test_http.py脚本发起HTTP请求测试：PS D:\modelbox-win10-x64-1.5.3\workspace\FCN\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe .\test_http.py测试图片的分割结果res.jpg将保存在FCN工程data目录下：三、小结本节介绍了如何使用ModelArts和ModelBox训练开发一个FCN果蔬病虫害分割的AI应用，我们只需要准备模型文件以及简单的配置即可创建一个HTTP服务。同时我们可以了解到FCN网络的基本结构、数据处理和模型训练方法，以及对应推理应用的逻辑。----转自博客：https://bbs.huaweicloud.com/blogs/449045

深海鱼类检测（ModelBox）一、模型训练和转换YOLOX是YOLO系列的优化版本，引入了解耦头、数据增强、无锚点以及标签分类等目标检测领域的优秀进展，拥有较好的精度表现，同时对工程部署友好。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、ModelBox 应用开发1. 创建工程在ModelBox sdk目录下使用create.bat创建fish_det工程：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t server -n fish_det -s car_det ... success: create fish_det in D:\modelbox-win10-x64-1.5.3\workspacecreate.bat工具的参数中，-t参数，表示所创建实例的类型，包括server（ModelBox工程）、python（Python功能单元）、c++（C++功能单元）、infer（推理功能单元）等；-n参数，表示所创建实例的名称；-s参数，表示将使用后面参数值代表的模板创建工程，而不是创建空的工程。2. 修改推理功能单元下载转换好的yolox_fish.onnx模型到fish_det\model目录下，修改推理功能单元yolox_infer.toml模型的配置文件：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. [base] name = "yolox_infer" device = "cpu" version = "1.0.0" description = "fish detection" entry = "./yolox_fish.onnx" # model file path, use relative path type = "inference" virtual_type = "onnx" # inference engine type: win10 now only support onnx group_type = "Inference" # flowunit group attribution, do not change # input port description, suporrt multiple input ports [input] [input.input1] name = "input" type = "float" device = "cpu" # output port description, suporrt multiple output ports [output] [output.output1] name = "output" type = "float" 3. 修改后处理功能单元我们的模型的输入大小为320，类别数量是1，修改fish_det\etc\flowunit\yolox_post目录下的yolox_post.toml配置文件：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. # Basic config [base] name = "yolox_post" # The FlowUnit name device = "cpu" # The device the flowunit runs on，cpu，cuda，ascend。 version = "1.0.0" # The version of the flowunit description = "description" # The description of the flowunit entry = "yolox_post@yolox_postFlowUnit" # Python flowunit entry function type = "python" # Fixed value group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ... # Flowunit Type stream = false # Whether the flowunit is a stream flowunit condition = false # Whether the flowunit is a condition flowunit collapse = false # Whether the flowunit is a collapse flowunit collapse_all = false # Whether the flowunit will collapse all the data expand = false # Whether the flowunit is a expand flowunit [config] net_h = 320 net_w = 320 num_classes = 1 strides = ['8', '16', '32'] conf_threshold = 0.25 iou_threshold = 0.45 [input] [input.input1] name = "in_feat" type = "float" [output] [output.output1] name = "out_data" type = "string" 4. 修改绘图功能单元我们这里只有一个类别，所以修改coco_car_labels = [0]只检测鱼这个类别：... def decode_car_bboxes(self, bbox_str, input_shape): try: coco_car_labels = [0] # fish det_result = json.loads(bbox_str)['det_result'] if (det_result == "None"): return [] bboxes = json.loads(det_result) car_bboxes = list(filter(lambda x: int(x[5]) in coco_car_labels, bboxes)) except Exception as ex: modelbox.error(str(ex)) return [] else: for bbox in car_bboxes: bbox[0] = int(bbox[0] * input_shape[1]) bbox[1] = int(bbox[1] * input_shape[0]) bbox[2] = int(bbox[2] * input_shape[1]) bbox[3] = int(bbox[3] * input_shape[0]) return car_bboxes ... 5. 修改应用的流程图修改image_resize图像预处理功能单元参数image_width=320, image_height=320与模型的输入大小保持一致：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. [driver] dir = ["${HILENS_APP_ROOT}/etc/flowunit", "${HILENS_APP_ROOT}/etc/flowunit/cpp", "${HILENS_APP_ROOT}/model", "${HILENS_MB_SDK_PATH}/flowunit"] skip-default = true [profile] profile=false trace=false dir="${HILENS_DATA_DIR}/mb_profile" [graph] format = "graphviz" graphconf = """digraph fish_det { node [shape=Mrecord] queue_size = 1 batch_size = 1 input1[type=input,flowunit=input,device=cpu,deviceid=0] data_source_parser[type=flowunit, flowunit=data_source_parser, device=cpu, deviceid=0] video_demuxer[type=flowunit, flowunit=video_demuxer, device=cpu, deviceid=0] video_decoder[type=flowunit, flowunit=video_decoder, device=cpu, deviceid=0, pix_fmt=bgr] image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=320, image_height=320] image_transpose[type=flowunit, flowunit=packed_planar_transpose, device=cpu, deviceid=0] normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="1,1,1"] car_detection[type=flowunit, flowunit=yolox_infer, device=cpu, deviceid=0, batch_size = 1] yolox_post[type=flowunit, flowunit=yolox_post, device=cpu, deviceid=0] draw_car_bbox[type=flowunit, flowunit=draw_car_bbox, device=cpu, deviceid=0] video_out[type=flowunit, flowunit=video_out, device=cpu, deviceid=0] input1:input -> data_source_parser:in_data data_source_parser:out_video_url -> video_demuxer:in_video_url video_demuxer:out_video_packet -> video_decoder:in_video_packet video_decoder:out_video_frame -> image_resize:in_image image_resize:out_image -> image_transpose:in_image image_transpose:out_image -> normalize:in_data normalize:out_data -> car_detection:input car_detection:output -> yolox_post:in_feat video_decoder:out_video_frame -> draw_car_bbox:in_image yolox_post:out_data -> draw_car_bbox:in_bbox draw_car_bbox:out_image -> video_out:in_video_frame }""" [flow] desc = "fish_det run in modelbox-win10-x64" 在命令行中运行.\create.bat -t editor即可打开ModelBox图编排界面，可以实时修改并查看项目的流程图：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t editor6. 配置应用的输入输出下载测试视频到fish_det\data目录下，修改应用fish_det\bin\mock_task.toml配置文件：# 用于本地mock文件读取任务，脚本中已经配置了IVA_SVC_CONFIG环境变量, 添加了此文件路径 ########### 请确定使用linux的路径类型，比如在windows上要用 D:/xxx/xxx 不能用D:\xxx\xxx ########### # 任务的参数为一个压缩并转义后的json字符串 # 直接写需要转义双引号， 也可以用 content_file 添加一个json文件 [common] content = "{\"param_str\":\"string param\",\"param_int\":10,\"param_float\":10.5}" # 任务输入配置，mock模拟目前仅支持一路rtsp或者本地url, 当前支持以下几种输入方式： # 1. rtsp摄像头或rtsp视频流：type="rtsp", url="rtsp://xxx.xxx" (type为rtsp的时候，支持视频中断自动重连) # 2. 设备自带摄像头或者USB摄像头：type="url"，url="摄像头编号,比如 0 或者 1 等" (需配合local_camera功能单元使用) # 3. 本地视频文件：type="url"，url="视频文件路径" (可以是相对路径 -- 相对这个mock_task.toml文件, 也支持从环境变量${HILENS_APP_ROOT}所在目录文件输入) # 4. http服务：type="url", url="http://xxx.xxx"(指的是任务作为http服务启动，此处需填写对外暴露的http服务地址，需配合httpserver类的功能单元使用) [input] type = "url" url = "${HILENS_APP_ROOT}/data/Test_ROV_video_h264_decim.mp4" # 任务输出配置，当前支持以下几种输出方式： # 1. rtsp视频流：type="local", url="rtsp://xxx.xxx" # 2. 本地屏幕：type="local", url="0:xxx" (设备需要接显示器，系统需要安装桌面) # 3. 本地视频文件：type="local"，url="视频文件路径" (可以是相对路径——相对这个mock_task.toml文件, 也支持输出到环境变量${HILENS_DATA_DIR}所在目录或子目录) # 4. http服务：type="webhook", url="http://xxx.xxx" (指的是任务产生的数据上报给某个http服务，此处需填写上传的http服务地址) [output] type = "local" url = "0" 7. 运行应用在fish_det工程目录下执行.\bin\main.bat运行应用，本地屏幕上会自动弹出鱼群的实时检测画面：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\fish_det PS D:\modelbox-win10-x64-1.5.3\workspace\fish_det> .\bin\main.bat三、小结本节介绍了如何使用ModelArts和ModelBox训练开发一个YOLOX鱼类目标检测的AI应用，我们只需要准备模型并配置对应的toml文件，即可快速实现模型的高效推理和部署。 ----转自博客：https://bbs.huaweicloud.com/blogs/449038

动物分类（ModelBox）一、模型训练与转换Inception V3，GoogLeNet的改进版本，采用InceptionModule和全局平均池化层，v3一个最重要的改进是分解（Factorization），将7x7分解成两个一维的卷积（1x7,7x1），3x3也是一样（1x3,3x1），这样的好处，既可以加速计算（多余的计算能力可以用来加深网络），又可以将1个conv拆成2个conv，使得网络深度进一步增加，增加了网络的非线性。模型的训练与转换教程已经开放在AI Gallery中，其中包含训练数据、训练代码、模型转换脚本。在ModelArts的Notebook环境中训练后，再转换成对应平台的模型格式：onnx格式可以用在Windows设备上，RK系列设备上需要转换为rknn格式。二、ModelBox 应用开发1. 创建工程在ModelBox sdk目录下使用create.bat创建InceptionV3工程：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t server -n InceptionV3 ... success: create InceptionV3 in D:\modelbox-win10-x64-1.5.3\workspacecreate.bat工具的参数中，-t参数，表示所创建实例的类型，包括server（ModelBox工程）、python（Python功能单元）、c++（C++功能单元）、infer（推理功能单元）等；-n参数，表示所创建实例的名称；-s参数，表示将使用后面参数值代表的模板创建工程，而不是创建空的工程。2. 创建推理功能单元在ModelBox sdk目录下使用create.bat创建inceptionv3_infer推理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t infer -n inceptionv3_infer -p InceptionV3 ... success: create infer inceptionv3_infer in D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3/model/inceptionv3_infercreate.bat工具使用时，-t infer即表示创建的是推理功能单元；-n xxx_infer表示创建的功能单元名称为xxx_infer；-p表示所创建的功能单元属于InceptionV3应用。下载转换好的InceptionV3.onnx模型到InceptionV3\model目录下，修改推理功能单元inceptionv3_infer.toml模型的配置文件：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [base] name = "inceptionv3_infer" device = "cpu" version = "1.0.0" description = "your description" entry = "./InceptionV3.onnx" # model file path, use relative path type = "inference" virtual_type = "onnx" # inference engine type: win10 now only support onnx group_type = "Inference" # flowunit group attribution, do not change # Input ports description [input] [input.input1] # input port number, Format is input.input[N] name = "Input" # input port name type = "float" # input port data type ,e.g. float or uint8 device = "cpu" # input buffer type: cpu, win10 now copy input from cpu # Output ports description [output] [output.output1] # output port number, Format is output.output[N] name = "Output" # output port name type = "float" # output port data type ,e.g. float or uint83. 创建后处理功能单元在ModelBox sdk目录下使用create.bat创建inceptionv3_post后处理功能单元:PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t python -n inceptionv3_post -p InceptionV3 ... success: create python inceptionv3_post in D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3/etc/flowunit/inceptionv3_postcreate.bat工具使用时，-t python即表示创建的是通用功能单元；-n xxx_post表示创建的功能单元名称为xxx_post；-p表示所创建的功能单元属于InceptionV3应用。a. 修改配置文件我们的模型有一个输入和输出，总共包含90种动物类别：# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. # Basic config [base] name = "inceptionv3_post" # The FlowUnit name device = "cpu" # The flowunit runs on cpu version = "1.0.0" # The version of the flowunit type = "python" # Fixed value, do not change description = "description" # The description of the flowunit entry = "inceptionv3_post@inceptionv3_postFlowUnit" # Python flowunit entry function group_type = "Generic" # flowunit group attribution, change as Input/Output/Image/Generic ... # Flowunit Type stream = false # Whether the flowunit is a stream flowunit condition = false # Whether the flowunit is a condition flowunit collapse = false # Whether the flowunit is a collapse flowunit collapse_all = false # Whether the flowunit will collapse all the data expand = false # Whether the flowunit is a expand flowunit # The default Flowunit config [config] num_classes = 90 # Input ports description [input] [input.input1] # Input port number, the format is input.input[N] name = "in_feat" # Input port name type = "float" # Input port type # Output ports description [output] [output.output1] # Output port number, the format is output.output[N] name = "out_data" # Output port name type = "string" # Output port typeb. 修改逻辑代码# Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. #!/usr/bin/env python # -*- coding: utf-8 -*- import _flowunit as modelbox import numpy as np import json class inceptionv3_postFlowUnit(modelbox.FlowUnit): # Derived from modelbox.FlowUnit def __init__(self): super().__init__() def open(self, config): # Open the flowunit to obtain configuration information self.params = {} self.params['num_classes'] = config.get_int('num_classes') return modelbox.Status.StatusCode.STATUS_SUCCESS def process(self, data_context): # Process the data in_feat = data_context.input("in_feat") out_data = data_context.output("out_data") # inceptionv3_post process code. # Remove the following code and add your own code here. for buffer_feat in in_feat: feat_data = np.array(buffer_feat.as_object(), copy=False) clsse = np.argmax(feat_data).astype(np.int32).item() score = feat_data[clsse].astype(np.float32).item() result = {"clsse": clsse, "score":score} result_str = json.dumps(result) out_buffer = modelbox.Buffer(self.get_bind_device(), result_str) out_data.push_back(out_buffer) return modelbox.Status.StatusCode.STATUS_SUCCESS def close(self): # Close the flowunit return modelbox.Status() def data_pre(self, data_context): # Before streaming data starts return modelbox.Status() def data_post(self, data_context): # After streaming data ends return modelbox.Status() def data_group_pre(self, data_context): # Before all streaming data starts return modelbox.Status() def data_group_post(self, data_context): # After all streaming data ends return modelbox.Status() 4. 修改应用的流程图InceptionV3工程graph目录下存放流程图，默认的流程图InceptionV3.toml与工程同名：# Copyright (C) 2020 Huawei Technologies Co., Ltd. All rights reserved. [driver] dir = ["${HILENS_APP_ROOT}/etc/flowunit", "${HILENS_APP_ROOT}/etc/flowunit/cpp", "${HILENS_APP_ROOT}/model", "${HILENS_MB_SDK_PATH}/flowunit"] skip-default = true [profile] profile=false trace=false dir="${HILENS_DATA_DIR}/mb_profile" [graph] format = "graphviz" graphconf = """digraph InceptionV3 { node [shape=Mrecord] queue_size = 4 batch_size = 1 input1[type=input,flowunit=input,device=cpu,deviceid=0] httpserver_sync_receive[type=flowunit, flowunit=httpserver_sync_receive_v2, device=cpu, deviceid=0, time_out_ms=5000, endpoint="http://0.0.0.0:1234/v1/InceptionV3", max_requests=100] image_decoder[type=flowunit, flowunit=image_decoder, device=cpu, key="image_base64", queue_size=4] image_resize[type=flowunit, flowunit=resize, device=cpu, deviceid=0, image_width=224, image_height=224] normalize[type=flowunit, flowunit=normalize, device=cpu, deviceid=0, standard_deviation_inverse="0.003921568627450,0.003921568627450,0.003921568627450"] inceptionv3_infer[type=flowunit, flowunit=inceptionv3_infer, device=cpu, deviceid=0, batch_size=1] inceptionv3_post[type=flowunit, flowunit=inceptionv3_post, device=cpu, deviceid=0] httpserver_sync_reply[type=flowunit, flowunit=httpserver_sync_reply_v2, device=cpu, deviceid=0] input1:input -> httpserver_sync_receive:in_url httpserver_sync_receive:out_request_info -> image_decoder:in_encoded_image image_decoder:out_image -> image_resize:in_image image_resize:out_image -> normalize:in_data normalize:out_data -> inceptionv3_infer:Input inceptionv3_infer:Output -> inceptionv3_post:in_feat inceptionv3_post:out_data -> httpserver_sync_reply:in_reply_info }""" [flow] desc = "InceptionV3 run in modelbox-win10-x64" 在命令行中运行.\create.bat -t editor即可打开ModelBox图编排界面，可以实时修改并查看项目的流程图：PS D:\modelbox-win10-x64-1.5.3> .\create.bat -t editor5. 运行应用在InceptionV3工程目录下执行.\bin\main.bat运行应用：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3 PS D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3> .\bin\main.bat在InceptionV3工程data目录下新建test_http.py测试脚本：#!/usr/bin/env python # -*- coding: utf-8 -*- # Copyright (c) Huawei Technologies Co., Ltd. 2022. All rights reserved. import os import cv2 import json import base64 import http.client class HttpConfig: '''http调用的参数配置''' def __init__(self, host_ip, port, url, img_base64_str): self.hostIP = host_ip self.Port = port self.httpMethod = "POST" self.requstURL = url self.headerdata = { "Content-Type": "application/json" } self.test_data = { "image_base64": img_base64_str } self.body = json.dumps(self.test_data) def read_image(img_path): '''读取图片数据并转为base64编码的字符串''' img_data = cv2.imread(img_path) img_data = cv2.cvtColor(img_data, cv2.COLOR_BGR2RGB) img_str = cv2.imencode('.jpg', img_data)[1].tobytes() img_bin = base64.b64encode(img_str) img_base64_str = str(img_bin, encoding='utf8') return img_data, img_base64_str def decode_result_str(result_str): try: result = json.loads(result_str) except Exception as ex: print(str(ex)) return [] else: return result labels = ['antelope', 'badger', 'bat', 'bear', 'bee', 'beetle', 'bison', 'boar', 'butterfly', 'cat', 'caterpillar', 'chimpanzee', 'cockroach', 'cow', 'coyote', 'crab', 'crow', 'deer', 'dog', 'dolphin', 'donkey', 'dragonfly', 'duck', 'eagle', 'elephant', 'flamingo', 'fly', 'fox', 'goat', 'goldfish', 'goose', 'gorilla', 'grasshopper', 'hamster', 'hare', 'hedgehog', 'hippopotamus', 'hornbill', 'horse', 'hummingbird', 'hyena', 'jellyfish', 'kangaroo', 'koala', 'ladybugs', 'leopard', 'lion', 'lizard', 'lobster', 'mosquito', 'moth', 'mouse', 'octopus', 'okapi', 'orangutan', 'otter', 'owl', 'ox', 'oyster', 'panda', 'parrot', 'pelecaniformes', 'penguin', 'pig', 'pigeon', 'porcupine', 'possum', 'raccoon', 'rat', 'reindeer', 'rhinoceros', 'sandpiper', 'seahorse', 'seal', 'shark', 'sheep', 'snake', 'sparrow', 'squid', 'squirrel', 'starfish', 'swan', 'tiger', 'turkey', 'turtle', 'whale', 'wolf', 'wombat', 'woodpecker', 'zebra'] def test_image(img_path, ip, port, url): '''单张图片测试''' img_data, img_base64_str = read_image(img_path) http_config = HttpConfig(ip, port, url, img_base64_str) conn = http.client.HTTPConnection(host=http_config.hostIP, port=http_config.Port) conn.request(method=http_config.httpMethod, url=http_config.requstURL, body=http_config.body, headers=http_config.headerdata) response = conn.getresponse().read().decode() print('response: ', response) result = decode_result_str(response) clsse, score = result["clsse"], result["score"] result_str = f"{labels[clsse]}:{round(score, 2)}" cv2.putText(img_data, result_str, (0, 100), cv2.FONT_HERSHEY_TRIPLEX, 4, (0, 255, 0), 2) cv2.imwrite('./result-' + os.path.basename(img_path), img_data[..., ::-1]) if __name__ == "__main__": port = 1234 ip = "127.0.0.1" url = "/v1/InceptionV3" img_folder = './test_imgs' file_list = os.listdir(img_folder) for img_file in file_list: print("\n================ {} ================".format(img_file)) img_path = os.path.join(img_folder, img_file) test_image(img_path, ip, port, url) 在InceptionV3工程data目录下新建test_imgs文件夹存放测试图片：在另一个终端中进入InceptionV3工程目录data文件夹下运行test_http.py脚本发起HTTP请求测试：PS D:\modelbox-win10-x64-1.5.3> cd D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3\data PS D:\modelbox-win10-x64-1.5.3\workspace\InceptionV3\data> D:\modelbox-win10-x64-1.5.3\python-embed\python.exe .\test_http.py ================ 61cf5127ce.jpg ================ response: {"clsse": 63, "score": 0.9996486902236938} ================ 7e2a453559.jpg ================ response: {"clsse": 81, "score": 0.999880313873291} 在InceptionV3工程data目录下即可查看测试图片的推理结果：三、小结本节介绍了如何使用ModelArts和ModelBox训练开发一个InceptionV3动物图片分类的AI应用，我们只需要准备模型文件以及简单的配置即可创建一个HTTP服务。同时我们可以了解到InceptionV3网络的基本结构、数据处理和模型训练方法，以及对应推理应用的逻辑。----转自博客：https://bbs.huaweicloud.com/blogs/449036

如何使用 Python 开发 AI 图编排应用本文将介绍使用Python开发一个简单的AI图编排应用，我们的目标是实现AI应用在RK3588上灵活编排和高效部署。首先我们定义的图是由边和节点组成的有向无环图，边代表任务队列，表示数据在节点之间的流动关系，每个节点都是一个计算单元，用于处理特定的任务。之后我们可以定义一组的处理特定任务的函数节点也称为计算单元，例如：read_frame、model_infer、kf_tracker、draw_boxes、push_frame、redis_push，分别用于读取视频、模型检测、目标跟踪、图像绘制、视频输出以及结果推送。每个节点可以有一个输入和多个输出，数据在节点之间是单向流动的，节点之间通过边进行连接，每个节点通过队列消费和传递数据。代码地址：https://github.com/HouYanSong/modelbox-rk3588一. 计算节点的实现我们在Json文件中定义每一个节点的的数据结构并使用Python进行代码实现：读流计算单元有4个参数：pull_video_url、height、width、fps，分别代表视频地址、视频高度和宽度以及读取帧率，它仅作为生产者，产生的数据可以输出到多个队列。"read_frame": { "config": { "pull_video_url": { "type": "str", "required": true, "default": null, "desc": "pull video url", "source": "mp4|flv|rtmp|rtsp" }, "height": { "type": "int", "required": true, "default": null, "max": 1440, "min": 720, "desc": "video height" }, "width": { "type": "int", "required": true, "default": null, "max": 1920, "min": 960, "desc": "video width" }, "fps": { "type": "int", "required": true, "default": null, "max": 15, "min": 5, "desc": "frame rate" } }, "multi_output": [] } 函数代码的实现如下，我们可以对视频文件或者视频流使用ffmpeg进行硬件解码，并将解码后的帧数据写入到队列中，用于后续任务节点的计算。def read_frame(share_dict, flowunit_data, queue_dict, data): pull_video_url = flowunit_data["config"]["pull_video_url"] height = flowunit_data["config"]["height"] width = flowunit_data["config"]["width"] fps = flowunit_data["config"]["fps"] ffmpeg_cmd = [ 'ffmpeg', '-c:v', 'h264_rkmpp', '-i', pull_video_url, '-r', f'{fps}', '-loglevel', 'info', '-s', f'{width}x{height}', '-an', '-f', 'rawvideo', '-pix_fmt', 'bgr24', 'pipe:' ] ffmpeg_process = sp.Popen(ffmpeg_cmd, stdout=sp.PIPE, stderr=sp.DEVNULL, bufsize=10**7) index = 0 while True: index += 1 raw_frame = ffmpeg_process.stdout.read(width * height * 3) if not raw_frame: break else: frame = np.frombuffer(raw_frame, dtype=np.uint8).reshape((height, width, -1)) data["frame"] = frame for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) # 读取结束，图片数据置为None data["frame"] = None for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) ffmpeg_process.stdout.close() ffmpeg_process.terminate() 推理计算单元的函数定义如下，它有一个输入和多个输出，我们可以指定模型和配置文件路径以及单次图像推理的批次大小等参数。"model_infer": { "config": { "model_file": { "type": "str", "required": true, "default": null, "desc": "model file path, rk3588 mostly ends with .rknn" }, "model_info": { "type": "str", "required": true, "default": null, "desc": "model info file path, mostly use json file" }, "batch_size": { "type": "int", "required": true, "default": null, "max": 8, "min": 1, "desc": "batch size" } }, "single_input": null, "multi_output": [] } 对应的函数实现如下，这里我们通过创建线程池的方式对图像进行批量推理，BatchSize的大小代表创建线程池的数量，将一个批次的推理结果写入到输出队列中，输出队列不唯一，可以为空或有多个输出队列。def model_infer(share_dict, flowunit_data, queue_dict, data): model_file = flowunit_data["config"]["model_file"] model_info = flowunit_data["config"]["model_info"] batch_size = flowunit_data["config"]["batch_size"] rknn_lite_list = [] for i in range(batch_size): rknn_lite = RKNNLite() rknn_lite.load_rknn(model_file) rknn_lite.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) rknn_lite_list.append(rknn_lite) with open(model_info, "r") as f: model_info = json.load(f) labels = [] for label in list(model_info["model_classes"].values()): labels.append(label) IMG_SIZE = model_info["input_shape"][0][-2:] OBJ_THRESH = model_info["conf_threshold"] NMS_THRESH = model_info["nms_threshold"] exist = False index = 0 while True: index += 1 image_batch = [] if flowunit_data["single_input"] is not None: for i in range(batch_size): data = queue_dict[flowunit_data["single_input"]].get() # 图片数据为None就退出循环 if data["frame"] is None: exist = True break image_batch.append(data) else: break with ThreadPoolExecutor(max_workers=batch_size) as executor: results = list(executor.map(infer_single_image, [(data["frame"], rknn_lite_list[i % batch_size], IMG_SIZE, OBJ_THRESH, NMS_THRESH) for i, data in enumerate(image_batch)])) for i, (boxes, classes, scores) in enumerate(results): classes = [labels[class_id] for class_id in classes] data = image_batch[i] if data.get("boxes") is None: data["boxes"] = boxes data["classes"] = classes data["scores"] = scores else: data["boxes"].extend(boxes) data["classes"].extend(classes) data["scores"].extend(scores) for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) if exist: break # 读取结束，图片数据置为None data["frame"] = None for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) for rknn_lite in rknn_lite_list: rknn_lite.release() 跟踪功能单元的可以对推理结果添加跟踪ID，如果没有推理结果，则直接返回原始数据，其定义如下："kf_tracker": { "config": {}, "single_input": null, "multi_output": [] } 对应的函数代码实现如下：def kf_tracker(share_dict, flowunit_data, queue_dict, data): tracker = CentroidKF_Tracker(max_lost=30) index = 0 while True: index += 1 if flowunit_data["single_input"] is not None: data = queue_dict[flowunit_data["single_input"]].get() else: break # 图片数据为None就退出循环 if data["frame"] is None: break boxes, classes, scores = data.get("boxes"), data.get("classes"), data.get("scores") boxes = np.array(boxes) classes = np.array(classes) scores = np.array(scores) boxes[:, 2] = boxes[:, 2] - boxes[:, 0] boxes[:, 3] = boxes[:, 3] - boxes[:, 1] results = tracker.update(boxes, scores, classes) boxes = [] classes = [] scores = [] tracks = [] for result in results: frame_num, id, bb_left, bb_top, bb_width, bb_height, confidence, x, y, z, class_id = result boxes.append([bb_left, bb_top, bb_left + bb_width, bb_top + bb_height]) classes.append(class_id) scores.append(confidence) tracks.append(id) data["boxes"] = boxes data["classes"] = classes data["scores"] = scores data["tracks"] = tracks for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) # 读取结束，图片数据置为None data["frame"] = None for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) 绘制功能单元可以对检测和跟踪结果进行绘制，如果检测结果或跟踪结果为空，则直接返回原始数据，其定义如下："draw_boxes": { "single_input": null, "config": {}, "multi_output": [] } 代码逻辑如下：def draw_boxes(share_dict, flowunit_data, queue_dict, data): index = 0 while True: index += 1 if flowunit_data["single_input"] is not None: data = queue_dict[flowunit_data["single_input"]].get() else: break # 图片数据为None就退出循环 if data["frame"] is None: break boxes, classes, scores = data.get("boxes"), data.get("classes"), data.get("scores") if boxes is not None: tracks = data.get("tracks") if tracks is not None: for boxe, clss, track in zip(boxes, classes, tracks): cv2.rectangle(data["frame"], (boxe[0], boxe[1]), (boxe[2], boxe[3]), (0, 255, 0), 2) cv2.putText(data["frame"], f"{clss} {track}", (boxe[0], boxe[1] - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2) else: for boxe, clss, conf in zip(boxes, classes, scores): cv2.rectangle(data["frame"], (boxe[0], boxe[1]), (boxe[2], boxe[3]), (0, 255, 0), 2) cv2.putText(data["frame"], f"{clss} {conf * 100:.2f}%", (boxe[0], boxe[1] - 10), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2) for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) # 读取结束，图片数据置为None data["frame"] = None for queue_name in flowunit_data["multi_output"]: queue_dict[queue_name].put(data) 输出功能单元可以将视频帧编码成视频输到到视频文件或者推流到RTMP服务器，其参数定义如下："push_frame": { "config": { "push_video_url": { "type": "str", "required": true, "default": null, "desc": "push video url", "source": "rtmp|flv|mp4" }, "format": { "type": "str", "required": true, "default": null, "desc": "vodeo format", "source": "flv|mp4" }, "height": { "type": "int", "required": true, "default": null, "max": 1920, "min": 720, "desc": "video height" }, "width": { "type": "int", "required": true, "default": null, "max": 1920, "min": 960, "desc": "video width" }, "fps": { "type": "int", "required": true, "default": null, "max": 15, "min": 5, "desc": "frame rate" } }, "single_input": null } push_video_url参数是推流地址，也可以输出到本地视频文件。format参数指定视频格式，支持flv和mp4。height和width为视频分辨率，fps是输出帧率。它仅作为消费者，具体函数代码实现如下：def push_frame(share_dict, flowunit_data, queue_dict, data): push_video_url = flowunit_data["config"]["push_video_url"] format = flowunit_data["config"]["format"] height = flowunit_data["config"]["height"] width = flowunit_data["config"]["width"] fps = flowunit_data["config"]["fps"] process_stdin = ( ffmpeg .input('pipe:', format='rawvideo', pix_fmt='bgr24', s="{}x{}".format(width, height), framerate=fps) .filter('fps', fps=fps, round='up') .output( push_video_url, vcodec='h264_rkmpp', bitrate='2500k', f=format, g=fps, an=None, timeout='0' ) .overwrite_output() .run_async(cmd=["ffmpeg", "-re"], pipe_stdin=True) ) index = 0 while True: index += 1 if flowunit_data["single_input"] is not None: data = queue_dict[flowunit_data["single_input"]].get() else: break # 图片数据为None就退出循环 if data["frame"] is None: break frame = data["frame"] frame = cv2.resize(frame, (width, height)) process_stdin.stdin.write(frame.tobytes()) process_stdin.stdin.close() process_stdin.terminate() 消息功能单元可以将检测或跟踪结果发送到Redis服务器，具体可以根据实际情况进行调整。"redis_push": { "config": { "task_id": { "type": "str", "required": true, "default": null, "desc": "task id" }, "host": { "type": "str", "required": true, "default": null, "desc": "redis host" }, "port": { "type": "int", "required": true, "default": null, "desc": "redis port" }, "username": { "type": "str", "required": true, "default": null, "desc": "redis username" }, "password": { "type": "str", "required": true, "default": null, "desc": "redis password" }, "db": { "type": "int", "required": true, "default": null, "desc": "redis db" } }, "single_input": null } 同样，它也仅作为消费者，只有一个输入，具体函数代码如下：def redis_push(share_dict, flowunit_data, queue_dict, data): task_id = flowunit_data["config"]["task_id"] host = flowunit_data["config"]["host"] port = flowunit_data["config"]["port"] username = flowunit_data["config"]["username"] password = flowunit_data["config"]["password"] db = flowunit_data["config"]["db"] r = redis.Redis( host = host, port = port, username = username, password = password, db = db, decode_responses = True ) index = 0 while True: index += 1 if flowunit_data["single_input"] is not None: data = queue_dict[flowunit_data["single_input"]].get() else: break # 图片数据为None就退出循环 if data["frame"] is None: break track_objs = [] height, width = data["frame"].shape[:2] boxes, classes, scores, tracks = data.get("boxes"), data.get("classes"), data.get("scores"), data.get("tracks") if boxes is not None: for boxe, clss, conf, track in zip(boxes, classes, scores, tracks): x1 = float(boxe[0] / width) y1 = float(boxe[1] / height) x2 = float(boxe[2] / width) y2 = float(boxe[3] / height) track_obj = { "bbox": [x1, y1, x2, y2], "track_id": int(track), "class_id": 0, "class_name": str(clss) } track_objs.append(track_obj) key = 'vision:track:' + str(task_id) + ':frame:' + str(index) value = json.dumps({"track_result": track_objs}) r.set(key, value) r.expire(key, 2) print(track_objs) r.close() 二、流程图编排定义好节点，我们就可以定义管道也就是“边”将“节点”的输入和输出连接起来，这里我们定义6条边也就是实例化6个队列，在配置文件中声明每条管道的名称以及队列的最大容量。"queue_size": 16, "queue_list": [ "frame_queue", "infer_queue_1", "infer_queue_2", "track_queue", "draw_queue_1", "draw_queue_2" ] 之后就是对每一个节点的参数进行配置，并定义功能单元的输入和输出。"graph_edge": { "读流功能单元": { "read_frame": { "config": { "pull_video_url": "/home/orangepi/workspace/modelbox/data/car.mp4", "height": 720, "width": 1280, "fps": 20 }, "multi_output": [ "frame_queue" ] } }, "推理功能单元": { "model_infer": { "config": { "model_file": "/home/orangepi/workspace/modelbox/model/yolov8n_800x800_int8.rknn", "model_info": "/home/orangepi/workspace/modelbox/model/yolov8n_800x800_int8.json", "batch_size": 8 }, "single_input": "frame_queue", "multi_output": [ "infer_queue_1", "infer_queue_2" ] } }, "跟踪功能单元_2": { "kf_tracker": { "config": {}, "single_input": "infer_queue_2", "multi_output": [ "track_queue" ] } }, "绘图功能单元_1": { "draw_boxes": { "config": {}, "single_input": "infer_queue_1", "multi_output": [ "draw_queue_1" ] } }, "绘图功能单元_2": { "draw_boxes": { "single_input": "track_queue", "config": {}, "multi_output": [ "draw_queue_2" ] } }, "推流功能单元_1": { "push_frame": { "config": { "push_video_url": "/home/orangepi/workspace/modelbox/output/det_result.mp4", "format": "mp4", "height": 720, "width": 1280, "fps": 20 }, "single_input": "draw_queue_1" } }, "推流功能单元_2": { "push_frame": { "config": { "push_video_url": "/home/orangepi/workspace/modelbox/output/track_result.mp4", "format": "mp4", "height": 720, "width": 1280, "fps": 20 }, "single_input": "draw_queue_2" } } } 每个功能单元需要起一个节点名称用于功能单元的创建，每个节点名称保证全局唯一，正如字典中的键值不能重复。之后根据这份图文件编排启动AI应用，Python代码如下：import os import sys import json import argparse sys.path.append(os.path.join(os.path.dirname(__file__), '..')) from etc.flowunit import * from multiprocessing import Process, Queue, Manager if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('graph_path', type=str, nargs='?', default='/home/orangepi/workspace/modelbox/graph/person_car.json') args = parser.parse_args() # 初始化数据 data = {"frame": None} config = {} # 读取流程图 with open(args.graph_path) as f: graph = json.load(f) # 创建队列 queue_dict = {} queue_size = graph["queue_size"] for queue_name in graph["queue_list"]: queue_dict[queue_name] = Queue(maxsize=queue_size) with Manager() as manager: # 创建共享字典 share_dict = manager.dict() # 创建进程 process_list = [] graph_edge = graph["graph_edge"] for process in graph_edge.keys(): p = Process(target=eval(list(graph_edge[process].keys())[0]), args=(share_dict, list(graph_edge[process].values())[0], queue_dict, data,)) process_list.append(p) print("=============Start Process...=============") # 启动进程 for p in process_list: p.start() # 等待进程结束 for p in process_list: p.join() print("==========All Process Finished.===========") 这里我们读取一段测试视频分别将检测结果和跟踪结果保存为两个视频文件输出到output目录下：(python-3.9.15) orangepi@orangepi5plus:~$ python /home/orangepi/workspace/modelbox/graph/graph.py /home/orangepi/workspace/modelbox/graph/person_car.json =============Start Process...============= ffmpeg version 04f5eaa Copyright (c) 2000-2023 the FFmpeg developers built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04) configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga libavutil 58. 29.100 / 58. 29.100 libavcodec 60. 31.102 / 60. 31.102 libavformat 60. 16.100 / 60. 16.100 libavdevice 60. 3.100 / 60. 3.100 libavfilter 9. 12.100 / 9. 12.100 libswscale 7. 5.100 / 7. 5.100 libswresample 4. 12.100 / 4. 12.100 libpostproc 57. 3.100 / 57. 3.100 ffmpeg version 04f5eaa Copyright (c) 2000-2023 the FFmpeg developers built with gcc 11 (Ubuntu 11.4.0-1ubuntu1~22.04) configuration: --prefix=/usr --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga libavutil 58. 29.100 / 58. 29.100 libavcodec 60. 31.102 / 60. 31.102 libavformat 60. 16.100 / 60. 16.100 libavdevice 60. 3.100 / 60. 3.100 libavfilter 9. 12.100 / 9. 12.100 libswscale 7. 5.100 / 7. 5.100 libswresample 4. 12.100 / 4. 12.100 libpostproc 57. 3.100 / 57. 3.100 W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.190] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.191] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.192] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.248] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.338] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.338] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.339] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.384] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.459] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.459] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.460] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.504] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.606] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.606] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.608] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.658] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.761] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.761] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.762] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.814] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:47.910] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:47.910] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:47.912] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:47.962] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:48.069] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:48.070] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:48.071] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:48.122] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) W rknn-toolkit-lite2 version: 2.3.2 I RKNN: [13:10:48.228] RKNN Runtime Information, librknnrt version: 2.3.2 (429f97ae6b@2025-04-09T09:09:27) I RKNN: [13:10:48.228] RKNN Driver Information, version: 0.9.6 I RKNN: [13:10:48.229] RKNN Model Information, version: 2, toolkit version: 1.4.0-22dcfef4(compiler version: 1.4.0 (3b4520e4f@2022-09-05T12:50:09)), target: RKNPU v2, target platform: rk3588, framework name: ONNX, framework layout: NCHW, model inference type: static_shape W RKNN: [13:10:48.280] query RKNN_QUERY_INPUT_DYNAMIC_RANGE error, rknn model is static shape type, please export rknn with dynamic_shapes W Query dynamic range failed. Ret code: RKNN_ERR_MODEL_INVALID. (If it is a static shape RKNN model, please ignore the above warning message.) Input #0, rawvideo, from 'pipe:': Duration: N/A, start: 0.000000, bitrate: 442368 kb/s Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24, 1280x720, 442368 kb/s, 20 tbr, 20 tbn Stream mapping: Stream #0:0 (rawvideo) -> fps:default fps:default -> Stream #0:0 (h264_rkmpp) Output #0, mp4, to '/home/orangepi/workspace/modelbox/output/det_result.mp4': Metadata: encoder : Lavf60.16.100 Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), bgr24(progressive), 1280x720, q=2-31, 2000 kb/s, 20 fps, 10240 tbn Metadata: encoder : Lavc60.31.102 h264_rkmpp Input #0, rawvideo, from 'pipe:': 0kB time=N/A bitrate=N/A speed=N/A Duration: N/A, start: 0.000000, bitrate: 442368 kb/s Stream #0:0: Video: rawvideo (BGR[24] / 0x18524742), bgr24, 1280x720, 442368 kb/s, 20 tbr, 20 tbn Stream mapping: Stream #0:0 (rawvideo) -> fps:default fps:default -> Stream #0:0 (h264_rkmpp) Output #0, mp4, to '/home/orangepi/workspace/modelbox/output/track_result.mp4': Metadata: encoder : Lavf60.16.100 Stream #0:0: Video: h264 (High) (avc1 / 0x31637661), bgr24(progressive), 1280x720, q=2-31, 2000 kb/s, 20 fps, 10240 tbn Metadata: encoder : Lavc60.31.102 h264_rkmpp [out#0/mp4 @ 0x558f2625e0] video:2330kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.062495% frame= 132 fps= 19 q=-0.0 Lsize= 2331kB time=00:00:06.55 bitrate=2915.8kbits/s speed=0.924x Exiting normally, received signal 15. [out#0/mp4 @ 0x557e4875e0] video:1990kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.072192% frame= 131 fps= 18 q=-0.0 Lsize= 1991kB time=00:00:06.50 bitrate=2509.6kbits/s speed= 0.9x ==========All Process Finished.=========== Exiting normally, received signal 15.应用推理的帧率取决于视频读取的帧率以及耗时最久的功能单元，实测FPS约为20左右，满足AI实时检测的场景。

源码编译，环境搭建在瑞芯微RK3588芯片的边缘盒子上，modelbox服务都可以正常启动，运行正常，问题是：在启动时报下面的错误，无法运行rknn模型，看日志是无法找到rknn驱动和一个库的依赖无法找到，但是尝试了各种方法还是无法解决谁能给点思路，谢谢，解决后我会分享构建过程

我是在rk3588型号的设备上部署modelbox，现在遇到如下问题：1.相关文档不是健全，不知道如何入手，参考了一些帖子遇到了编译问题，环境配置等问题，还在不断摸索。2.源码编译说明太简单了，遇到问题不知道该如何解决，希望能够增加对应的文档，并且更新下源码就更好了。3.现在镜像也无法下载到，所以通过镜像也无法解决环境搭建问题。如果能够协助编译一个rk系列的镜像就更好了。希望能够针对上面问题的提示和帮助，谢谢

modelbox 源码执行时无法访问obs，导致desc.json 文件无法下载，功能不能正常使用，截图如下查看solution.py脚本代码时应该是无法访问https://obs.cn-north-4.myhuaweicloud.com导致的，请确认。

the scheduler caught an error:stop operation这是什么报错

使用yolov3进行口罩检测。与车辆检测不同，口罩检测需要自行准备模型和配置文件，但这并没有增加太多难度，因为所需资源的下载和使用都非常便捷。工程创建与模型配置 使用create.py脚本创建了名为mask_det_yolo3的工程，并利用-t infer选项创建了名为mask_infer的推理功能单元。这一步骤在工程目录的model下生成了必要的配置文件。模型与配置文件的整合 将下载的rknpu2.zip中的rknn模型文件和配置文件复制到工程目录中，替换原有的mask_infer.toml。mask_infer.toml文件定义了模型输出，涉及头部、面部和口罩的信息。功能单元的开发 接着，使用-t python选项创建了两个功能单元：yolo3_post用于后处理，draw_mask_bbox用于在原图上绘制展示框。这些功能单元的逻辑是判断是否佩戴口罩——如果面部检测框与口罩检测框的重合度超过设定阈值，则判定为佩戴口罩；否则，判定为未佩戴。流程图与任务配置 串接流程图（graph/mask_det_yolo3.toml）和配置任务输入输出（bin/mock_task.toml）是实现口罩检测的关键步骤。下载的资源中已经包含了配置好的文件，只需拷贝到相应目录即可。测试视频的准备与应用运行 准备工作完成后，将测试视频拷贝到data目录，并在工程目录下执行build_project.sh和bin/main.sh来运行应用。视频文件中的戴口罩检测结果被成功输出。视频流的探索 下载的资源中还包括了一个camera配置文件，该文件允许将输入输出都配置为视频流，这为实时口罩检测提供了可能。结语 通过这次试用，ModelBox端云协同AI开发套件的口罩检测功能得到了充分的展示。从模型下载到配置，再到功能单元的开发和流程图的串接，每一个步骤都体现了ModelBox的灵活性和强大能力。这不仅是技术的实践，更是对AI应用潜力的一次深入挖掘。

初识ModelBox：SDK的快速入门 试用的第一步，自然是从SDK的安装开始。考虑到效率，直接以root用户身份进行操作。安装了必要的库dos2unix和numpy opencv-python后，解压缩SDK包，一个丰富的开发环境展现在眼前，包含了bin、doc、etc等多个目录，每个目录都有其独特的用途。创建Hello World工程：探索目录结构 使用create.py脚本创建了一个名为hello_world的工程，这个工程的目录结构清晰有序，从CMake文件到bin目录，再到data和dependence等，每一个部分都为接下来的开发工作做好了准备。开发第一个Flowunit：视频上的文字叠加 在etc/flowunit目录下，新增了一个名为draw_text的python flowunit。通过修改draw_text.py文件中的process函数，实现了在视频帧上叠加"Hello World"文字的功能。这一改动，虽然简单，却标志着功能单元处理逻辑的初步构建。配置流程图和任务：从本地视频到RTSP流 接着，调整了hello_world.toml流程图配置和mock_task.toml任务配置，使得应用能够处理本地的mp4视频文件，并将处理结果输出到指定路径。执行构建和运行脚本后，虽然遇到了一些报错信息，但目标文件成功生成，视频播放时叠加了预期的文字效果。扩展到视频流：实时文字叠加 不满足于仅处理视频文件，进一步尝试了视频流的处理。通过修改配置文件，将输入源改为摄像头，输出到RTSP流，实现了在实时视频流上叠加文字的功能。车辆检测：探索ModelBox的高级应用 在完成基础的文字叠加后，进一步探索了ModelBox的高级应用——车辆检测。使用内置的yolox模型进行视频画面中车辆的检测，并在检测到的车辆周围绘制边框。这一过程涉及到模型推理单元的使用，ModelBox内置的rknn推理引擎和推理逻辑，大大简化了开发过程。HTTP服务：提供远程车辆检测API 更进一步，通过配置http服务，使得车辆检测功能可以通过HTTP接口进行调用，这为远程车辆检测提供了便利。结语 通过这一系列的尝试和探索，ModelBox端云协同AI开发套件展现出了其强大的功能和灵活性。无论是视频上的文字叠加，还是复杂的车辆检测，ModelBox都能够提供高效的解决方案。这不仅是一次技术的尝试，更是一次对AI开发潜力的深刻认识。

在一个月的试用期内，有幸体验了ModelBox端云协同AI开发套件（RK3568），心中满是激动。收到开发套件的那一刻，便迫不及待地按照《上手指南》开始了探索之旅。首先映入眼帘的是整齐摆放的开发套件组件，随后在安装过程中，逐步展现出其形态。待板子上电并启动系统后，一切准备就绪。操作系统的制作是第一步，没有系统的板子上电后仅显示绿灯常亮。系统启动镜像需要写入到Micro SD卡中，而家中的读卡器恰好派上用场。虽然balenaEtcher镜像读写工具下载缓慢，但急性子的我选择放弃等待，转而在Linux环境下，使用Ubuntu 20.04系统自带的磁盘映像写入器完成了镜像的写入。将镜像成功写入32G的TF卡后，系统启动便水到渠成。接下来，面临两种登录开发板的方式选择：一是直接连接显示器和键盘鼠标，安装Ubuntu桌面进行开发；二是通过远程连接工具，如VS Code中的Remote-SSH，从PC端登录开发板。为了便捷，选择了前者，将hdmi线连接至荣耀智慧屏作为显示器，并通过nmcli命令连接无线网络，顺利完成了系统登录。系统登录后，安装了桌面环境mate，虽然下载和安装过程中遇到了一些挑战，如系统源替换和自动升级的关闭，但最终都得到了解决。进入桌面后，虽然操作略显卡顿，但通过调整分辨率和关闭一些系统进程，问题得到了缓解。随后，按照指南在HiLens管理控制台专业版进行了设备注册、下载固件（Device_Agent）和证书，并将其传送到开发板进行安装。值得一提的是，使用十元优惠券激活了一个月的设备使用权。开发环境的准备工作告一段落后，便开始了SDK的下载和真正的开发工作。整个试用过程充满了挑战和学习，ModelBox端云协同AI开发套件（RK3568）的体验让人印象深刻，期待未来能够探索更多的可能性。

边缘盒子芯片不支持gpu（海思3521DV200）需要运行cpu版本推理模型，谁可以提供一个modelbox torch版本镜像，搞了几天了始终不行，运行就报环境不支持。

我使用的是modelbox-develop-mindspore_1.6.1-cann_5.0.4-ubuntu-aarch64镜像，按照教程运行modelbox-tool develop -s正常，但是运行modelbox命令提示找不到文件，导致我无法修改ACL,浏览器访问图排界面显示拒绝访问。界面如下：

例如对一路图像同时进行OCR识别和目标检测，有案例吗？