多模态模型在做开发检测任务时，如何区分真正未知目标和相似类别，并且避免过度自信误检？

在进行预测任务时，如何在不增加token的前提下，兼顾全局与细节？

unet架构中跳跃连接有什么作用？

 温州市数据局（市人工智能局）关于邀请研发团队开展“人工智能+”场景研发的公告 各有关高校、科研院所、人工智能企业及研发团队：为深入贯彻国家“人工智能+”行动部署，加快推进人工智能与实体经济深度融合，打造人工智能创新发展先行市，我局依托“AI行业翻译官”机制，深入全市重点行业企业开展需求摸排，累计形成三批“人工智能+”场景需求榜单，共269项。现面向社会公开发布，诚邀广大研发团队来温开展场景研发与落地转化。现将有关事项公告如下：一、需求概况本次发布的场景需求共269项，覆盖汽车零部件、电气、泵阀、智能装备、鞋服、生命健康、数字经济、社会治理、新材料、新能源、机械制造、教玩具等多个行业领域。场景需求均来源于企业真实生产经营场景，每项均列明业务痛点、技术诉求，可直接用于研发立项与方案设计。（具体场景清单见以下链接：https://www.kdocs.cn/l/cmrXWoFs5Rjq）二、邀请对象（一）在人工智能、机器视觉、工业软件、智能装备等领域具备研发与工程化落地能力的高等院校、科研院所及创新团队；（二）从事机器视觉质检、预测性维护、智能排产、行业大模型、文档智能、生成式设计等方向技术研发的科技企业与创业团队；（三）有意向来温开展场景研发、成果转化或设立研发机构的各类创新主体。三、报名与联系方式有意向的团队可结合自身技术方向，对应榜单中的相关场景需求，向我局提交团队简介、近年相关落地案例、拟对接的场景方向及初步技术思路，由我局统一登记并转交需求企业。联系人及联系方式：温州市数据局（市人工智能局），0577-88961376。温州市图灵人工智能高等研究院，19706095685，WZTLAI@163.com。温州市数据局（市人工智能局）2026年10月23日 

计算机视觉的基本定义和核心目标是什么？

前言2026 年备受关注的开源 AI 智能体 OpenClaw（昵称小龙虾），GitHub 星标突破 28 万，凭借本地运行、零代码配置、自动执行任务的特性获得广泛认可。本文面向零基础用户，采用整合优化的一键部署包，无需命令行操作、无需手动配置环境，10 分钟即可完成部署。跟随教程，你也可以搭建专属 “数字员工”，高效处理各类电脑操作。适配平台：Windows 10/11（64 位）｜新手友好｜全程可视化｜低技术门槛点击下方链接，获取最新版 OpenClaw Windows 一键部署包，无需注册，直接下载：下载地址： 补充：文件大小约 361MB，下载速度取决于网络环境，建议使用浏览器自带下载工具或稳定下载软件，避免中断。下载完成后，在桌面或下载目录会得到一个 .zip 压缩包。后续将持续更新 OpenClaw 技能扩展、本地大模型接入、聊天工具联动等进阶教程，手把手教你把 “小龙虾” 调教为全能助手。一、OpenClaw（小龙虾）是什么？核心优势拆解很多人误以为 OpenClaw 仅是对话型 AI，实际上它是能够直接操控电脑的数字员工—— 接收自然语言指令后，自动拆解任务、调用工具、完成操作，全程无需人工干预。因 Logo 为红色龙虾，“Claw” 寓意执行任务的 “钳子”，国内用户将部署与调试过程戏称为 “养虾”。自 2026 年初获得关注后，GitHub 星标快速增长至 28 万 +，成为近年热度较高的开源 AI 项目。核心优势（新手必看）✅ 本地运行：数据留存于本地设备，隐私安全性强，降低敏感信息泄露风险。✅ 零代码门槛：无需编程基础、无需命令行操作，一键部署即可使用。✅ 跨平台兼容：支持 Windows/Mac/Linux 系统，可对接微信 / 飞书 / Slack 等平台，远程下达指令。✅ 开箱即用：整合版一键部署包内置全部运行依赖与基础技能，解压即可启动。✅ 全能执行：支持文件整理、邮件发送、表格制作、浏览器自动化、数据处理等场景。二、安装前必看（避坑关键！不看易踩雷）安装、解压与运行前，务必彻底关闭 360 安全卫士、360 杀毒、腾讯电脑管家、火绒等所有安全软件。重点说明：OpenClaw 需要操作系统资源、读写本地文件、模拟键鼠操作，这类行为易被安全软件误判为风险程序，进而拦截或删除核心文件，导致部署失败或无法启动。正确操作：关闭所有安全软件后，再解压安装包并运行启动程序，可正常使用。项目为开源性质，可前往 GitHub 核验安全性。三、第一步：下载并解压一键部署包1. 获取整合包点击下方链接，获取最新版 OpenClaw Windows 一键部署包，无需注册，直接下载：下载地址：  补充：文件大小约 361MB，建议使用浏览器自带下载工具或稳定下载软件，避免中断。下载完成后，在桌面或下载目录会得到一个 .zip 压缩包。2. 解压文件（避免损坏）⚠️ 不建议使用 Windows 自带解压工具，易出现文件损坏或权限不足问题。推荐使用WinRAR 或 7-Zip（常规版本即可）。操作步骤：找到下载完成的 Openclaw-Windows-2.3.12.zip 压缩包。  右键点击压缩包，选择「解压到当前文件夹」或「解压到 "Openclaw-Windows-2.3.12"」。  解压完成后，得到 Openclaw-win 文件夹，确认内含 Openclaw Windows一键启动.exe（红色龙虾图标），即解压正常  。四、第二步：启动一键安装程序（可视化操作）进入解压后的 Openclaw-win 文件夹，双击 Openclaw Windows一键启动.exe。  若弹出「Windows 已保护你的电脑」提示，点击「更多信息」→「仍要运行」，放行程序。启动后进入 OpenClaw 欢迎界面，点击「开始使用」进入安装配置环节。五、第三步：完成部署安装（自动运行，无需干预）1. 设置安装路径（关键）   必须使用纯英文路径，禁止包含中文、空格或特殊符号。推荐路径：plaintextD:\OpenClawE:\AI\OpenClaw禁止路径示例：plaintextD:\软件\OpenClawD:\小龙虾C:\Program Files\OpenClaw2. 开始自动安装勾选用户协议后，点击「开始安装」，程序自动完成以下操作：  检测 Windows 系统环境与依赖安装运行所需组件部署核心服务与配置适配系统权限设置创建桌面快捷方式⚠️ 注意：部署过程中请勿关闭窗口，否则会导致部署中断，需重新操作。3. 第一次启动等待（正常现象）  安装完成后，程序自动启动 OpenClaw 主程序。第一次启动时，Gateway 服务需初始化，会显示加载提示，耐心等待 1-3 分钟即可自动跳转至对话窗口。后续启动速度显著提升，几秒内即可打开。六、第四步：开始使用你的 “数字员工”（附实操示例）  部署完成后进入 OpenClaw 主界面，右上角显示 **「Gateway 在线」** 即代表部署成功，可正式开始使用。主界面说明（易懂版）右上角：Gateway 状态（在线为正常，离线为异常）、重启按钮、运行日志、Tokens 额度（内置 28 万额度，可体验基础功能）。左侧菜单栏：切换「本地」「渠道」，查看历史对话记录。底部输入框：直接发送自然语言指令，触发 AI 执行任务。实操示例（直接复制可用）无需复杂设置，在输入框发送指令，OpenClaw 会自动拆解并执行：帮我整理 D 盘下载文件夹里的图片，按日期分类并新建对应文件夹存放。打开浏览器，搜索 2026 年 AI 发展趋势，整理为 Excel 表格并保存至桌面。打开微信，给备注 “同事 A” 发送消息：本周工作总结已发送至你邮箱，请查收。遍历桌面所有 Word 文档，提取标题与核心内容并生成汇总表格。✅ 提示：指令描述越具体，AI 执行精度越高，建议明确标注分类方式、保存路径等细节。七、常见问题与避坑指南（新手必看）汇总部署与使用中高频出现的 4 个问题，遇到问题可快速参考：Q1：启动时被安全软件拦截，核心文件被删除？A：彻底关闭所有安全软件（含后台进程），重新解压安装包并运行启动程序；若文件已被隔离，在安全软件隔离区恢复 Openclaw-win 文件夹内全部文件，再重新部署。Q2：安装提示 “路径包含中文 / 特殊字符”，无法继续？A：修改安装路径为纯英文（如 D:\OpenClaw），删除中文、空格与特殊符号，重新点击「开始安装」。Q3：Gateway 持续显示离线，无法发送指令？A：1. 确认安全软件已关闭、安装路径为纯英文；2. 点击主界面右上角「重启」按钮，重启 Gateway 服务；3. 若无效，关闭 OpenClaw 后重新运行「一键启动.exe」。Q4：第一次启动加载缓慢，长时间显示加载中？A：属于正常现象！第一次启动需初始化依赖文件与服务，等待 1-3 分钟即可，后续启动会快速完成。八、结尾福利 & 引流（关注获取更多干货）恭喜你完成 OpenClaw 部署，成功拥有专属 “数字员工”，告别重复繁琐的电脑操作！后续福利 & 教程预告OpenClaw 技能扩展：添加 PDF 转 Word、批量邮件发送等实用功能。本地大模型接入：实现离线使用 OpenClaw，进一步保障隐私安全。聊天工具联动：将 OpenClaw 接入微信 / 飞书，随时远程下达指令。常见问题汇总：梳理 “养虾” 过程中各类问题的解决方案。再次附上下载地址，方便获取：一键部署包：  ❤️ 觉得内容实用，欢迎点赞、收藏、关注，你的支持是持续更新的动力，后续将分享更多 OpenClaw 实操干货。 

前言：语言不是世界的全部过去两年，大语言模型（LLM）让我们惊叹于AI对文字的理解与生成能力。但一个显而易见的事实是：人类感知世界，从来不只是靠语言。我们看到一张照片，瞬间能理解其中的情绪、关系、场景；我们看过一段视频，能记住动作的连贯与冲突；我们扫描一份扫描件，能区分哪些是文字、哪些是印章、哪些是手写批注。这些能力，传统的大语言模型都不具备——因为它只有“耳朵”，没有“眼睛”。多模态大模型的出现，填平了这个鸿沟。它让AI同时处理文本、图像、视频、音频，让机器真正拥有了“看懂”世界的能力。而对于企业而言，这意味着大量以前无法处理的业务场景，现在有了全新的解决方案。一、什么是多模态与视觉大模型？1.1 基本概念多模态大模型是指能够理解和生成多种模态信息（文本、图像、视频、音频、3D等）的统一模型架构。其中，视觉大模型是多模态领域中最核心、应用最广泛的分支，专注于图像和视频的理解与生成。用一个简单的类比：传统LLM多模态大模型只能读文字描述可以直接“看”图片听说过“猫”这个字能认出照片里那只橘猫分析Excel表格数据能看懂扫描的合同、发票、手写便签生成文字方案能画出设计图、标注病灶、生成视频脚本+画面1.2 关键能力对比能力维度传统视觉模型（如ResNet、YOLO）多模态大模型（如GPT-4V、Qwen-VL、LLaVA）任务范围单一任务（分类、检测、分割）通用理解+对话+推理开放性固定类别（如1000类ImageNet）开放词汇，任意描述与语言融合分离（感知后交给文字模型）原生融合，图文互相解释少样本能力需要大量标注零样本/少样本即可泛化复杂推理弱（只能给标签）强（说出为什么、怎么做）一句话总结：传统模型是“会分类的眼睛”，多模态大模型是“会思考的眼睛+会说话的大脑”。二、核心技术架构2.1 主流架构范式当前多模态大模型的主流架构是三段式：text代码语言：javascript AI代码解释 输入图像 → 视觉编码器 → 特征对齐模块 → 大语言模型 → 文本输出 (Vision Encoder) (Projector/Adapter) (LLM)视觉编码器：将图像切块，提取视觉特征。代表模型：CLIP ViT、SigLIP、EVA-CLIP对齐模块：将视觉特征映射到LLM的文本特征空间（简单的MLP或交叉注意力）大语言模型：接受对齐后的视觉token + 文本token，统一生成回答代表性模型：LLaVA：最简单架构，MLP对齐，效果惊人地好Qwen-VL-Max：阿里系，中文支持优秀，文档理解强GPT-4V/4o：闭源，多模态推理能力最强Claude 3.5：长上下文+视觉，适合大文档分析CogVLM：深度融合视觉与语言，所有参数都参与2.2 生成式多模态理解只是第一步。生成式多模态模型可以画出内容：文生图：Stable Diffusion 3、Midjourney、DALL-E 3、FLUX图生图：ControlNet、InstructPix2Pix视频生成：Sora、Runway Gen-3、Kling多模态对话生成：Gemini、GPT-4o（实时语音+视觉）企业场景中，往往是“理解+生成”组合使用：先分析现有图片（多模态理解模型），再生成修改版或衍生图（扩散模型）。2.3 视频理解与处理视频是多模态中最具挑战的模态，因为涉及时间维度。主要技术路线：关键帧采样：抽N帧，当成多张图片处理（简单，但丢失时序）时空注意力：在帧之间加时间注意力层（如Video-LLaMA）原生视频模型：统一处理时空token（挑战极大，但长远方向）企业级稳定可用的方案目前主要是关键帧+时序提示，配合专有视频理解模型（如阿里灵杰视频大模型）。三、企业应用场景落地3.1 智能文档处理 —— 最成熟、ROI最高的场景痛点：企业里到处都是“非结构化文档”——扫描合同、发票、手写便签、带表格的报告、带印章的审批单。传统OCR只能输出文字，无法理解布局、表格、印章位置、签字区域。多模态解决方案：直接输入文档图片，大模型理解布局 + 提取信息 + 结构化输出示例：上传一份采购合同，模型输出“合同编号、双方名称、金额、有效期、关键条款”结构化JSON表格：不仅识别单元格文字，还理解表头与数据行的关系效果：某物流公司：面单识别准确率从OCR的78% → 多模态模型的96%某银行：信用卡申请材料人工审核时间从15分钟 → 2分钟（AI预审+人工抽检）成本对比：多模态API单张图片约0.001-0.01美元，比人工便宜1-2个数量级3.2 工业视觉质检痛点：制造业中产品外观检测依赖人工目检（疲劳、主观、不一致）或传统机器视觉（需定制开发，难以适应新产品）。多模态方案：零样本缺陷检测：输入“划痕 > 2mm、气泡直径 > 1mm”的文字描述，模型自动判断小样本泛化：新产品上线，只需拍5-10张合格/不合格样张，模型即可学会多模态解释：不仅输出“不合格”，还圈出缺陷位置，并说出理由（“左下角有明显划痕”）代表案例：某光伏企业用多模态模型检测电池片隐裂，替代了90%的人工目检，不良漏检率从3%降到0.5%。3.3 电商与内容审核痛点：电商平台每天数亿张商品图、用户晒图、评论图，需要识别违规内容（色情、暴力、政治敏感）以及商品与描述是否一致。多模态方案：图文一致性检查：商品主图 vs 文字描述，判断“是否美颜过度/货不对版”细粒度违规识别：不仅判断“裸露”，还能区分“医疗示意图”和“色情内容”多模态搜索：用户上传一张衣服照片，搜同款（理解款式、颜色、材质）效果：某电商平台内容审核人效提升300%，误杀率下降40%（因为模型不仅匹配关键词，还能理解上下文）。3.4 医疗影像辅助诊断场景：X光片、CT、MRI、病理切片的初步筛查。多模态模型可以同时读取影像 + 临床文本记录 + 检验报告，给出综合建议。需要特别说明：目前能力定位是“辅助”而非“诊断”。落地时必须：通过医疗器械认证（中国NMPA三类或美国FDA）、有人工复核流程、模型可解释（圈出可疑区域）。实际成效：某三甲医院肺部CT结节检出率：医生单人为83%，医生+多模态辅助提升到96%，且读片时间减少35%。3.5 视频监控与行为分析场景：工厂安全帽检测、零售店客流分析、养老院跌倒监测、工地违规操作识别。多模态优势：传统模型只能做单一目标检测（“有没有帽子”）。多模态可以：结合自然语言描述进行事件搜索（“上午10点以后穿红衣服的人在A区域停留超过5分钟”），跨摄像头推理。四、工程化落地关键挑战4.1 计算成本：多模态是“大胃王”一张图片进入多模态模型，会被转换成几百甚至上千个视觉token（相当于几百个单词）。一张高清图可能消耗普通文本对话30-50倍的计算量。应对策略：图像预处理：大图降采样（如最长边1024像素）动态分辨率：根据内容复杂度决定分多少个token缓存：相同图片（如发票模板）复用视觉特征混合使用：简单任务用小型多模态模型（如LLaVA-7B），复杂任务用云端大模型4.2 幻觉问题：视觉幻觉 > 文本幻觉模型可能“看错”甚至“看到不存在的东西”。比如把影子识别为裂缝，把褶皱识别为破损。缓解方法：提示约束：强制要求“如果没有看到明确的X，请回答‘未检测到’”时序一致性（视频）：多帧联合推理，避免单帧误判与确定性算法结合：OCR用于精准文字、检测模型用于定位、大模型用于语义理解，三者投票人工确认机制：高风险场景（医疗、安防）设置低置信度转人工4.3 数据隐私与合规很多视觉数据是高度敏感的——人脸、医疗影像、内部文档。工程实践：本地化部署：使用开源模型（如LLaVA、Qwen-VL）在私有云或边缘端运行数据脱敏：进入模型前自动模糊人脸、车牌、印章审计日志：记录每一次图片输入和模型的输出，以备合规审查分区域处理：不同敏感等级的数据走不同管道4.4 评估难：什么是“看懂了”？文本任务有准确率、BLEU等指标。但多模态理解很难自动化评估——模型说“图中有三个人”，实际上有两个半（一个被遮挡），算对还是错？目前方案：结构化任务用自动指标（如VQA准确率、表格提取的F1值）开放任务用AI评估（更强的模型当裁判，如GPT-4V评估LLaVA）高价值场景保留人工抽检（5%-10%）五、技术选型建议5.1 模型选择指南场景推荐模型原因中文文档、合同、表格Qwen-VL-Max / Qwen2-VL中文预训练充分，OCR与布局理解强通用场景、英文为主GPT-4V / GPT-4o推理能力最强，多模态对话流畅数据不出域、开源可部署LLaVA-NeXT / CogVLM2效果接近闭源，支持本地化视频理解阿里灵杰 / Video-LLaMA专有时序建模高精度低延迟OCRPaddleOCR + 轻量多模态传统OCR先提取文字，再用多模态做理解5.2 避坑指南误区正解什么图都直接扔给大模型低质量图片（模糊、反光、倾斜）先做预处理；无关信息（水印、边框）先裁剪认为模型能“完美识别”所有小字分辨率是关键——小字区域可先检测+裁剪放大忽略图片中的文字多模态模型不一定擅长密集OCR，复杂表格/长文本建议专用OCR+LLM组合直接用于高风险决策产线质检、医疗筛查必须有“人工在回路”六、未来3年演进方向原生多模态模型：不再拼凑视觉编码器+LLM，而是真正统一的原生多模态架构（Gemini走这条路，GPT-4o也是）。好处：任意模态输入输出、时序建模自然、模态间知识深度融合。长视频理解：从3-5秒片段到10分钟以上长视频。技术路线包括记忆压缩、分层注意力、关键事件摘要。企业应用：会议自动摘要、质检长流程监控、培训视频智能切片。多模态Agent：Agent不仅要调用工具，还要能“看”界面。如UI自动化Agent：看一眼网页截图，理解布局，自主执行点击、输入、滚动。对于遗留系统自动化、RPA升级意义重大。3D与空间智能：从2D到3D。大模型理解点云、深度图、多视角图像，应用于工业机器人抓取、仓库库存扫描、AR辅助装配。苹果Vision Pro类设备成熟后将加速这一趋势。端侧多模态：手机、摄像头、边缘设备上运行小规模多模态模型（3B-7B参数）。2025年下半年已出现可流畅运行的端侧模型，2026年将进一步普及。结语：多模态不是“锦上添花”，而是“场景解锁”有一个很能说明问题的案例：某家物流企业原本用传统LLM+RAG做了“智能客服”，效果平平——因为客户经常发来破损包裹的照片，而纯文本模型面对“你看这个地方凹进去了”完全无能为力。换了多模态模型之后，客户直接发图->模型自动识别破损类型、面积、位置->系统自动推送理赔流程。满意度提升了，人工成本下降了。

RK3588部署CNN-LSTM驾驶行为识别模型CNN（卷积神经网络）擅长提取图像的空间特征，LSTM（长短期记忆网络）则擅长处理序列数据的时间特征。首先使用CNN提取视频每一帧特征，之后将提取出的所有特征送入LSTM捕捉视频中的时空特征并对视频特征序列进行分类，实现正常驾驶、闭眼、打哈欠、打电话、左顾右盼5种驾驶行为的识别。一. 模型训练我们在ModelArts创建Notebook完成模型的训练，使用规格是GPU: 1*Pnt1(16GB)|CPU: 8核 64GB，镜像为tensorflow_2.1.0-cuda_10.1-py_3.7-ubuntu_18.04，首先下载数据集：import os import moxing as mox if not os.path.exists('fatigue_driving'): mox.file.copy_parallel('obs://modelbox-course/fatigue_driving', 'fatigue_driving') if not os.path.exists('rknn_toolkit2-2.3.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl'): mox.file.copy_parallel('obs://modelbox-course/rknn_toolkit2-2.3.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl', 'rknn_toolkit2-2.3.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl') 该数据集包含1525段视频，总共有5个类别：0:正常驾驶、1:闭眼、2:打哈欠、3:打电话、4:左顾右盼我们从原视频中裁剪出主驾驶位画面，并将画面缩放到特征提取网络的输入大小：def crop_driving_square(frame): h, w = frame.shape[:2] start_x = w // 2 end_x = w start_y = 0 end_y = h return frame[start_y:end_y, start_x:end_x] 使用在imagenet上预训练的MobileNetV2网络作为卷积基创建并保存图像特征提取器：def get_feature_extractor(): feature_extractor = keras.applications.mobilenet_v2.MobileNetV2( weights = 'imagenet', include_top = False, pooling = 'avg', input_shape = (IMG_SIZE, IMG_SIZE, 3) ) preprocess_input = keras.applications.mobilenet_v2.preprocess_input inputs = keras.Input((IMG_SIZE, IMG_SIZE, 3)) preprocessed = preprocess_input(inputs) outputs = feature_extractor(preprocessed) model = keras.Model(inputs, outputs, name = 'feature_extractor') return model feature_extractor = get_feature_extractor() feature_extractor.save('feature_extractor') feature_extractor.summary() Model: "feature_extractor" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_2 (InputLayer) [(None, 256, 256, 3)] 0 _________________________________________________________________ tf_op_layer_truediv (TensorF [(None, 256, 256, 3)] 0 _________________________________________________________________ tf_op_layer_sub (TensorFlowO [(None, 256, 256, 3)] 0 _________________________________________________________________ mobilenetv2_1.00_224 (Model) (None, 1280) 2257984 ================================================================= Total params: 2,257,984 Trainable params: 2,223,872 Non-trainable params: 34,112 设置网络的输入大小为256x256，每隔6帧截取一帧提取视频的图像特征，特征向量的大小为1280，最终得到每个视频的特征序列，序列的最大长度为40，不足用0补齐：def load_video(file_name): cap = cv2.VideoCapture(file_name) frame_interval = 6 frames = [] count = 0 while True: ret, frame = cap.read() if not ret: break if count % frame_interval == 0: frame = crop_driving_square(frame) frame = cv2.resize(frame, (IMG_SIZE, IMG_SIZE)) frame = frame[:, :, [2, 1, 0]] frames.append(frame) count += 1 return np.array(frames) def load_data(videos, labels): video_features = [] for video in tqdm(videos): frames = load_video(video) counts = len(frames) # 如果帧数小于MAX_SEQUENCE_LENGTH if counts < MAX_SEQUENCE_LENGTH: # 补白 diff = MAX_SEQUENCE_LENGTH - counts # 创建全0的numpy数组 padding = np.zeros((diff, IMG_SIZE, IMG_SIZE, 3)) # 数组拼接 frames = np.concatenate((frames, padding)) # 获取前MAX_SEQUENCE_LENGTH帧画面 frames = frames[:MAX_SEQUENCE_LENGTH, :] # 批量提取图像特征 video_feature = feature_extractor.predict(frames) video_features.append(video_feature) return np.array(video_features), np.array(labels) video_features, classes = load_data(videos, labels) video_features.shape, classes.shape((1525, 40, 1280), (1525,)) 总共提取了1525个视频的特征序列，按照8:2的比例划分训练集和测试集（batchsize的大小设为16）:batch_size = 16 dataset = tf.data.Dataset.from_tensor_slices((video_features, classes)) dataset = dataset.shuffle(len(videos)) test_count = int(len(videos) * 0.2) train_count = len(videos) - test_count dataset_train = dataset.skip(test_count).cache().repeat() dataset_test = dataset.take(test_count).cache().repeat() train_dataset = dataset_train.shuffle(train_count).batch(batch_size) test_dataset = dataset_test.shuffle(test_count).batch(batch_size) train_dataset, train_count, test_dataset, test_count(<BatchDataset shapes: ((None, 40, 1280), (None,)), types: (tf.float32, tf.int64)>, 1220, <BatchDataset shapes: ((None, 40, 1280), (None,)), types: (tf.float32, tf.int64)>, 305) 之后创建LSTM提取视频特征序列的时间信息送入Dense分类器，模型的定义如下：def video_cls_model(class_vocab): # 类别数量 classes_num = len(class_vocab) # 定义模型 model = keras.Sequential([ layers.Input(shape=(MAX_SEQUENCE_LENGTH, NUM_FEATURES)), layers.LSTM(64, return_sequences=True), layers.Flatten(), layers.Dense(classes_num, activation='softmax') ]) # 编译模型 model.compile(optimizer = keras.optimizers.Adam(1e-5), loss = keras.losses.SparseCategoricalCrossentropy(from_logits=False), metrics = ['accuracy'] ) return model # 模型实例化 model = video_cls_model(np.unique(labels)) # 保存检查点 checkpoint = keras.callbacks.ModelCheckpoint(filepath='best.h5', monitor='val_loss', save_weights_only=True, save_best_only=True, verbose=1, mode='min') # 模型结构 model.summary() 网络的输入大小为(N, 40, 1280)，使用softmax进行激活，输出5个类别的概率：Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= lstm (LSTM) (None, 40, 64) 344320 _________________________________________________________________ flatten (Flatten) (None, 2560) 0 _________________________________________________________________ dense (Dense) (None, 5) 12805 ================================================================= Total params: 357,125 Trainable params: 357,125 Non-trainable params: 0 _________________________________________________________________实验表明模型训练300个Epoch基本收敛：history = model.fit(train_dataset, epochs = 300, steps_per_epoch = train_count // batch_size, validation_steps = test_count // batch_size, validation_data = test_dataset, callbacks=[checkpoint]) plt.plot(history.epoch, history.history['loss'], 'r', label='loss') plt.plot(history.epoch, history.history['val_loss'], 'g--', label='val_loss') plt.title('LSTM') plt.xlabel('Epoch') plt.ylabel('Loss') plt.legend() plt.plot(history.epoch, history.history['accuracy'], 'r', label='acc') plt.plot(history.epoch, history.history['val_accuracy'], 'g--', label='val_acc') plt.title('LSTM') plt.xlabel('Epoch') plt.ylabel('Accuracy') plt.legend() 加载模型最优权重，模型在测试集上的分类准确率为95.8%，保存为saved_model格式：model.load_weights('best.h5') model.evaluate(dataset.batch(batch_size)) model.save('saved_model') 96/96 [==============================] - 0s 5ms/step - loss: 0.2169 - accuracy: 0.9580 [0.21687692414949802, 0.9580328] 二、模型转换首先将图像特征提取器feature_extractor转为tflite格式，并开启模型量化：import tensorflow as tf converter = tf.lite.TFLiteConverter.from_saved_model('feature_extractor') converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS] converter.post_training_quantize = True # 模型量化 tflite_model = converter.convert() with open('mbv2.tflite', 'wb') as f: f.write(tflite_model) 再将视频序列分类模型转为onnx格式，由于lstm参数量较少，不需要进行量化：python -m tf2onnx.convert --saved-model saved_model --output lstm.onnx --opset 12 最后导出RKNN格式的模型，可根据需要设置target_platform为rk3568/rk3588：from rknn.api import RKNN rknn = RKNN(verbose=False) rknn.config(target_platform="rk3588") rknn.load_tflite(model="mbv2.tflite") rknn.build(do_quantization=False) rknn.export_rknn('mbv2.rknn') rknn.release() rknn = RKNN(verbose=False) rknn.config(target_platform="rk3588") rknn.load_onnx( model="lstm.onnx", inputs=['input_3'], # 输入节点名称 input_size_list=[[1, 40, 1280]] # 固定输入尺寸 ) rknn.build(do_quantization=False) rknn.export_rknn('lstm.rknn') rknn.release() 三、模型部署我们在RK3588上部署MobileNetV2和LSTM模型，以下是板侧的推理代码：import os import cv2 import glob import shutil import imageio import numpy as np from IPython.display import Image from rknnlite.api import RKNNLite MAX_SEQUENCE_LENGTH = 40 IMG_SIZE = 256 NUM_FEATURES = 1280 def crop_driving_square(img): h, w = img.shape[:2] start_x = w // 2 end_x = w start_y = 0 end_y = h result = img[start_y:end_y, start_x:end_x] return result def load_video(file_name): cap = cv2.VideoCapture(file_name) # 每隔多少帧抽取一次 frame_interval = 6 frames = [] count = 0 while True: ret, frame = cap.read() if not ret: break # 每隔frame_interval帧保存一次 if count % frame_interval == 0: # 中心裁剪 frame = crop_driving_square(frame) # 缩放 frame = cv2.resize(frame, (IMG_SIZE, IMG_SIZE)) # BGR -> RGB [0,1,2] -> [2,1,0] frame = frame[:, :, [2, 1, 0]] frames.append(frame) count += 1 cap.release() return np.array(frames).astype(np.uint8) # 获取视频特征序列 def getVideoFeat(frames): frames_count = len(frames) # 如果帧数小于MAX_SEQUENCE_LENGTH if frames_count < MAX_SEQUENCE_LENGTH: # 补白 diff = MAX_SEQUENCE_LENGTH - frames_count # 创建全0的numpy数组 padding = np.zeros((diff, IMG_SIZE, IMG_SIZE, 3)) # 数组拼接 frames = np.concatenate((frames, padding)) # 取前MAX_SEQ_LENGTH帧 frames = frames[:MAX_SEQUENCE_LENGTH,:] frames = frames.astype(np.float32) # 提取视频每一帧特征 feats = [] for frame in frames: frame = np.expand_dims(frame, axis=0) result = rknn_lite_mbv2.inference(inputs=[frame]) feats.append(result[0]) return feats rknn_lite_mbv2 = RKNNLite() rknn_lite_lstm = RKNNLite() rknn_lite_mbv2.load_rknn('model/mbv2.rknn') rknn_lite_lstm.load_rknn('model/lstm.rknn') rknn_lite_mbv2.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) rknn_lite_lstm.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2) files = glob.glob("video/*.mp4") for video_path in files: label_to_name = {0:'正常驾驶', 1:'闭眼', 2:'打哈欠', 3:'打电话', 4:'左顾右盼'} frames = load_video(video_path) frames = frames[:MAX_SEQUENCE_LENGTH] imageio.mimsave('test.gif', frames, durations=10, loop=0) display(Image(open('test.gif', 'rb').read())) feats = getVideoFeat(frames) feats = np.concatenate(feats, axis=0) feats = np.expand_dims(feats, axis=0) preds = rknn_lite_lstm.inference(inputs=[feats])[0][0] for i in np.argsort(preds)[::-1][:5]: print('{}: {}%'.format(label_to_name[i], round(preds[i]*100, 2))) rknn_lite_mbv2.release() rknn_lite_lstm.release() 最终的视频识别效果如下：🚀四、本文小结本文详细阐述了基于RK3588平台的CNN-LSTM驾驶行为识别模型全流程，利用MobileNetV2提取图像的空间特征、LSTM处理视频的时序特征完成对正常驾驶、闭眼、打哈欠、打电话和左顾右盼5类驾驶行为的精准识别，在ModelArts上训练达到95.8%分类准确率，并分别将mbv2.tflite和lstm.onnx转换为RKNN格式实现板侧的高效推理部署。

在CUDA编程中，一个CUDA Kernel是由众多线程（threds）组成，而这些线程又可以被组织成一个或多个block块。在同一线程块中，线程ID是从0开始连续编号的，可以通过内置变量threadIdx来获取：// 获取本线程的索引，blockIdx 指的是线程块的索引，blockDim 指的是线程块的大小，threadIdx 指的是本线程块中的线程索引 int tid = blockIdx.x * blockDim.x + threadIdx.x; 以对图像的归一化处理为例，需要对图片中的每一个像素点的三个通道值分别除以255，相比于使用CPU进行串行计算，我们可以使用CUDA核函数创建更多的线程和线程块来充分利用GPU的并行处理能力：// 计算需要的线程总量（高度 x 宽度）:640*640=409600 int jobs = dst_height * dst_width; // 一个线程块包含256个线程 int threads = 256; // 计算线程块的数量（向上取整） int blocks = ceil(jobs / (float)threads); // 调用kernel函数 preprocess_kernel<<<blocks, threads>>>( img_buffer_device, dst, dst_width, dst_height, jobs); // 函数的参数 这里我们定义每个线程块的线程数量为256，线程块的数量为ceil(jobs / (float)threads)，总的线程总量要大于等图片的像素数量。当启动Kernel函数时，GPU上的每个线程都会执行相同的程序代码，从而实现更高效的并行计算，函数具体实现如下：// 一个线程处理一个像素点 __global__ void preprocess_kernel( uint8_t *src, float *dst, int dst_width, int dst_height, int edge) { int tid = blockDim.x * blockIdx.x + threadIdx.x; if (tid >= edge) return; int dx = tid % dst_width; // 计算当前线程对应的目标图像的x坐标 int dy = tid / dst_width; // 计算当前线程对应的目标图像的y坐标 // normalization（对原图中(x,y)坐标的像素点3个通道进行归一化） float c0 = src[dy * dst_width * 3 + dx * 3 + 0] / 255.0f; float c1 = src[dy * dst_width * 3 + dx * 3 + 1] / 255.0f; float c2 = src[dy * dst_width * 3 + dx * 3 + 2] / 255.0f; // bgr to rgb float t = c2; c2 = c0; c0 = t; // rgbrgbrgb to rrrgggbbb // NHWC to NCHW int area = dst_width * dst_height; float *pdst_c0 = dst + dy * dst_width + dx; float *pdst_c1 = pdst_c0 + area; float *pdst_c2 = pdst_c1 + area; *pdst_c0 = c0; *pdst_c1 = c1; *pdst_c2 = c2; } 其中tid是本线程的索引，dst_width和dst_height是图像的宽和高，edge是图片的像素数量，每一个线程处理一个像素点。由于线程索引是从0开始计数的，我们要确保tid不能超过图片的像素数量edge：int tid = blockDim.x * blockIdx.x + threadIdx.x; if (tid >= edge) return; 由于图像数据以行优先（row-major）顺序连续存储在内存中，每个像素由3个字节表示（BGR）。为了获取每个线程所处理的像素点在内存中的起始位置，我们可以先计算当前线程所对应图像的x和y坐标即dx和dy：int dx = tid % dst_width; // 计算当前线程对应的目标图像的x坐标 int dy = tid / dst_width; // 计算当前线程对应的目标图像的y坐标 然后获取当前线程所处理的像素点在内存中的起始位置：dy * dst_width * 3 + dx * 3，*3是因为每个像素点有3个通道值，在内存中的排列方式为:BGRBGRBGR...，最后再/255对原图中(x,y)坐标的像素点3个通道值进行归一化：// normalization float c0 = src[dy * dst_width * 3 + dx * 3 + 0] / 255.0f; float c1 = src[dy * dst_width * 3 + dx * 3 + 1] / 255.0f; float c2 = src[dy * dst_width * 3 + dx * 3 + 2] / 255.0f; dy * dst_width * 3：定位到第dy行的起始位置dx * 3：在当前行中定位到第dx个像素的起始位置+ 0, + 1, + 2：分别访问B、G、R三个通道的值除以255交换变量做BGR到RGB的通道转换：// bgr to rgb float t = c2; c2 = c0; c0 = t; 目标图像（RGB）像素点在内存中的排列方式为RRR...GGG...BBB，当前像素点R通道的值在目标图像中内存地址为(dst + dy * dst_width + dx)，G通道的值在目标图像中内存地址为(dst + dy * dst_width + dx) + area，加上1个通道的偏移量area，以此类推，完成对图像的通道转换：// NHWC to NCHW // rgbrgbrgb to rrrgggbbb int area = dst_width * dst_height; float *pdst_c0 = dst + dy * dst_width + dx; float *pdst_c1 = pdst_c0 + area; float *pdst_c2 = pdst_c1 + area; *pdst_c0 = c0; *pdst_c1 = c1; *pdst_c2 = c2;

Ascend310部署Qwen-VL-7B实现吸烟动作识别OrangePi AI Studio Pro是基于2个昇腾310P处理器的新一代高性能推理解析卡，提供基础通用算力+超强AI算力，整合了训练和推理的全部底层软件栈，实现训推一体。其中AI半精度FP16算力约为176TFLOPS，整数Int8精度可达352TOPS，本文将带领大家在Ascend 310P上部署Qwen2.5-VL-7B多模态理解大模型实现吸烟动作的识别。一、环境配置我们在OrangePi AI Stuido上使用Docker容器部署MindIE：docker pull swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.1.RC1-300I-Duo-py311-openeuler24.03-ltsroot@orangepi:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie 2.1.RC1-300I-Duo-py311-openeuler24.03-lts 0574b8d4403f 3 months ago 20.4GB langgenius/dify-web 1.0.1 b2b7363571c2 8 months ago 475MB langgenius/dify-api 1.0.1 3dd892f50a2d 8 months ago 2.14GB langgenius/dify-plugin-daemon 0.0.4-local 3f180f39bfbe 8 months ago 1.35GB ubuntu/squid latest dae40da440fe 8 months ago 243MB postgres 15-alpine afbf3abf6aeb 8 months ago 273MB nginx latest b52e0b094bc0 9 months ago 192MB swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie 1.0.0-300I-Duo-py311-openeuler24.03-lts 74a5b9615370 10 months ago 17.5GB redis 6-alpine 6dd588768b9b 10 months ago 30.2MB langgenius/dify-sandbox 0.2.10 4328059557e8 13 months ago 567MB semitechnologies/weaviate 1.19.0 8ec9f084ab23 2 years ago 52.5MB之后创建一个名为start-docker.sh的启动脚本，内容如下：NAME=$1 if [ $# -ne 1 ]; then echo "warning: need input container name.Use default: mindie" NAME=mindie fi docker run --name ${NAME} -it -d --net=host --shm-size=500g \ --privileged=true \ -w /usr/local/Ascend/atb-models \ --device=/dev/davinci_manager \ --device=/dev/hisi_hdc \ --device=/dev/devmm_svm \ --entrypoint=bash \ -v /models:/models \ -v /data:/data \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/sbin:/usr/local/sbin \ -v /home:/home \ -v /tmp:/tmp \ -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime \ -e http_proxy=$http_proxy \ -e https_proxy=$https_proxy \ -e "PATH=/usr/local/python3.11.6/bin:$PATH" \ swr.cn-south-1.myhuaweicloud.com/ascendhub/mindie:2.1.RC1-300I-Duo-py311-openeuler24.03-ltsbash start-docker.sh启动容器后，我们需要替换几个文件并安装Ascend-cann-nnal软件包：root@orangepi:~# docker exec -it mindie bash Welcome to 5.15.0-126-generic System information as of time: Sat Nov 15 22:06:48 CST 2025 System load: 1.87 Memory used: 6.3% Swap used: 0.0% Usage On: 33% Users online: 0 [root@orangepi atb-models]# cd /usr/local/Ascend/ascend-toolkit/8.2.RC1/lib64/ [root@orangepi lib64]# ls /data/fix_openeuler_docker/fixhccl/8.2hccl/ libhccl.so libhccl_alg.so libhccl_heterog.so libhccl_plf.so [root@orangepi lib64]# cp /data/fix_openeuler_docker/fixhccl/8.2hccl/* ./ cp: overwrite './libhccl.so'? cp: overwrite './libhccl_alg.so'? cp: overwrite './libhccl_heterog.so'? cp: overwrite './libhccl_plf.so'? [root@orangepi lib64]# source /usr/local/Ascend/ascend-toolkit/set_env.sh [root@orangepi lib64]# chmod +x /data/fix_openeuler_docker/Ascend-cann-nnal/Ascend-cann-nnal_8.3.RC1_linux-x86_64.run [root@orangepi lib64]# /data/fix_openeuler_docker/Ascend-cann-nnal/Ascend-cann-nnal_8.3.RC1_linux-x86_64.run --install --quiet [NNAL] [20251115-22:41:45] [INFO] LogFile:/var/log/ascend_seclog/ascend_nnal_install.log [NNAL] [20251115-22:41:45] [INFO] Ascend-cann-atb_8.3.RC1_linux-x86_64.run --install --install-path=/usr/local/Ascend/nnal --install-for-all --quiet --nox11 start WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [NNAL] [20251115-22:41:58] [INFO] Ascend-cann-atb_8.3.RC1_linux-x86_64.run --install --install-path=/usr/local/Ascend/nnal --install-for-all --quiet --nox11 install success [NNAL] [20251115-22:41:58] [INFO] Ascend-cann-SIP_8.3.RC1_linux-x86_64.run --install --install-path=/usr/local/Ascend/nnal --install-for-all --quiet --nox11 start [NNAL] [20251115-22:41:59] [INFO] Ascend-cann-SIP_8.3.RC1_linux-x86_64.run --install --install-path=/usr/local/Ascend/nnal --install-for-all --quiet --nox11 install success [NNAL] [20251115-22:41:59] [INFO] Ascend-cann-nnal_8.3.RC1_linux-x86_64.run install success Warning!!! If the environment variables of atb and asdsip are set at the same time, unexpected consequences will occur. Import the corresponding environment variables based on the usage scenarios: atb for large model scenarios, asdsip for embedded scenarios. Please make sure that the environment variables have been configured. If you want to use atb module: - To take effect for current user, you can exec command below: source /usr/local/Ascend/nnal/atb/set_env.sh or add "source /usr/local/Ascend/nnal/atb/set_env.sh" to ~/.bashrc. If you want to use asdsip module: - To take effect for current user, you can exec command below: source /usr/local/Ascend/nnal/asdsip/set_env.sh or add "source /usr/local/Ascend/nnal/asdsip/set_env.sh" to ~/.bashrc. [root@orangepi lib64]# cat /usr/local/Ascend/nnal/atb/latest/version.info Ascend-cann-atb : 8.3.RC1 Ascend-cann-atb Version : 8.3.RC1.B106 Platform : x86_64 branch : 8.3.rc1-0702 commit id : 16004f23040e0dcdd3cf0c64ecf36622487038ba修改推理使用的逻辑NPU核心为0,1，测试多模态理解大模型：Qwen2.5-VL-7B-Instruct：运行结果表明，Qwen2.5-VL-7B-Instruct在2 x Ascned 310P上推理平均每秒可以输出20个tokens，同时准确理解画面中的人物信息和行为动作。[root@orangepi atb-models]# bash examples/models/qwen2_vl/run_pa.sh --model_path /models/Qwen2.5-VL-7B-Instruct/ --input_image /root/pic/test.jpg [2025-11-15 22:12:49,663] torch.distributed.run: [WARNING] [2025-11-15 22:12:49,663] torch.distributed.run: [WARNING] ***************************************** [2025-11-15 22:12:49,663] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. [2025-11-15 22:12:49,663] torch.distributed.run: [WARNING] ***************************************** /usr/local/lib64/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source? warn( /usr/local/lib64/python3.11/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: 'libc10_cuda.so: cannot open shared object file: No such file or directory'If you don't plan on using image functionality from `torchvision.io`, you can ignore this warning. Otherwise, there might be something wrong with your environment. Did you have `libjpeg` or `libpng` installed before building `torchvision` from source? warn( 2025-11-15 22:12:53.250 7934 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu] [processid] [threadid] [llmmodels] [loglevel] [file:line] [status code] msg 2025-11-15 22:12:53.250 7933 LLM log default format: [yyyy-mm-dd hh:mm:ss.uuuuuu] [processid] [threadid] [llmmodels] [loglevel] [file:line] [status code] msg [2025-11-15 22:12:53.250] [7934] [139886327420160] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter. [2025-11-15 22:12:53.250] [7933] [139649439929600] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter. [2025-11-15 22:12:53.250] [7934] [139886327420160] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter. [2025-11-15 22:12:53.250] [7933] [139649439929600] [llmmodels] [WARN] [model_factory.cpp:28] deepseekV2_DecoderModel model already exists, but the duplication doesn't matter. [2025-11-15 22:12:53.250] [7934] [139886327420160] [llmmodels] [WARN] [model_factory.cpp:28] llama_LlamaDecoderModel model already exists, but the duplication doesn't matter. [2025-11-15 22:12:53.250] [7933] [139649439929600] [llmmodels] [WARN] [model_factory.cpp:28] llama_LlamaDecoderModel model already exists, but the duplication doesn't matter. [2025-11-15 22:12:55,335] [7934] [139886327420160] [llmmodels] [INFO] [cpu_binding.py-254] : rank_id: 1, device_id: 1, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [2025-11-15 22:12:55,336] [7934] [139886327420160] [llmmodels] [INFO] [cpu_binding.py-280] : process 7934, new_affinity is [8, 9, 10, 11, 12, 13, 14, 15], cpu count 8 [2025-11-15 22:12:55,356] [7933] [139649439929600] [llmmodels] [INFO] [cpu_binding.py-254] : rank_id: 0, device_id: 0, numa_id: 0, shard_devices: [0, 1], cpus: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15] [2025-11-15 22:12:55,357] [7933] [139649439929600] [llmmodels] [INFO] [cpu_binding.py-280] : process 7933, new_affinity is [0, 1, 2, 3, 4, 5, 6, 7], cpu count 8 [2025-11-15 22:12:56,032] [7933] [139649439929600] [llmmodels] [INFO] [model_runner.py-156] : model_runner.quantize: None, model_runner.kv_quant_type: None, model_runner.fa_quant_type: None, model_runner.dtype: torch.float16 [2025-11-15 22:13:01,826] [7933] [139649439929600] [llmmodels] [INFO] [dist.py-81] : initialize_distributed has been Set [2025-11-15 22:13:01,827] [7933] [139649439929600] [llmmodels] [INFO] [model_runner.py-187] : init tokenizer done Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`. [2025-11-15 22:13:02,070] [7934] [139886327420160] [llmmodels] [INFO] [dist.py-81] : initialize_distributed has been Set Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.48, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`. [W InferFormat.cpp:62] Warning: Cannot create tensor with NZ format while dim < 2, tensor will be created with ND format. (function operator()) [W InferFormat.cpp:62] Warning: Cannot create tensor with NZ format while dim < 2, tensor will be created with ND format. (function operator()) [2025-11-15 22:13:08,435] [7933] [139649439929600] [llmmodels] [INFO] [flash_causal_qwen2.py-153] : >>>> qwen_QwenDecoderModel is called. [2025-11-15 22:13:08,526] [7934] [139886327420160] [llmmodels] [INFO] [flash_causal_qwen2.py-153] : >>>> qwen_QwenDecoderModel is called. [2025-11-15 22:13:16.666] [7933] [139649439929600] [llmmodels] [WARN] [operation_factory.cpp:42] OperationName: TransdataOperation not find in operation factory map [2025-11-15 22:13:16.698] [7934] [139886327420160] [llmmodels] [WARN] [operation_factory.cpp:42] OperationName: TransdataOperation not find in operation factory map [2025-11-15 22:13:22,379] [7933] [139649439929600] [llmmodels] [INFO] [model_runner.py-282] : model: FlashQwen2vlForCausalLM( (rotary_embedding): PositionRotaryEmbedding() (attn_mask): AttentionMask() (vision_tower): Qwen25VisionTransformerPretrainedModelATB( (encoder): Qwen25VLVisionEncoderATB( (layers): ModuleList( (0-31): 32 x Qwen25VLVisionLayerATB( (attn): VisionAttention( (qkv): TensorParallelColumnLinear( (linear): FastLinear() ) (proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (mlp): VisionMlp( (gate_up_proj): TensorParallelColumnLinear( (linear): FastLinear() ) (down_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (norm1): BaseRMSNorm() (norm2): BaseRMSNorm() ) ) (patch_embed): FastPatchEmbed( (proj): TensorReplicatedLinear( (linear): FastLinear() ) ) (patch_merger): PatchMerger( (patch_merger_mlp_0): TensorParallelColumnLinear( (linear): FastLinear() ) (patch_merger_mlp_2): TensorParallelRowLinear( (linear): FastLinear() ) (patch_merger_ln_q): BaseRMSNorm() ) ) (rotary_pos_emb): VisionRotaryEmbedding() ) (language_model): FlashQwen2UsingMROPEForCausalLM( (rotary_embedding): PositionRotaryEmbedding() (attn_mask): AttentionMask() (transformer): FlashQwenModel( (wte): TensorEmbeddingWithoutChecking() (h): ModuleList( (0-27): 28 x FlashQwenLayer( (attn): FlashQwenAttention( (rotary_emb): PositionRotaryEmbedding() (c_attn): TensorParallelColumnLinear( (linear): FastLinear() ) (c_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (mlp): QwenMLP( (act): SiLU() (w2_w1): TensorParallelColumnLinear( (linear): FastLinear() ) (c_proj): TensorParallelRowLinear( (linear): FastLinear() ) ) (ln_1): QwenRMSNorm() (ln_2): QwenRMSNorm() ) ) (ln_f): QwenRMSNorm() ) (lm_head): TensorParallelHead( (linear): FastLinear() ) ) ) [2025-11-15 22:13:24,268] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-134] : hbm_capacity(GB): 87.5078125, init_memory(GB): 11.376015624962747 [2025-11-15 22:13:24,789] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-342] : pa_runner: PARunner(model_path=/models/Qwen2.5-VL-7B-Instruct/, input_text=请用超过500个字详细说明图片的内容，并仔细判断画面中的人物是否有吸烟动作。, max_position_embeddings=None, max_input_length=16384, max_output_length=1024, max_prefill_tokens=-1, load_tokenizer=True, enable_atb_torch=False, max_prefill_batch_size=None, max_batch_size=1, dtype=torch.float16, block_size=128, model_config=ModelConfig(num_heads=14, num_kv_heads=2, num_kv_heads_origin=4, head_size=128, k_head_size=128, v_head_size=128, num_layers=28, device=npu:0, dtype=torch.float16, soc_info=NPUSocInfo(soc_name='', soc_version=200, need_nz=True, matmul_nd_nz=False), kv_quant_type=None, fa_quant_type=None, mapping=Mapping(world_size=2, rank=0, num_nodes=1,pp_rank=0, pp_groups=[[0], [1]], micro_batch_size=1, attn_dp_groups=[[0], [1]], attn_tp_groups=[[0, 1]], attn_inner_sp_groups=[[0], [1]], attn_cp_groups=[[0], [1]], attn_o_proj_tp_groups=[[0], [1]], mlp_tp_groups=[[0, 1]], moe_ep_groups=[[0], [1]], moe_tp_groups=[[0, 1]]), cla_share_factor=1, model_type=qwen2_5_vl, enable_nz=False), max_memory=93960798208, [2025-11-15 22:13:24,794] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-122] : ---------------Begin warm_up--------------- [2025-11-15 22:13:24,794] [7933] [139649439929600] [llmmodels] [INFO] [cache.py-154] : kv cache will allocate 0.46484375GB memory [2025-11-15 22:13:24,821] [7934] [139886327420160] [llmmodels] [INFO] [cache.py-154] : kv cache will allocate 0.46484375GB memory [2025-11-15 22:13:24,827] [7933] [139649439929600] [llmmodels] [INFO] [generate.py-1139] : ------total req num: 1, infer start-------- [2025-11-15 22:13:26,002] [7934] [139886327420160] [llmmodels] [INFO] [flash_causal_qwen2.py-680] : <<<<<<<after transdata k_caches[0].shape=torch.Size([136, 16, 128, 16]) [2025-11-15 22:13:26,023] [7933] [139649439929600] [llmmodels] [INFO] [flash_causal_qwen2.py-676] : <<<<<<< ori k_caches[0].shape=torch.Size([136, 16, 128, 16]) [2025-11-15 22:13:26,023] [7933] [139649439929600] [llmmodels] [INFO] [flash_causal_qwen2.py-680] : <<<<<<<after transdata k_caches[0].shape=torch.Size([136, 16, 128, 16]) [2025-11-15 22:13:26,024] [7933] [139649439929600] [llmmodels] [INFO] [flash_causal_qwen2.py-705] : >>>>>>id of kcache is 139645634198608 id of vcache is 139645634198320 [2025-11-15 22:13:34,363] [7933] [139649439929600] [llmmodels] [INFO] [generate.py-1294] : Prefill time: 9476.590633392334ms, Prefill average time: 9476.590633392334ms, Decode token time: 54.94809150695801ms, E2E time: 9531.538724899292ms [2025-11-15 22:13:34,363] [7934] [139886327420160] [llmmodels] [INFO] [generate.py-1294] : Prefill time: 9452.020645141602ms, Prefill average time: 9452.020645141602ms, Decode token time: 54.654598236083984ms, E2E time: 9506.675243377686ms [2025-11-15 22:13:34,366] [7933] [139649439929600] [llmmodels] [INFO] [generate.py-1326] : -------------------performance dumped------------------------ [2025-11-15 22:13:34,371] [7933] [139649439929600] [llmmodels] [INFO] [generate.py-1329] : | batch_size | input_seq_len | output_seq_len | e2e_time(ms) | prefill_time(ms) | decoder_token_time(ms) | prefill_count | prefill_average_time(ms) | |-------------:|----------------:|-----------------:|---------------:|-------------------:|-------------------------:|----------------:|---------------------------:| | 1 | 16384 | 2 | 9531.54 | 9476.59 | 54.95 | 1 | 9476.59 | /usr/local/lib64/python3.11/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True). warnings.warn( [2025-11-15 22:13:35,307] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-148] : warmup_memory(GB): 15.75 [2025-11-15 22:13:35,307] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-153] : ---------------End warm_up--------------- /usr/local/lib64/python3.11/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True). warnings.warn( [2025-11-15 22:13:35,363] [7933] [139649439929600] [llmmodels] [INFO] [generate.py-1139] : ------total req num: 1, infer start-------- [2025-11-15 22:13:50,021] [7933] [139649439929600] [llmmodels] [INFO] [generate.py-1294] : Prefill time: 1004.0028095245361ms, Prefill average time: 1004.0028095245361ms, Decode token time: 13.301290491575836ms, E2E time: 14611.222982406616ms [2025-11-15 22:13:50,021] [7934] [139886327420160] [llmmodels] [INFO] [generate.py-1294] : Prefill time: 1067.9974555969238ms, Prefill average time: 1067.9974555969238ms, Decode token time: 13.300292536193908ms, E2E time: 14674.196720123291ms [2025-11-15 22:13:50,025] [7933] [139649439929600] [llmmodels] [INFO] [generate.py-1326] : -------------------performance dumped------------------------ [2025-11-15 22:13:50,028] [7933] [139649439929600] [llmmodels] [INFO] [generate.py-1329] : | batch_size | input_seq_len | output_seq_len | e2e_time(ms) | prefill_time(ms) | decoder_token_time(ms) | prefill_count | prefill_average_time(ms) | |-------------:|----------------:|-----------------:|---------------:|-------------------:|-------------------------:|----------------:|---------------------------:| | 1 | 1675 | 1024 | 14611.2 | 1004 | 13.3 | 1 | 1004 | [2025-11-15 22:13:50,035] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-385] : Question[0]: [{'image': '/root/pic/test.jpg'}, {'text': '请用超过500个字详细说明图片的内容，并仔细判断画面中的人物是否有吸烟动作。'}] [2025-11-15 22:13:50,035] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-386] : Answer[0]: 这张图片展示了一个无人机航拍的场景，画面中可以看到两名工人站在一个雪地或冰面上。他们穿着橙色的安全背心和红色的安全帽，显得非常醒目。背景中可以看到一些雪地和一些金属结构，可能是桥梁或工业设施的一部分。 从图片的细节来看，画面右侧的工人右手放在嘴边，似乎在吸烟。他的姿势和动作与吸烟者的典型姿势相符。然而，由于图片的分辨率和角度限制，无法完全确定这个动作是否真实发生。如果要准确判断，可能需要更多的视频片段或更清晰的图像。 从无人机航拍的角度来看，这个场景可能是在进行某种工业或建筑项目的检查或监控。两名工人可能正在进行现场检查或讨论工作事宜。雪地和金属结构表明这可能是一个寒冷的冬季，或者是一个寒冷的气候区域。 无人机航拍技术在工业和建筑领域中非常常见，因为它可以提供高空视角，帮助工程师和管理人员更好地了解现场情况。这种技术不仅可以节省时间和成本，还可以提高工作效率和安全性。在进行航拍时，确保遵守当地的法律法规和安全规定是非常重要的。 总的来说，这张图片展示了一个无人机航拍的场景，画面中两名工人站在雪地上，其中一人似乎在吸烟。虽然无法完全确定这个动作是否真实发生，但根据他们的姿势和动作，可以合理推测这个动作的存在。 [2025-11-15 22:13:50,035] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-387] : Generate[0] token num: 282 [2025-11-15 22:13:50,035] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-389] : Latency(s): 14.721353530883789 [2025-11-15 22:13:50,035] [7933] [139649439929600] [llmmodels] [INFO] [run_pa.py-390] : Throughput(tokens/s): 19.15584728050956 本文详细介绍了在OrangePi AI Studio上使用Docker容器部署MindIE环境并运行Qwen2.5-VL-7B-Instruct多模态大模型实现吸烟动作识别的完整过程，验证了在Ascned 310p设备上运行多模态理解大模型的可靠性。

松材线虫病边缘模型训练与推理部署本文详细介绍了松材线虫病检测的边缘模型训练与推理部署全流程。首先，针对无人机拍摄的4032×3024原始图像进行预处理，缩放到1024×1024避免内存溢出，并定义了9个类别（包括麻栎、罩网、疑似、早期、轻度、中度、重度、死亡和逾年）。随后采用20%重叠率对图像进行切分，生成训练集60000张、验证集6495张的sahi数据集。模型训练基于yolo11s.yaml配置，在pwd数据集上进行10个Epoch的训练，虽然实际应用建议至少100个Epoch。评估结果显示，模型在pwd（重度）类别上表现最佳（mAP50达0.707），而pwd_early（早期）类别表现较差。为提升推理效率，将模型导出为TensorRT FP16引擎，GPU推理速度提升高达5倍，单张图片推理耗时约20ms。最后，通过Gradio构建了用户友好的检测应用，实现了松材线虫病的实时检测功能，为林业病害监测提供了有效的技术解决方案，具有较强的实用价值和推广前景。1. 原始数据无人机拍摄原始图像大小是4032 x 3024，这里缩放到1024 x 1024，避免在模型训练时内存溢出：%%writefile pwd.yaml # Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..] path: /home/jetson/ultralytics/dataset/pwd # dataset root dir (absolute path) train: train/images # train images (relative to 'path') val: val/images # val images (relative to 'path') test: # test images (optional) # Classes，类别 names: 0: hardwood # 麻栎 1: net # 罩网 2: abnormal # 疑似 3: pwd_pre_early # 早期 4: pwd_early # 轻度 5: pwd_moderate # 中度 6: pwd # 重度 7: dead_recent # 死亡 8: dead # 逾年 Overwriting pwd.yaml训练集3000张图像，验证集529张图像，查看验证集标注情况：import os import cv2 import yaml import random import numpy as np from matplotlib import pyplot as plt %matplotlib inline with open('pwd.yaml', 'r', encoding='utf-8') as f: data = yaml.load(f.read(), Loader=yaml.FullLoader) classes = data['names'] file_path = os.path.join(data['path'], 'val/images') file_list = os.listdir(file_path) img_paths = random.sample(file_list, 4) img_lists = [] for img_path in img_paths: img_path = os.path.join(file_path, img_path) img = cv2.imread(img_path) h, w, _ = img.shape tl = round(0.002 * (h + w) / 2) + 1 color = (0, 255, 255) if img_path.endswith('.png'): with open(img_path.replace("images", "labels").replace(".png", ".txt")) as f: labels = f.readlines() if img_path.endswith('.jpg'): with open(img_path.replace("images", "labels").replace(".jpg", ".txt")) as f: labels = f.readlines() if img_path.endswith('.jpeg'): with open(img_path.replace("images", "labels").replace(".jpeg", ".txt")) as f: labels = f.readlines() for label in labels: l, x, y, wc, hc = [float(x) for x in label.strip().split()] x1 = int((x - wc / 2) * w) y1 = int((y - hc / 2) * h) x2 = int((x + wc / 2) * w) y2 = int((y + hc / 2) * h) cv2.rectangle(img, (x1, y1), (x2, y2), color, thickness=tl, lineType=cv2.LINE_AA) cv2.putText(img,classes[int(l)],(x1, y1-10), cv2.FONT_HERSHEY_SIMPLEX, 1, color, 2, cv2.LINE_AA) img_lists.append(cv2.resize(img, (1024, 1024))) image = np.concatenate([np.concatenate(img_lists[:2], axis=1), np.concatenate(img_lists[2:], axis=1)], axis=0) cv2.imwrite("sample-pwd.png", image) plt.rcParams["figure.figsize"] = (16, 16) plt.imshow(image[:,:,::-1]) plt.axis('off') plt.show() 2. 切分数据对dataset/pwd数据集进行图像切分，切分大小为1024 x 1024，重叠率是20%，生成新的数据集dataset/pwd-sahi：%%writefile pwd-sahi.yaml # Train/val/test sets as 1) dir: path/to/imgs, 2) file: path/to/imgs.txt, or 3) list: [path/to/imgs1, path/to/imgs2, ..] path: /home/jetson/ultralytics/dataset/pwd-sahi # dataset root dir (absolute path) train: train/images # train images (relative to 'path') val: val/images # val images (relative to 'path') test: # test images (optional) # Classes，类别 names: 0: hardwood # 麻栎 1: net # 罩网 2: abnormal # 疑似 3: pwd_pre_early # 早期 4: pwd_early # 轻度 5: pwd_moderate # 中度 6: pwd # 重度 7: dead_recent # 死亡 8: dead # 逾年 Overwriting pwd-sahi.yaml其中训练集60000张图像（部分为背景图），验证集6495张图像（不含背景图），查看验证集的标注情况：import os import cv2 import yaml import random import numpy as np from matplotlib import pyplot as plt %matplotlib inline with open('pwd-sahi.yaml', 'r', encoding='utf-8') as f: data = yaml.load(f.read(), Loader=yaml.FullLoader) classes = data['names'] file_path = os.path.join(data['path'], 'val/images') file_list = os.listdir(file_path) img_paths = random.sample(file_list, 4) img_lists = [] for img_path in img_paths: img_path = os.path.join(file_path, img_path) img = cv2.imread(img_path) h, w, _ = img.shape tl = round(0.002 * (h + w) / 2) + 1 color = (0, 255, 255) if img_path.endswith('.png'): with open(img_path.replace("images", "labels").replace(".png", ".txt")) as f: labels = f.readlines() if img_path.endswith('.jpg'): with open(img_path.replace("images", "labels").replace(".jpg", ".txt")) as f: labels = f.readlines() if img_path.endswith('.jpeg'): with open(img_path.replace("images", "labels").replace(".jpeg", ".txt")) as f: labels = f.readlines() for label in labels: l, x, y, wc, hc = [float(x) for x in label.strip().split()] x1 = int((x - wc / 2) * w) y1 = int((y - hc / 2) * h) x2 = int((x + wc / 2) * w) y2 = int((y + hc / 2) * h) cv2.rectangle(img, (x1, y1), (x2, y2), color, thickness=tl, lineType=cv2.LINE_AA) cv2.putText(img,classes[int(l)],(x1, y1-10), cv2.FONT_HERSHEY_SIMPLEX, 1, color, 2, cv2.LINE_AA) img_lists.append(cv2.resize(img, (1024, 1024))) image = np.concatenate([np.concatenate(img_lists[:2], axis=1), np.concatenate(img_lists[2:], axis=1)], axis=0) cv2.imwrite("sample-pwd-sahi.png", image) plt.rcParams["figure.figsize"] = (16, 16) plt.imshow(image[:,:,::-1]) plt.axis('off') plt.show() 3. 模型训练我们加载yolo11s.yaml模型的配置文件在dataset/pwd数据集上训练10个Epoch，模型的训练结果保存在pine_wilt_disease/yolo11s_10目录下：%%writefile train.py from ultralytics import YOLO # Load a model model = YOLO('yolo11s.yaml') # load yaml model # Train the model results = model.train(data='pwd.yaml', epochs=10, imgsz=640, workers=4, batch=8, project="pine_wilt_disease", name="yolo11s_10") Overwriting train.py在终端中运行：/home/jetson/ultralytics/train.sh在另一个终端中运行/home/jetson/ultralytics/tensorboard.sh可以监控模型的训练情况：4. 模型评估加载训练好的模型，这里我们仅训练了10个Epoch，实际训练至少100个Epoch才能取得较好的效果：from ultralytics import YOLO # Load a model model = YOLO('pine_wilt_disease/yolo11s_10/weights/best.pt') # load the best model # Evaluate the model metrics = model.val( data='pwd.yaml', # 数据集配置 imgsz=640, # 模型输入大小 workers=4, # 数据加载线程 batch=8, # 验证批次大小 plots=True, # 生成验证结果图 split='val' # 指定使用验证集 ) Ultralytics 8.3.55 🚀 Python-3.10.12 torch-2.5.0a0+872d972e41.nv24.08 CUDA:0 (Orin, 7620MiB) YOLO11s summary (fused): 238 layers, 9,416,283 parameters, 0 gradients, 21.3 GFLOPs val: Scanning /home/jetson/ultralytics/dataset/pwd/val/labels.cache... 529 images, 0 backgrounds, 0 corrupt: 100%|██████████| 529/529 [00:00<?, ?it/s] Class Images Instances Box(P R mAP50 mAP50-95): 100%|██████████| 67/67 [00:22<00:00, 2.92it/s] all 529 8612 0.602 0.445 0.442 0.261 net 383 4210 0.681 0.625 0.683 0.401 pwd_early 167 375 1 0 0.0362 0.0168 pwd_moderate 253 503 0.4 0.225 0.224 0.108 pwd 383 1141 0.582 0.765 0.707 0.462 dead_recent 376 1456 0.503 0.443 0.444 0.26 dead 229 927 0.444 0.613 0.557 0.318 Speed: 0.9ms preprocess, 24.5ms inference, 0.0ms loss, 4.1ms postprocess per image Results saved to runs/detect/val注意，图片实际标注的类别只有6类，不包含麻栎和疑似。5. 模型导出导出到TensorRT，GPU推理速度提升高达5倍：https://docs.ultralytics.com/zh/integrations/tensorrt/from ultralytics import YOLO model = YOLO("pine_wilt_disease/yolo11s_10/weights/best.pt") # TensorRT FP16 model.export(format="engine", imgsz=640, batch=1, half=True) WARNING ⚠️ TensorRT requires GPU export, automatically assigning device=0 Ultralytics 8.3.55 🚀 Python-3.10.12 torch-2.5.0a0+872d972e41.nv24.08 CUDA:0 (Orin, 7620MiB) YOLO11s summary (fused): 238 layers, 9,416,283 parameters, 0 gradients, 21.3 GFLOPs PyTorch: starting from 'pine_wilt_disease/yolo11s_10/weights/best.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 13, 8400) (18.3 MB) ONNX: starting export with onnx 1.17.0 opset 19... ONNX: slimming with onnxslim 0.1.47... ONNX: export success ✅ 3.2s, saved as 'pine_wilt_disease/yolo11s_10/weights/best.onnx' (36.2 MB) TensorRT: starting export with TensorRT 10.7.0... [11/15/2025-16:23:19] [TRT] [I] [MemUsageChange] Init CUDA: CPU -2, GPU +0, now: CPU 1395, GPU 7158 (MiB) [11/15/2025-16:23:25] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +970, GPU +258, now: CPU 2322, GPU 7418 (MiB) [11/15/2025-16:23:26] [TRT] [I] ---------------------------------------------------------------- [11/15/2025-16:23:26] [TRT] [I] Input filename: pine_wilt_disease/yolo11s_10/weights/best.onnx [11/15/2025-16:23:26] [TRT] [I] ONNX IR version: 0.0.9 [11/15/2025-16:23:26] [TRT] [I] Opset version: 19 [11/15/2025-16:23:26] [TRT] [I] Producer name: pytorch [11/15/2025-16:23:26] [TRT] [I] Producer version: 2.5.0 [11/15/2025-16:23:26] [TRT] [I] Domain: [11/15/2025-16:23:26] [TRT] [I] Model version: 0 [11/15/2025-16:23:26] [TRT] [I] Doc string: [11/15/2025-16:23:26] [TRT] [I] ---------------------------------------------------------------- TensorRT: input "images" with shape(1, 3, 640, 640) DataType.FLOAT TensorRT: output "output0" with shape(1, 13, 8400) DataType.FLOAT TensorRT: building FP16 engine as pine_wilt_disease/yolo11s_10/weights/best.engine [11/15/2025-16:23:26] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored. [11/15/2025-16:28:08] [TRT] [I] Compiler backend is used during engine build. [11/15/2025-16:31:48] [TRT] [I] Detected 1 inputs and 1 output network tensors. [11/15/2025-16:31:53] [TRT] [I] Total Host Persistent Memory: 543184 bytes [11/15/2025-16:31:53] [TRT] [I] Total Device Persistent Memory: 0 bytes [11/15/2025-16:31:53] [TRT] [I] Max Scratch Memory: 2764800 bytes [11/15/2025-16:31:53] [TRT] [I] [BlockAssignment] Started assigning block shifts. This will take 162 steps to complete. [11/15/2025-16:31:53] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 19.653ms to assign 10 blocks to 162 nodes requiring 19046912 bytes. [11/15/2025-16:31:53] [TRT] [I] Total Activation Memory: 19046400 bytes [11/15/2025-16:31:53] [TRT] [I] Total Weights Memory: 18914082 bytes [11/15/2025-16:31:53] [TRT] [I] Compiler backend is used during engine execution. [11/15/2025-16:31:53] [TRT] [I] Engine generation completed in 506.948 seconds. [11/15/2025-16:31:53] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 2 MiB, GPU 140 MiB TensorRT: export success ✅ 519.0s, saved as 'pine_wilt_disease/yolo11s_10/weights/best.engine' (21.6 MB) Export complete (519.7s) Results saved to /home/jetson/ultralytics/pine_wilt_disease/yolo11s_10/weights Predict: yolo predict task=detect model=pine_wilt_disease/yolo11s_10/weights/best.engine imgsz=640 half Validate: yolo val task=detect model=pine_wilt_disease/yolo11s_10/weights/best.engine imgsz=640 data=pwd.yaml half Visualize: https://netron.app导出FP16精度的量化模型大概需要10分钟左右。6. 模型推理使用TensorRT引擎加载模型对验证集的部分图片进行推理，每张图片的推理耗时约20ms：import cv2 import glob from ultralytics import YOLO import matplotlib.pyplot as plt %matplotlib inline # Load the TensorRT engine model model = YOLO("pine_wilt_disease/yolo11s_10/weights/best.engine") # Define the prediction function def predict(image_path): reuslts = model.predict(image_path, conf=0.45, iou=0.55) return reuslts[0].plot() # Load the images for inference images_path = glob.glob("dataset/pwd/val/images/*.jpeg") # Perform inference and display results for image_path in images_path[:10]: result = predict(image_path) result = cv2.cvtColor(result, cv2.COLOR_BGR2RGB) result = cv2.resize(result, (4032 // 4, 3024 // 4)) plt.imshow(result) plt.axis("off") plt.show() WARNING ⚠️ Unable to automatically guess model task, assuming 'task=detect'. Explicitly define task for your model, i.e. 'task=detect', 'segment', 'classify','pose' or 'obb'. Loading pine_wilt_disease/yolo11s_10/weights/best.engine for TensorRT inference... [11/15/2025-16:31:54] [TRT] [I] Loaded engine size: 21 MiB [11/15/2025-16:31:54] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +18, now: CPU 0, GPU 36 (MiB) image 1/1 /home/jetson/ultralytics/dataset/pwd/val/images/1d1d160a-ae4f-4fe4-801d-f001d4e7ff6d.jpeg: 640x640 4 nets, 1 pwd, 1 dead_recent, 20.1ms Speed: 45.6ms preprocess, 20.1ms inference, 49.3ms postprocess per image at shape (1, 3, 640, 640) ... 构建Gradio应用程序，上传图片实现松材线虫病检测的功能：至此，本章结束。

Pi0☁云端推理部署Pi0 是一个通用机器人策略基础模型，专为解决机器人学习中的数据稀缺、泛化能力差和鲁棒性不足等核心挑战而设计。借鉴大语言模型的训练方法，Pi0 通过大规模预训练掌握广泛的机器人操作技能，并能通过微调快速适应具体任务需求。该模型支持高效的数据利用与快速部署，在有限数据下也能实现良好性能，为机器人智能化提供了一种可扩展的解决方案。1. 环境配置在之前LeRobot安装及使用教程和Pi0 模型训练的基础上，Pi0的推理也建议使用云端部署的方式运行。a. 我租赁的服务器是H20，在服务端（openpi环境）中运行：cd /root/autodl-tmp/openpi_episode1_student uv run scripts/serve_policy.py policy:checkpoint --policy.config=enpei_robot_demo_move_toy_low_mem_finetune --policy.dir=checkpoints/enpei_robot_demo_move_toy_low_mem_finetune/my_experiment/9999启动服务器后，他会暴露一个服务端口（默认是 6006）：warning: The `tool.uv.dev-dependencies` field (used in `packages/openpi-client/pyproject.toml`) is deprecated and will be removed in a future release; use `dependency-groups.dev` instead INFO:root:Loading model... INFO:2025-11-09 14:40:27,523:jax._src.xla_bridge:925: Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig' INFO:jax._src.xla_bridge:Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig' INFO:2025-11-09 14:40:27,524:jax._src.xla_bridge:925: Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory INFO:jax._src.xla_bridge:Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory INFO:absl:orbax-checkpoint version: 0.11.13 INFO:absl:Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7f78fd908250> INFO:absl:Restoring checkpoint from /root/autodl-tmp/openpi_episode1_student/checkpoints/enpei_robot_demo_move_toy_low_mem_finetune/my_experiment/9999/params. INFO:absl:[thread=MainThread] Failed to get flag value for EXPERIMENTAL_ORBAX_USE_DISTRIBUTED_PROCESS_ID. INFO:absl:[process=0] /jax/checkpoint/read/bytes_per_sec: 1.1 GiB/s (total bytes: 6.1 GiB) (time elapsed: 5 seconds) (per-host) INFO:absl:Finished restoring checkpoint in 5.72 seconds from /root/autodl-tmp/openpi_episode1_student/checkpoints/enpei_robot_demo_move_toy_low_mem_finetune/my_experiment/9999/params. INFO:root:Loaded norm stats from /root/autodl-tmp/openpi_episode1_student/assets/enpei_robot_demo_move_toy_low_mem_finetune/hou/demo_move_toy_openpi INFO:root:Loaded norm stats from /root/autodl-tmp/openpi_episode1_student/checkpoints/enpei_robot_demo_move_toy_low_mem_finetune/my_experiment/9999/assets/hou/demo_move_toy_openpi INFO:root:Creating server (host: autodl-container-9af746813b-5ccfe20e, ip: 172.17.0.8) INFO:websockets.server:server listening on 0.0.0.0:6006 INFO:websockets.server:connection open INFO:openpi.serving.websocket_policy_server:Connection from ('127.0.0.1', 53688) opened INFO:openpi.serving.websocket_policy_server:Connection from ('127.0.0.1', 53688) closed INFO:websockets.server:connection open INFO:openpi.serving.websocket_policy_server:Connection from ('127.0.0.1', 56818) opened INFO:openpi.serving.websocket_policy_server:Connection from ('127.0.0.1', 56818) closed在AutoDL中使用它的端口转发服务（自定义服务）：b. 在客户端（lerobot环境）进入Lerobot仓库代码，中安装openpi-client：cd ./packages/openpi-client pip install -e . c. 启动机械臂进行归零，安装腕部相机和夹爪，查看相机ID:(base) hou@hou-Ubuntu:~/workspace/lerobot_single_student$ conda activate lerobot (lerobot) hou@hou-Ubuntu:~/workspace/lerobot_single_student$ python -m lerobot.episode_default_position --ip="localhost" --port=12345 INFO 2025-11-09 14:25:56 _position.py:33 Connected to EnpeiRobot controller at localhost:12345 INFO 2025-11-09 14:26:02 _position.py:46 Moving to default position, estimated time: 1.72s INFO 2025-11-09 14:26:04 _position.py:50 Successfully moved to default position (lerobot) hou@hou-Ubuntu:~/workspace/lerobot_single_student$ python -m lerobot.find_cameras opencv --- Detected Cameras --- Camera #0: Name: OpenCV Camera @ /dev/video0 Type: OpenCV Id: /dev/video0 Backend api: V4L2 Default stream profile: Format: 0.0 Width: 640 Height: 480 Fps: 30.0 -------------------- Camera #1: Name: OpenCV Camera @ /dev/video2 Type: OpenCV Id: /dev/video2 Backend api: V4L2 Default stream profile: Format: 0.0 Width: 640 Height: 480 Fps: 30.0 -------------------- Camera #2: Name: OpenCV Camera @ /dev/video4 Type: OpenCV Id: /dev/video4 Backend api: V4L2 Default stream profile: Format: 0.0 Width: 640 Height: 480 Fps: 30.0 -------------------- Finalizing image saving... Image capture finished. Images saved to outputs/captured_images2. Pi0 推理启动客户端，在终端中运行run.sh：host 服务器地址（这里因为用了 AutoDL 本地转发，所以是 localhost）port 服务器端口instruction 文本指令，保持和采集数据一致enpei_use_radian 需要使用弧度制#!/bin/bash python -m lerobot.test_openpi \ --robot.ip_address="localhost" \ --robot.port=12345 \ --robot.type=enpei_follower \ --robot.id=enpei_follower \ --robot.cameras="{ handeye: {type: opencv, index_or_path: 4, width: 320, height: 240, fps: 30}, fixed: {type: opencv, index_or_path: 0, width: 320, height: 240, fps: 30}}" \ --host=localhost \ --port=6006 \ --instruction="Put the toy to the white box" \ --fps=30 \ --enpei_use_radian=true (lerobot) hou@hou-Ubuntu:~/workspace/lerobot_single_student$ ./run.sh INFO 2025-11-09 14:48:58 nt_policy.py:30 Waiting for server at ws://localhost:6006... Connected to remote policy server at localhost:6006 Server metadata: {} Using instruction: Put the toy to the white box 设置enpei follower机器人角度单位: 弧度 INFO 2025-11-09 14:48:58 follower.py:134 Connected to EnpeiRobot controller INFO 2025-11-09 14:49:03 a_opencv.py:176 OpenCVCamera(4) connected. INFO 2025-11-09 14:49:05 a_opencv.py:176 OpenCVCamera(0) connected. INFO 2025-11-09 14:49:05 follower.py:156 enpei_follower EnpeiFollower connected. Robot connected successfully 运行策略，目标FPS: 30 推理线程启动 等待第一个动作生成... wait for frame: 0.0018558219999249559 wait for frame: 0.0018982179999511573 Loop duration: 1.4765s, Real FPS: 0.68, Action FPS: 33.87, Action count: 50, Action shape: (50, 7) Loop duration: 1.4735s, Real FPS: 0.68, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.4755s, Real FPS: 0.68, Action FPS: 33.89, Action count: 50, Action shape: (50, 7) Loop duration: 1.4973s, Real FPS: 0.67, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5040s, Real FPS: 0.66, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5590s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5400s, Real FPS: 0.65, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.4911s, Real FPS: 0.67, Action FPS: 33.91, Action count: 50, Action shape: (50, 7) Loop duration: 1.4938s, Real FPS: 0.67, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.4853s, Real FPS: 0.67, Action FPS: 33.92, Action count: 50, Action shape: (50, 7) Loop duration: 1.4957s, Real FPS: 0.67, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.4930s, Real FPS: 0.67, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.4914s, Real FPS: 0.67, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.4878s, Real FPS: 0.67, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.4911s, Real FPS: 0.67, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.4914s, Real FPS: 0.67, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5189s, Real FPS: 0.66, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.4915s, Real FPS: 0.67, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5207s, Real FPS: 0.66, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.4865s, Real FPS: 0.67, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5499s, Real FPS: 0.65, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.6458s, Real FPS: 0.61, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.6337s, Real FPS: 0.61, Action FPS: 33.98, Action count: 50, Action shape: (50, 7) Loop duration: 1.5200s, Real FPS: 0.66, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5218s, Real FPS: 0.66, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5184s, Real FPS: 0.66, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5273s, Real FPS: 0.65, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5247s, Real FPS: 0.66, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5285s, Real FPS: 0.65, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5576s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5193s, Real FPS: 0.66, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5519s, Real FPS: 0.64, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5240s, Real FPS: 0.66, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.6051s, Real FPS: 0.62, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5251s, Real FPS: 0.66, Action FPS: 33.92, Action count: 50, Action shape: (50, 7) Loop duration: 1.5533s, Real FPS: 0.64, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5564s, Real FPS: 0.64, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5579s, Real FPS: 0.64, Action FPS: 33.92, Action count: 50, Action shape: (50, 7) Loop duration: 1.5550s, Real FPS: 0.64, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5526s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5507s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5511s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5557s, Real FPS: 0.64, Action FPS: 33.92, Action count: 50, Action shape: (50, 7) Loop duration: 1.5514s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5539s, Real FPS: 0.64, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5573s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5767s, Real FPS: 0.63, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5736s, Real FPS: 0.64, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5553s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5537s, Real FPS: 0.64, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5514s, Real FPS: 0.64, Action FPS: 33.98, Action count: 50, Action shape: (50, 7) Loop duration: 1.6261s, Real FPS: 0.61, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.6378s, Real FPS: 0.61, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5520s, Real FPS: 0.64, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5588s, Real FPS: 0.64, Action FPS: 33.98, Action count: 50, Action shape: (50, 7) Loop duration: 1.5567s, Real FPS: 0.64, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5534s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5481s, Real FPS: 0.65, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.6520s, Real FPS: 0.61, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5514s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5552s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5494s, Real FPS: 0.65, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) ^C Client stopped by user 推理线程结束测试视频：3. 小结本文介绍了Pi0云端推理部署的完整流程。首先配置好服务器端环境并启动策略服务，然后在客户端安装所需依赖并连接机械臂与摄像头。实测下来，即使对于没见过的云宝机械臂的抓取放置成功率也能达到100%，体现了模型强大的泛化能力！

Pi0 模型训练Pi0 是一个通用机器人策略基础模型，专为解决机器人学习中的数据稀缺、泛化能力差和鲁棒性不足等核心挑战而设计。借鉴大语言模型的训练方法，Pi0 通过大规模预训练掌握广泛的机器人操作技能，并能通过微调快速适应具体任务需求。该模型支持高效的数据利用与快速部署，在有限数据下也能实现良好性能，为机器人智能化提供了一种可扩展的解决方案。1. 环境安装在LeRobot安装及使用教程的基础上，训练Pi0需要显存较大的显卡，我租赁的实例是 H20，使用VS Code SSH远程连接服务器：# ssh -p 17825 root@region-42.seetacloud.com Host region-42 HostName region-42.seetacloud.com User root Port 17825 在安装Pi0的过程中需要访问Github和HuggingFace等国外网站，建议开启AutoDL内置的学术加速服务，由于该加速服务可能对正常网络造成一定影响，安装过程中如果失败可以多次尝试，当不再需要时建议取消学术加速。# 启用学术加速 source /etc/network_turbo cd /root/autodl-tmp git clone https://github.com/enpeizhao/openpi_episode1_student.git cd openpi_episode1_student/ pip install uv GIT_LFS_SKIP_SMUDGE=1 uv sync GIT_LFS_SKIP_SMUDGE=1 uv pip install -e . # 取消学术加速 unset http_proxy && unset https_proxy2. 转换数据Pi0训练必须是弧度制数据，安装ffmpeg转换为openpi格式：sudo apt update sudo apt install ffmpeg uv run ./examples/libero/lerobot2oppi.py \ --source-repo-id=hou/demo_move_toy \ --target-repo-id=hou/demo_move_toy_openpi \ --output-path=./demo_move_toy_openpi \ --source-dataset-root=/root/autodl-tmp/demo_move_toy \ --max-episodes=100 3. 修改配置以下是我的配置文件，为了节省训练时间num_train_steps设置为10_000：# 单臂配置：openpi_episode1_student/src/openpi/training/config.py TrainConfig( name="enpei_robot_demo_move_toy_low_mem_finetune", # Here is an example of loading a pi0 model for LoRA fine-tuning. model=pi0.Pi0Config(paligemma_variant="gemma_2b_lora", action_expert_variant="gemma_300m_lora"), data=LeRobotLiberoDataConfig( repo_id="hou/demo_move_toy_openpi", # 数据集repo_od root="./demo_move_toy_openpi", # 数据集路径 base_config=DataConfig(prompt_from_task=True), ), weight_loader=weight_loaders.CheckpointWeightLoader("/root/autodl-tmp/pi0_base/params"), num_train_steps=10_000, # The freeze filter defines which parameters should be frozen during training. # We have a convenience function in the model config that returns the default freeze filter # for the given model config for LoRA finetuning. Just make sure it matches the model config # you chose above. freeze_filter=pi0.Pi0Config( paligemma_variant="gemma_2b_lora", action_expert_variant="gemma_300m_lora" ).get_freeze_filter(), # Turn off EMA for LoRA finetuning. ema_decay=None, ), 4. 模型训练# 计算 normalization uv run scripts/compute_norm_stats.py --config-name enpei_robot_demo_move_toy_low_mem_finetune # 开启训练 XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py enpei_robot_demo_move_toy_low_mem_finetune --exp-name=my_experiment --overwrite在训练过程中可以看到模型的LOSS在逐渐降低，也可以在终端中查看GPU的利用率，采集100个episodes数据使用H20迭代10_000步大概需要花费9个小时：训练过程中Pi0会保存多个时间点的权重checkpoint，每次约消耗磁盘12G，因此建议将代码目录放到/root/autodl-tmp数据盘目录下，这样训练过程中产生的权重文件会自动保存在/root/autodl-tmp/checkpoints下，避免系统盘因空间不足导致训练失败。5. 本文小结本文介绍了在H20实例上安装和训练Pi0模型的完整流程，包括环境配置、数据转换、修改训练配置以及模型训练等关键步骤，并提供了相关命令和注意事项，帮助读者快速上手Pi0模型训练。

如何在Python中调用C++版本的ByteTrack跟踪算法这个项目提供了基于ByteTrack-TensorRT的Python插件，并在原有算法基础上提供了跟踪目标的类别信息，Jetson Orin Nano在之前YOLOv5插件的基础上实现高达83 FPS的实时检测跟踪性能。⚡ 极致性能: 基于TensorRT优化，充分利用硬件加速📦 开箱即用：构建过程简单，快速部署您的跟踪应用🐍 Python 友好: 使用Pybind11提供简洁Python接口📱 边缘设备优化: 特别针对Jetson边缘设备进行适配Build plugin首先安装必要的库克隆仓库构建项目，注意JetPack 5.x版本才能正常运行：sudo apt update sudo apt install ffmpeg sudo apt install pybind11-dev sudo apt install libeigen3-dev git cone https://github.com/HouYanSong/bytetrack_pybind11.git cd bytetrack_pybind11 pip install pybind11 rm -fr build cmake -S . -B build cmake --build build[ 12%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/BYTETracker.cpp.o [ 25%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/STrack.cpp.o [ 37%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/kalmanFilter.cpp.o [ 50%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/lapjv.cpp.o [ 62%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/utils.cpp.o [ 75%] Linking CXX shared library libbytetrack.so [ 75%] Built target bytetrack [ 87%] Building CXX object CMakeFiles/bytetrack_trt.dir/bytetrack_trt.cpp.o [100%] Linking CXX shared module bytetrack_trt.cpython-38-aarch64-linux-gnu.so [100%] Built target bytetrack_trtRun demo我们提供了一个简单的Python示例，只需要导入C++构建的Python动态链接库就可以非常方便的调用ByteTrack跟踪算法，返回目标位置、跟踪ID和类别信息。import cv2 import time import ctypes # 加载依赖库 ctypes.CDLL("./yolov5_trt_plugin/libyolo_plugin.so", mode=ctypes.RTLD_GLOBAL) ctypes.CDLL("./yolov5_trt_plugin/libyolo_utils.so", mode=ctypes.RTLD_GLOBAL) ctypes.CDLL("./build/libbytetrack.so", mode=ctypes.RTLD_GLOBAL) # 导入YOLOv5检测器和ByteTrack跟踪器 from yolov5_trt_plugin import yolov5_trt from build import bytetrack_trt def draw_image(image, detections, tracks, fps): for track in tracks: x, y, w, h = track.tlwh track_id = track.track_id class_id = track.label x1, y1, x2, y2 = int(x), int(y), int(x+w), int(y+h) cv2.rectangle(image, (x1, y1), (x2, y2), (0, 255, 0), 2) cv2.putText(image, f"C:{class_id} T:{track_id}", (x1, y1 - 10), cv2.FONT_HERSHEY_PLAIN, 1.2, (0, 0, 255), 2) cv2.putText(image, f"FPS: {fps:.2f}", (10, 30), cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 0, 255), 2) return image def main(input_path, output_path): cap = cv2.VideoCapture(input_path) fps_value = int(cap.get(cv2.CAP_PROP_FPS)) width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) writer = cv2.VideoWriter(output_path, cv2.VideoWriter_fourcc(*'MJPG'), fps_value, (width, height)) detector = yolov5_trt.YOLOv5Detector("./yolov5_trt_plugin/yolov5s.engine", width, height) tracker = bytetrack_trt.BYTETracker(frame_rate = fps_value, track_buffer = 30) fps_list = [] frame_count = 0 total_time = 0.0 while cap.isOpened(): ret, frame = cap.read() if not ret: break start_time = time.time() # 目标检测 detections = detector.detect(input_image=frame, input_w=640, input_h=640, conf_thresh=0.45, nms_thresh=0.55) objects = [] for det in detections: x1, y1, x2, y2 = det['bbox'] rect = bytetrack_trt.RectFloat(x1, y1, x2-x1, y2-y1) # x, y, width, height obj = bytetrack_trt.Object() obj.rect = rect obj.label = det['class_id'] obj.prob = det['confidence'] objects.append(obj) # 目标跟踪 tracks = tracker.update(objects) process_time = time.time() - start_time current_fps = 1.0 / process_time if process_time > 0 else 0 frame_count += 1 total_time += process_time fps_list.append(current_fps) # 图像绘制 image = draw_image(frame, detections, tracks, current_fps) writer.write(image) cap.release() writer.release() if frame_count > 0: avg_fps = frame_count / total_time if total_time > 0 else 0 print(f"Processed {frame_count} frames") print(f"Average FPS: {avg_fps:.2f}") print(f"Min FPS: {min(fps_list):.2f}") print(f"Max FPS: {max(fps_list):.2f}") if __name__ == "__main__": input_video = "./media/sample_720p.mp4" output_video = "./result.avi" main(input_video, output_video) 仅需在终端中运行yolov5_bytetrack.py脚本：python yolov5_bytetrack.py[11/07/2025-17:13:10] [I] [TRT] Loaded engine size: 8 MiB Deserialize yoloLayer plugin: YoloLayer [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +536, GPU +702, now: CPU 841, GPU 3927 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +83, GPU +94, now: CPU 924, GPU 4021 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +7, now: CPU 0, GPU 7 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 924, GPU 4021 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +1, now: CPU 924, GPU 4022 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +11, now: CPU 0, GPU 18 (MiB) Init ByteTrack! Processed 1442 frames Average FPS: 83.78 Min FPS: 68.31 Max FPS: 113.35 Conclusion Remarks本文实现了ByteTrack-TensorRT跟踪算法的Python插件，并在原有算法基础上提供了跟踪目标的类别信息，Jetson Orin Nano (8GB)上的YOLOv5实时目标检测和跟踪速度高达80FPS，满足对快速运动目标的跟踪需求。

如何在Jetson上将YOLOv5实时检测速度提升至120+FPS这个项目提供了基于 Pybind11 的 TensorRT YOLOv5 插件 Python 绑定，实现了令人难以置信的实时目标检测性能！⚡ 超100FPS性能: 在 Jetson Orin Nano 上轻松实现超过 120 帧/秒的检测速度🎯 高精度检测: 基于成熟的 YOLOv5 架构，准确识别COCO数据集上的80类目标🔌 即插即用: 简单的 Python 接口，无需复杂的配置🛠️ 工业级优化: 采用 TensorRT 进行模型优化和加速1. Building the plugin首先安装必要的库克隆仓库构建项目，注意JetPack 5.x版本才能正常运行：sudo apt update sudo apt install ffmpeg sudo apt install pybind11-dev git clone https://github.com/HouYanSong/yolov5_trt_pybind11.git cd yolov5_trt_pybind11 pip install pybind11 rm -fr build cmake -S . -B build cmake --build build2. Model quantization生成量化图片对YOLOv5s模型进行Int8量化，保存量化后的模型：./media/gen_calib.sh ./build/build weights/yolov5s.onnx 1 ./media/ ./media/filelist.txt weights/yolov5s.engine[11/06/2025-11:57:36] [I] [TRT] [MemUsageChange] Init CUDA: CPU +221, GPU +0, now: CPU 249, GPU 4229 (MiB) [11/06/2025-11:57:39] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +302, GPU +277, now: CPU 574, GPU 4529 (MiB) [11/06/2025-11:57:39] [I] [TRT] ---------------------------------------------------------------- [11/06/2025-11:57:39] [I] [TRT] Input filename: weights/yolov5s.onnx [11/06/2025-11:57:39] [I] [TRT] ONNX IR version: 0.0.7 [11/06/2025-11:57:39] [I] [TRT] Opset version: 12 [11/06/2025-11:57:39] [I] [TRT] Producer name: [11/06/2025-11:57:39] [I] [TRT] Producer version: [11/06/2025-11:57:39] [I] [TRT] Domain: [11/06/2025-11:57:39] [I] [TRT] Model version: 0 [11/06/2025-11:57:39] [I] [TRT] Doc string: [11/06/2025-11:57:39] [I] [TRT] ---------------------------------------------------------------- [11/06/2025-11:57:39] [I] [TRT] No importer registered for op: YoloLayer_TRT. Attempting to import as plugin. [11/06/2025-11:57:39] [I] [TRT] Searching for plugin: YoloLayer_TRT, plugin_version: 1, plugin_namespace: [11/06/2025-11:57:39] [I] [TRT] Successfully created plugin: YoloLayer_TRT [11/06/2025-11:57:39] [I] sample0001.png [11/06/2025-11:57:39] [I] sample0002.png [11/06/2025-11:57:39] [I] sample0003.png [11/06/2025-11:57:39] [I] sample0004.png [11/06/2025-11:57:39] [I] sample0005.png [11/06/2025-11:57:39] [I] sample0006.png [11/06/2025-11:57:39] [I] sample0007.png [11/06/2025-11:57:39] [I] sample0008.png [11/06/2025-11:57:39] [I] sample0009.png [11/06/2025-11:57:39] [I] sample0010.png [11/06/2025-11:57:39] [I] sample0011.png [11/06/2025-11:57:39] [I] sample0012.png [11/06/2025-11:57:39] [I] sample0013.png [11/06/2025-11:57:39] [I] sample0014.png [11/06/2025-11:57:39] [I] sample0015.png [11/06/2025-11:57:39] [I] sample0016.png [11/06/2025-11:57:39] [I] sample0017.png [11/06/2025-11:57:39] [I] sample0018.png [11/06/2025-11:57:39] [I] sample0019.png [11/06/2025-11:57:39] [I] sample0020.png [11/06/2025-11:57:39] [I] sample0021.png [11/06/2025-11:57:39] [I] sample0022.png [11/06/2025-11:57:39] [I] sample0023.png [11/06/2025-11:57:39] [I] sample0024.png [11/06/2025-11:57:39] [I] sample0025.png [11/06/2025-11:57:39] [I] sample0026.png [11/06/2025-11:57:39] [I] sample0027.png [11/06/2025-11:57:39] [I] sample0028.png [11/06/2025-11:57:39] [I] sample0029.png [11/06/2025-11:57:39] [I] sample0030.png [11/06/2025-11:57:39] [I] sample0031.png [11/06/2025-11:57:39] [I] sample0032.png [11/06/2025-11:57:39] [I] sample0033.png [11/06/2025-11:57:39] [I] sample0034.png [11/06/2025-11:57:39] [I] sample0035.png [11/06/2025-11:57:39] [I] sample0036.png [11/06/2025-11:57:39] [I] sample0037.png [11/06/2025-11:57:39] [I] sample0038.png [11/06/2025-11:57:39] [I] sample0039.png [11/06/2025-11:57:39] [I] sample0040.png [11/06/2025-11:57:39] [I] sample0041.png [11/06/2025-11:57:39] [I] sample0042.png [11/06/2025-11:57:39] [I] sample0043.png [11/06/2025-11:57:39] [I] sample0044.png [11/06/2025-11:57:39] [I] sample0045.png [11/06/2025-11:57:39] [I] sample0046.png [11/06/2025-11:57:39] [I] sample0047.png [11/06/2025-11:57:39] [I] sample0048.png [11/06/2025-11:57:39] [I] sample0049.png [11/06/2025-11:57:39] [I] sample0050.png [11/06/2025-11:57:39] [I] sample0051.png [11/06/2025-11:57:39] [I] sample0052.png [11/06/2025-11:57:39] [I] sample0053.png [11/06/2025-11:57:39] [I] sample0054.png [11/06/2025-11:57:39] [I] sample0055.png [11/06/2025-11:57:39] [I] sample0056.png [11/06/2025-11:57:39] [I] sample0057.png [11/06/2025-11:57:39] [I] sample0058.png [11/06/2025-11:57:39] [I] sample0059.png [11/06/2025-11:57:39] [I] sample0060.png [11/06/2025-11:57:39] [I] sample0061.png [11/06/2025-11:57:39] [I] sample0062.png [11/06/2025-11:57:39] [I] sample0063.png [11/06/2025-11:57:39] [I] sample0064.png [11/06/2025-11:57:39] [I] sample0065.png [11/06/2025-11:57:39] [I] sample0066.png [11/06/2025-11:57:39] [I] sample0067.png [11/06/2025-11:57:39] [I] sample0068.png [11/06/2025-11:57:39] [I] sample0069.png [11/06/2025-11:57:39] [I] sample0070.png [11/06/2025-11:57:39] [I] sample0071.png [11/06/2025-11:57:39] [I] sample0072.png [11/06/2025-11:57:39] [I] sample0073.png [11/06/2025-11:57:39] [I] sample0074.png [11/06/2025-11:57:39] [I] sample0075.png [11/06/2025-11:57:39] [I] sample0076.png [11/06/2025-11:57:39] [I] sample0077.png [11/06/2025-11:57:39] [I] sample0078.png [11/06/2025-11:57:39] [I] sample0079.png [11/06/2025-11:57:39] [I] sample0080.png [11/06/2025-11:57:39] [I] sample0081.png [11/06/2025-11:57:39] [I] sample0082.png [11/06/2025-11:57:39] [I] sample0083.png [11/06/2025-11:57:39] [I] sample0084.png [11/06/2025-11:57:39] [I] sample0085.png [11/06/2025-11:57:39] [I] sample0086.png [11/06/2025-11:57:39] [I] sample0087.png [11/06/2025-11:57:39] [I] sample0088.png [11/06/2025-11:57:39] [I] sample0089.png [11/06/2025-11:57:39] [I] sample0090.png [11/06/2025-11:57:39] [I] sample0091.png [11/06/2025-11:57:39] [I] sample0092.png [11/06/2025-11:57:39] [I] sample0093.png [11/06/2025-11:57:39] [I] sample0094.png [11/06/2025-11:57:39] [I] sample0095.png [11/06/2025-11:57:39] [I] sample0096.png [11/06/2025-11:57:39] [I] sample0097.png [11/06/2025-11:57:39] [I] sample0098.png [11/06/2025-11:57:39] [I] sample0099.png [11/06/2025-11:57:39] [I] sample0100.png [11/06/2025-11:57:39] [I] sample0101.png [11/06/2025-11:57:39] [I] sample0102.png [11/06/2025-11:57:39] [I] sample0103.png [11/06/2025-11:57:39] [I] sample0104.png [11/06/2025-11:57:39] [I] sample0105.png [11/06/2025-11:57:39] [I] sample0106.png [11/06/2025-11:57:39] [I] sample0107.png [11/06/2025-11:57:39] [I] sample0108.png [11/06/2025-11:57:39] [I] sample0109.png [11/06/2025-11:57:39] [I] sample0110.png [11/06/2025-11:57:39] [I] sample0111.png [11/06/2025-11:57:39] [I] sample0112.png [11/06/2025-11:57:39] [I] sample0113.png [11/06/2025-11:57:39] [I] sample0114.png [11/06/2025-11:57:39] [I] sample0115.png [11/06/2025-11:57:39] [I] sample0116.png [11/06/2025-11:57:39] [I] sample0117.png [11/06/2025-11:57:39] [I] sample0118.png [11/06/2025-11:57:39] [I] sample0119.png [11/06/2025-11:57:39] [I] sample0120.png [11/06/2025-11:57:39] [I] sample0121.png [11/06/2025-11:57:39] [I] sample0122.png [11/06/2025-11:57:39] [I] sample0123.png [11/06/2025-11:57:39] [I] sample0124.png [11/06/2025-11:57:39] [I] sample0125.png [11/06/2025-11:57:39] [I] sample0126.png [11/06/2025-11:57:39] [I] sample0127.png [11/06/2025-11:57:39] [I] sample0128.png [11/06/2025-11:57:39] [I] sample0129.png [11/06/2025-11:57:39] [I] sample0130.png [11/06/2025-11:57:39] [I] sample0131.png [11/06/2025-11:57:39] [I] sample0132.png [11/06/2025-11:57:39] [I] sample0133.png [11/06/2025-11:57:39] [I] sample0134.png [11/06/2025-11:57:39] [I] sample0135.png [11/06/2025-11:57:39] [I] sample0136.png [11/06/2025-11:57:39] [I] sample0137.png [11/06/2025-11:57:39] [I] sample0138.png [11/06/2025-11:57:39] [I] sample0139.png [11/06/2025-11:57:39] [I] sample0140.png [11/06/2025-11:57:39] [I] sample0141.png [11/06/2025-11:57:39] [I] sample0142.png [11/06/2025-11:57:39] [I] sample0143.png [11/06/2025-11:57:39] [I] sample0144.png [11/06/2025-11:57:39] [I] sample0145.png CalibrationDataReader: 145 images, 145 batches. [11/06/2025-11:57:39] [I] [TRT] Reading Calibration Cache for calibrator: MinMaxCalibration [11/06/2025-11:57:39] [I] [TRT] Generated calibration scales using calibration cache. Make sure that calibration cache has latest scales. [11/06/2025-11:57:39] [I] [TRT] To regenerate calibration cache, please delete the existing one. TensorRT will generate a new calibration cache. [11/06/2025-11:57:39] [W] [TRT] Missing scale and zero-point for tensor DecodeNumDetection, expect fall back to non-int8 implementation for any layer consuming or producing given tensor [11/06/2025-11:57:39] [W] [TRT] Missing scale and zero-point for tensor DecodeDetectionClasses, expect fall back to non-int8 implementation for any layer consuming or producing given tensor [11/06/2025-11:57:39] [I] [TRT] ---------- Layers Running on DLA ---------- [11/06/2025-11:57:39] [I] [TRT] ---------- Layers Running on GPU ---------- [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.0/conv/Conv + PWN(PWN(/model.0/act/Sigmoid), /model.0/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.1/conv/Conv + PWN(PWN(/model.1/act/Sigmoid), /model.1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/cv1/conv/Conv + PWN(PWN(/model.2/cv1/act/Sigmoid), /model.2/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/cv2/conv/Conv + PWN(PWN(/model.2/cv2/act/Sigmoid), /model.2/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/m/m.0/cv1/conv/Conv + PWN(PWN(/model.2/m/m.0/cv1/act/Sigmoid), /model.2/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/m/m.0/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.2/m/m.0/cv2/act/Sigmoid), /model.2/m/m.0/cv2/act/Mul), /model.2/m/m.0/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/cv3/conv/Conv + PWN(PWN(/model.2/cv3/act/Sigmoid), /model.2/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.3/conv/Conv + PWN(PWN(/model.3/act/Sigmoid), /model.3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/cv1/conv/Conv + PWN(PWN(/model.4/cv1/act/Sigmoid), /model.4/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/cv2/conv/Conv + PWN(PWN(/model.4/cv2/act/Sigmoid), /model.4/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/m/m.0/cv1/conv/Conv + PWN(PWN(/model.4/m/m.0/cv1/act/Sigmoid), /model.4/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/m/m.0/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.4/m/m.0/cv2/act/Sigmoid), /model.4/m/m.0/cv2/act/Mul), /model.4/m/m.0/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/m/m.1/cv1/conv/Conv + PWN(PWN(/model.4/m/m.1/cv1/act/Sigmoid), /model.4/m/m.1/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/m/m.1/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.4/m/m.1/cv2/act/Sigmoid), /model.4/m/m.1/cv2/act/Mul), /model.4/m/m.1/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/cv3/conv/Conv + PWN(PWN(/model.4/cv3/act/Sigmoid), /model.4/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.5/conv/Conv + PWN(PWN(/model.5/act/Sigmoid), /model.5/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/cv1/conv/Conv + PWN(PWN(/model.6/cv1/act/Sigmoid), /model.6/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/cv2/conv/Conv + PWN(PWN(/model.6/cv2/act/Sigmoid), /model.6/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.0/cv1/conv/Conv + PWN(PWN(/model.6/m/m.0/cv1/act/Sigmoid), /model.6/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.0/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.6/m/m.0/cv2/act/Sigmoid), /model.6/m/m.0/cv2/act/Mul), /model.6/m/m.0/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.1/cv1/conv/Conv + PWN(PWN(/model.6/m/m.1/cv1/act/Sigmoid), /model.6/m/m.1/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.1/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.6/m/m.1/cv2/act/Sigmoid), /model.6/m/m.1/cv2/act/Mul), /model.6/m/m.1/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.2/cv1/conv/Conv + PWN(PWN(/model.6/m/m.2/cv1/act/Sigmoid), /model.6/m/m.2/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.2/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.6/m/m.2/cv2/act/Sigmoid), /model.6/m/m.2/cv2/act/Mul), /model.6/m/m.2/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/cv3/conv/Conv + PWN(PWN(/model.6/cv3/act/Sigmoid), /model.6/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.7/conv/Conv + PWN(PWN(/model.7/act/Sigmoid), /model.7/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/cv1/conv/Conv + PWN(PWN(/model.8/cv1/act/Sigmoid), /model.8/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/cv2/conv/Conv + PWN(PWN(/model.8/cv2/act/Sigmoid), /model.8/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/m/m.0/cv1/conv/Conv + PWN(PWN(/model.8/m/m.0/cv1/act/Sigmoid), /model.8/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/m/m.0/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.8/m/m.0/cv2/act/Sigmoid), /model.8/m/m.0/cv2/act/Mul), /model.8/m/m.0/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/cv3/conv/Conv + PWN(PWN(/model.8/cv3/act/Sigmoid), /model.8/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.9/cv1/conv/Conv + PWN(PWN(/model.9/cv1/act/Sigmoid), /model.9/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POOLING: /model.9/m/MaxPool [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POOLING: /model.9/m_1/MaxPool [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POOLING: /model.9/m_2/MaxPool [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.9/cv1/act/Mul_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.9/m/MaxPool_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.9/m_1/MaxPool_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.9/cv2/conv/Conv + PWN(PWN(/model.9/cv2/act/Sigmoid), /model.9/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.10/conv/Conv + PWN(PWN(/model.10/act/Sigmoid), /model.10/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] RESIZE: /model.11/Resize [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.11/Resize_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/cv1/conv/Conv + PWN(PWN(/model.13/cv1/act/Sigmoid), /model.13/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/cv2/conv/Conv + PWN(PWN(/model.13/cv2/act/Sigmoid), /model.13/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/m/m.0/cv1/conv/Conv + PWN(PWN(/model.13/m/m.0/cv1/act/Sigmoid), /model.13/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/m/m.0/cv2/conv/Conv + PWN(PWN(/model.13/m/m.0/cv2/act/Sigmoid), /model.13/m/m.0/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/cv3/conv/Conv + PWN(PWN(/model.13/cv3/act/Sigmoid), /model.13/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.14/conv/Conv + PWN(PWN(/model.14/act/Sigmoid), /model.14/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] RESIZE: /model.15/Resize [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.15/Resize_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.4/cv3/act/Mul_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/cv1/conv/Conv + PWN(PWN(/model.17/cv1/act/Sigmoid), /model.17/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/cv2/conv/Conv + PWN(PWN(/model.17/cv2/act/Sigmoid), /model.17/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/m/m.0/cv1/conv/Conv + PWN(PWN(/model.17/m/m.0/cv1/act/Sigmoid), /model.17/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/m/m.0/cv2/conv/Conv + PWN(PWN(/model.17/m/m.0/cv2/act/Sigmoid), /model.17/m/m.0/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/cv3/conv/Conv + PWN(PWN(/model.17/cv3/act/Sigmoid), /model.17/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.18/conv/Conv + PWN(PWN(/model.18/act/Sigmoid), /model.18/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.24/m.0/Conv + PWN(/model.24/Sigmoid) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.14/act/Mul_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/cv1/conv/Conv + PWN(PWN(/model.20/cv1/act/Sigmoid), /model.20/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/cv2/conv/Conv + PWN(PWN(/model.20/cv2/act/Sigmoid), /model.20/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/m/m.0/cv1/conv/Conv + PWN(PWN(/model.20/m/m.0/cv1/act/Sigmoid), /model.20/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/m/m.0/cv2/conv/Conv + PWN(PWN(/model.20/m/m.0/cv2/act/Sigmoid), /model.20/m/m.0/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/cv3/conv/Conv + PWN(PWN(/model.20/cv3/act/Sigmoid), /model.20/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.21/conv/Conv + PWN(PWN(/model.21/act/Sigmoid), /model.21/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.24/m.1/Conv + PWN(/model.24/Sigmoid_1) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.10/act/Mul_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/cv1/conv/Conv + PWN(PWN(/model.23/cv1/act/Sigmoid), /model.23/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/cv2/conv/Conv + PWN(PWN(/model.23/cv2/act/Sigmoid), /model.23/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/m/m.0/cv1/conv/Conv + PWN(PWN(/model.23/m/m.0/cv1/act/Sigmoid), /model.23/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/m/m.0/cv2/conv/Conv + PWN(PWN(/model.23/m/m.0/cv2/act/Sigmoid), /model.23/m/m.0/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/cv3/conv/Conv + PWN(PWN(/model.23/cv3/act/Sigmoid), /model.23/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.24/m.2/Conv + PWN(/model.24/Sigmoid_2) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] PLUGIN_V2: YoloLayer [11/06/2025-11:57:40] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +534, GPU +689, now: CPU 1137, GPU 5200 (MiB) [11/06/2025-11:57:41] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +83, GPU +132, now: CPU 1220, GPU 5332 (MiB) [11/06/2025-11:57:41] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored. [11/06/2025-12:00:45] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size will enable more tactics, please check verbose output for requested sizes. [11/06/2025-12:01:03] [I] [TRT] Total Activation Memory: 1115794944 [11/06/2025-12:01:03] [I] [TRT] Detected 1 inputs and 4 output network tensors. [11/06/2025-12:01:03] [I] [TRT] Total Host Persistent Memory: 175984 [11/06/2025-12:01:03] [I] [TRT] Total Device Persistent Memory: 614912 [11/06/2025-12:01:03] [I] [TRT] Total Scratch Memory: 0 [11/06/2025-12:01:03] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 7 MiB, GPU 553 MiB [11/06/2025-12:01:03] [I] [TRT] [BlockAssignment] Started assigning block shifts. This will take 67 steps to complete. [11/06/2025-12:01:03] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 2.77161ms to assign 6 blocks to 67 nodes requiring 10925056 bytes. [11/06/2025-12:01:03] [I] [TRT] Total Activation Memory: 10925056 [11/06/2025-12:01:04] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1557, GPU 5945 (MiB) [11/06/2025-12:01:04] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +0, now: CPU 1557, GPU 5945 (MiB) [11/06/2025-12:01:04] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +7, GPU +8, now: CPU 7, GPU 8 (MiB) Engine build success! Python call example以下是一个简单Python示例调用C++生成的动态链接库，仅需指定模型文件的路径和视频输入的大小，就能返回视频每一帧的检测结果，并且在视频推理过程中可以动态调整置信度和交并比等参数的阈值。import cv2 import time import ctypes ctypes.CDLL("./build/libyolo_plugin.so", mode=ctypes.RTLD_GLOBAL) ctypes.CDLL("./build/libyolo_utils.so", mode=ctypes.RTLD_GLOBAL) from build import yolov5_trt def draw_detections(image, detections, fps): for detection in detections: class_id = detection['class_id'] x1, y1, x2, y2 = detection['bbox'] confidence = detection['confidence'] cv2.rectangle(image, (x1, y1), (x2, y2), (0x27, 0xC1, 0x36), 2) cv2.putText(image, f"{class_id}:{confidence:.2f}", (x1, y1 - 10), cv2.FONT_HERSHEY_PLAIN, 1.2, (0x27, 0xC1, 0x36), 2) cv2.putText(image, f"FPS: {fps:.2f}", (10, 30), cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 0, 255), 2) return image def main(input_path, output_path): cap = cv2.VideoCapture(input_path) fps = int(cap.get(cv2.CAP_PROP_FPS)) width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) detector = yolov5_trt.YOLOv5Detector("./weights/yolov5s.engine", width, height) writer = cv2.VideoWriter(output_path, cv2.VideoWriter_fourcc(*'MJPG'), fps, (width, height)) fps_list = [] frame_count = 0 total_time = 0.0 while cap.isOpened(): ret, frame = cap.read() if not ret: break start_time = time.time() detections = detector.detect(input_image=frame, input_w=640, input_h=640, conf_thresh=0.45, nms_thresh=0.55) process_time = time.time() - start_time current_fps = 1.0 / process_time if process_time > 0 else 0 frame_count += 1 total_time += process_time fps_list.append(current_fps) image = draw_detections(frame, detections, current_fps) writer.write(image) cap.release() writer.release() if frame_count > 0: avg_fps = frame_count / total_time if total_time > 0 else 0 print(f"Processed {frame_count} frames") print(f"Average FPS: {avg_fps:.2f}") print(f"Min FPS: {min(fps_list):.2f}") print(f"Max FPS: {max(fps_list):.2f}") if __name__ == "__main__": input_video = "./media/sample_720p.mp4" output_video = "./result.avi" main(input_video, output_video) 对应的C++推理代码如下：#include "NvInfer.h" #include "logger.h" #include "common.h" #include "buffers.h" #include "utils/preprocess.h" #include "utils/postprocess.h" #include "utils/types.h" #include "utils/utils.h" #include <pybind11/pybind11.h> #include <pybind11/numpy.h> #include <pybind11/stl.h> #include <memory> #include <mutex> namespace py = pybind11; // 将numpy数组转换为cv::Mat cv::Mat numpy_to_mat(py::array_t<unsigned char>& input) { py::buffer_info buf_info = input.request(); if (buf_info.ndim == 3) { // 彩色图像 int height = buf_info.shape[0]; int width = buf_info.shape[1]; int channels = buf_info.shape[2]; cv::Mat mat(height, width, CV_8UC3, (unsigned char*)buf_info.ptr); return mat.clone(); } else if (buf_info.ndim == 2) { // 灰度图像 int height = buf_info.shape[0]; int width = buf_info.shape[1]; cv::Mat mat(height, width, CV_8UC1, (unsigned char*)buf_info.ptr); return mat.clone(); } throw std::runtime_error("Unsupported array dimensions"); } // 将cv::Mat转换为numpy数组 py::array_t<unsigned char> mat_to_numpy(cv::Mat& mat) { if (mat.empty()) { return py::array_t<unsigned char>(); } if (mat.channels() == 1) { // 灰度图像 auto result = py::array_t<unsigned char>({mat.rows, mat.cols}); auto buf = result.request(); memcpy(buf.ptr, mat.data, sizeof(unsigned char) * mat.total()); return result; } else { // 彩色图像 auto result = py::array_t<unsigned char>({mat.rows, mat.cols, mat.channels()}); auto buf = result.request(); memcpy(buf.ptr, mat.data, sizeof(unsigned char) * mat.total() * mat.channels()); return result; } } // 加载模型文件 std::vector<unsigned char> load_engine_file(const std::string &file_name) { std::vector<unsigned char> engine_data; std::ifstream engine_file(file_name, std::ios::binary); assert(engine_file.is_open() && "Unable to load engine file."); engine_file.seekg(0, engine_file.end); int length = engine_file.tellg(); engine_data.resize(length); engine_file.seekg(0, engine_file.beg); engine_file.read(reinterpret_cast<char *>(engine_data.data()), length); return engine_data; } // YOLOv5推理器类 class YOLOv5Detector { private: std::unique_ptr<nvinfer1::IRuntime> runtime; std::shared_ptr<nvinfer1::ICudaEngine> engine; std::unique_ptr<nvinfer1::IExecutionContext> context; std::unique_ptr<samplesCommon::BufferManager> buffers; bool initialized = false; public: YOLOv5Detector(const std::string& engine_file, int frame_width, int frame_height) { initialize(engine_file); int img_size = frame_width * frame_height; cuda_preprocess_init(img_size); // 申请cuda内存 } void initialize(const std::string& engine_file) { // ========== 1. 创建推理运行时runtime ========== runtime = std::unique_ptr<nvinfer1::IRuntime>(nvinfer1::createInferRuntime(sample::gLogger.getTRTLogger())); if (!runtime) { throw std::runtime_error("Failed to create TensorRT runtime"); } // ========== 2. 反序列化生成engine ========== auto plan = load_engine_file(engine_file); engine = std::shared_ptr<nvinfer1::ICudaEngine>(runtime->deserializeCudaEngine(plan.data(), plan.size())); if (!engine) { throw std::runtime_error("Failed to deserialize engine"); } // ========== 3. 创建执行上下文context ========== context = std::unique_ptr<nvinfer1::IExecutionContext>(engine->createExecutionContext()); if (!context) { throw std::runtime_error("Failed to create execution context"); } // ========== 4. 创建输入输出缓冲区 ========== buffers = std::make_unique<samplesCommon::BufferManager>(engine); initialized = true; } py::list detect(py::array_t<unsigned char>& input_image, int input_w=kInputW, int input_h=kInputH, float conf_thresh=kConfThresh, float nms_thresh=kNmsThresh) { if (!initialized) { throw std::runtime_error("Detector not initialized"); } // 将numpy数组转换为cv::Mat cv::Mat frame = numpy_to_mat(input_image); if (frame.empty()) { throw std::runtime_error("Invalid input image"); } // CUDA预处理 process_input_gpu(frame, (float *)buffers->getDeviceBuffer(kInputTensorName), input_w, input_h); // ========== 5. 执行推理 ========== context->executeV2(buffers->getDeviceBindings().data()); // 拷贝回host buffers->copyOutputToHost(); // 从buffer manager中获取模型输出 int32_t *num_det = (int32_t *)buffers->getHostBuffer(kOutNumDet); int32_t *cls = (int32_t *)buffers->getHostBuffer(kOutDetCls); float *conf = (float *)buffers->getHostBuffer(kOutDetScores); float *bbox = (float *)buffers->getHostBuffer(kOutDetBBoxes); // 执行nms（非极大值抑制） std::vector<Detection> bboxs; yolo_nms(bboxs, num_det, cls, conf, bbox, conf_thresh, nms_thresh); // 返回检测结果 py::list result_list; for (size_t j = 0; j < bboxs.size(); j++) { cv::Rect r = get_rect(frame, bboxs[j].bbox, input_w, input_h); py::dict detection; detection["class_id"] = (int)bboxs[j].class_id; detection["confidence"] = (float)bboxs[j].conf; detection["bbox"] = py::cast(std::vector<int>{r.x, r.y, r.x + r.width, r.y + r.height}); result_list.append(detection); } return result_list; } }; // Python绑定代码 PYBIND11_MODULE(yolov5_trt, m) { m.doc() = "YOLOv5 TensorRT Python bindings"; py::class_<YOLOv5Detector>(m, "YOLOv5Detector") .def(py::init<const std::string&, int, int>(), "Initialize detector with engine file", py::arg("engine_file"), py::arg("frame_width"), py::arg("frame_height")) .def("detect", &YOLOv5Detector::detect, "Perform detection on input image", py::arg("input_image"), py::arg("input_w") = kInputW, py::arg("input_h") = kInputH, py::arg("conf_thresh") = kConfThresh, py::arg("nms_thresh") = kNmsThresh); } 实际在Jetson Oron Nano (8GB)上对720P输入大小的视频进行目标检测，平均帧率稳定在120+ FPS，满足工业场景下对实时性的要求。python yolov5_infer.py[11/06/2025-15:23:26] [I] [TRT] Loaded engine size: 7 MiB Deserialize yoloLayer plugin: YoloLayer [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +536, GPU +955, now: CPU 830, GPU 4470 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +83, GPU +149, now: CPU 913, GPU 4619 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +7, now: CPU 0, GPU 7 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 913, GPU 4620 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +3, now: CPU 913, GPU 4623 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +11, now: CPU 0, GPU 18 (MiB) Processed 1442 frames Average FPS: 127.51 Min FPS: 75.75 Max FPS: 134.67 Conclusion Remarks最后我们还提供了ByteTrack跟踪算法的Python绑定，基于Pybind11实现，并在原有算法基础上提供了跟踪目标的类别信息，Jetson Orin Nano也能在此基础上也能实现高达83 FPS的实时目标检测和跟踪性能：ByteTrack-Pybind11: 高性能实时目标跟踪解决方案 🚀