基于 LSTM-Transformer 混合架构的跨尺度时间序列预测模型研究1. 引言智能体（AI Agent）在应对复杂任务时，通常需要具备环境感知、自主推理及预测能力。对于金融量化、智能调度、设备运维预警等任务领域，时间序列预测是智能体核心推理模块中的关键能力。传统的 RNN/LSTM 模型擅长捕捉时间依赖结构，但对长序列能力有限； Transformer 则具备优秀的全局依赖建模能力，却在小规模数据中表现不稳定。因此，将 LSTM 的局部时序优势 与 Transformer 的长期依赖与注意力机制 相结合，可以构造更强鲁棒性与泛化能力的混合预测模块，使 AI Agent 具备真正的数据感知与智能推理能力。2. 模型设计思路：LSTM + Transformer 的混合结构混合建模主要目标：模块优势适用作用LSTM捕捉连续时间依赖；适用于小数据场景作为前端特征提取器Transformer Encoder学习全局注意力与特征关联作为后端增强预测结构Linear Regression Layer简化输出预测最终预测输出层模型结构示意：输入序列 -> LSTM -> 中间时间特征 -> Transformer Encoder -> 全局特征 -> 输出预测3. 环境准备建议使用 Python + PyTorchpip install torch numpy matplotlib4. 代码实战：构建混合时间序列预测模型4.1 数据构造（模拟时序数据）import numpy as np import torch from torch import nn import matplotlib.pyplot as plt # 创建模拟时间序列数据 (sin 曲线 + 噪声) x = np.linspace(0, 100, 1000) data = np.sin(x) + np.random.normal(scale=0.1, size=len(x)) def create_dataset(series, input_len=30, pred_len=1): X, Y = [], [] for i in range(len(series) - input_len - pred_len): X.append(series[i:i+input_len]) Y.append(series[i+input_len:i+input_len+pred_len]) return np.array(X), np.array(Y) input_len = 30 pred_len = 1 X, Y = create_dataset(data, input_len, pred_len) # 转换为 Tensor X = torch.tensor(X, dtype=torch.float32).unsqueeze(-1) Y = torch.tensor(Y, dtype=torch.float32) 4.2 创建混合模型 LSTM + Transformerclass HybridLSTMTransformer(nn.Module): def __init__(self, input_dim=1, hidden_dim=64, num_layers=1, nhead=4, trans_layers=2): super().__init__() self.lstm = nn.LSTM(input_dim, hidden_dim, num_layers, batch_first=True) encoder_layer = nn.TransformerEncoderLayer(d_model=hidden_dim, nhead=nhead, batch_first=True) self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=trans_layers) self.fc = nn.Linear(hidden_dim, 1) def forward(self, x): lstm_out, _ = self.lstm(x) # [batch, seq, hidden] trans_out = self.transformer(lstm_out) out = self.fc(trans_out[:, -1, :]) # 取最后时刻结果 return out4.3 模型训练model = HybridLSTMTransformer() criterion = nn.MSELoss() optimizer = torch.optim.Adam(model.parameters(), lr=0.001) epochs = 20 for epoch in range(epochs): model.train() optimizer.zero_grad() output = model(X) loss = criterion(output, Y) loss.backward() optimizer.step() if epoch % 5 == 0: print(f"Epoch {epoch}/{epochs}, Loss: {loss.item():.6f}") 4.4 模型预测与可视化model.eval() preds = model(X).detach().numpy() plt.plot(Y[:300], label="Real") plt.plot(preds[:300], label="Prediction") plt.title("Hybrid Forecast") plt.legend() plt.show() 5. 实际部署到 AI Agent 的应用场景本模块可作为智能体的 “预测 Plug-in”：场景应用方式模块定位智能电网能耗预测多变量预测预测与策略调度子模块金融交易策略K线 + 宏观变量建模风控与决策子模块工业安全设备传感器异常预警监测与告警模块智能运维调度CPU/GPU/网络指标预测负载均衡优化决策可结合 Agent 框架（如 LangChain、Autogen、Janus、LazyLLM）进行封装并创建 predict_tool() 工具函数供自主调用。6. 总结本文构建了适用于 AI Agent 的 LSTM + Transformer 混合预测模块，能够有效提升小样本与长序列场景下的预测鲁棒性。该结构具有良好的泛化能力，适合作为 AI Agent 的可插拔模块，在决策、控制与风险预判中发挥关键作用。本文针对 AI Agent 在复杂环境下需要具备的未来趋势判断能力，提出了一种 融合式时间序列预测模块设计方案，将 LSTM 的局部时序建模能力 与 Transformer 的全局依赖特征获取能力 进行结合，从而在小样本、长依赖、多周期波动的预测任务中取得更优表现。同时通过 PyTorch 给出可直接运行的混合模型实现代码，帮助开发者快速构建可扩展的预测模块。综合来看，该混合结构具有以下优势：兼顾短期与长期依赖，避免 LSTM 的长期衰减与 Transformer 对数据规模的依赖；模型结构通用性强，可轻松嵌入任意 AI Agent 决策系统；适用领域广泛，包括金融量化、设备运维、智能电网、医疗监测及智能运维调度等；可扩展性强，能够与多变量特征融合，并支持与强化学习、AutoML、MLOps 平台结合，实现全流程自治决策。未来可考虑在以下方向进行升级与迭代：引入 Informer、TimesNet、Mamba、Hybrid CNN 等更先进架构；加入 注意力解释性模块 提升可解释性；与 知识图谱+大模型推理 结合，实现数据驱动 + 知识驱动的混合推理；在 生产环境中加入实时反馈与自适应更新机制，实现端到端闭环智能。通过以上技术整合，AI Agent 将不仅能够“看见当前”，还能够 预测未来并主动决策，从而迈向真正意义的自主智能体。

Agent 与外部工具交互接口标准化：RESTful 与 gRPC 的适配实现随着大模型驱动的智能体（Agent）在业务流程自动化、知识检索、数据分析、运维执行等场景中快速落地，如何让 Agent 可靠、高效、可扩展地调用外部工具（Tools） 成为系统架构设计的重要核心问题。在工程实践中，工具的接入形式多种多样：RESTful API、gRPC 服务、本地函数、本地可执行程序、消息队列等。为了实现 可维护性、可扩展性与可迁移性，构建统一的工具交互标准非常关键。本文重点聚焦——RESTful 与 gRPC 的接口标准化适配实现，并给出完整代码示例，帮助你在自己的 Agent 框架中实现通用化外部工具访问层。一、Agent 调用外部工具为什么需要标准化？1. 外部工具接口形式复杂企业内部服务常见的接口类型包括：RESTful：HTTP + JSON，通用性强，但性能一般；gRPC：二进制传输 + HTTP/2，高性能强类型，适合高频调用；Python/Java 本地方法：绑定不同语言环境；数据库/队列/搜索引擎：SQL、消息协议等。若每种接口都由 Agent 的业务逻辑单独适配，会导致：代码重复且难维护接口升级导致 Agent 要整体修改无法统一调用流程与权限管理2. 标准化的收益构建一个 Tool Adapter 层 可以带来：统一调用规范：Agent 调用工具无需关心背后的协议可扩展性强：新增工具只需扩展一个 Adapter 类日志可统一跟踪：便于审计、故障定位降低 Agent 实现复杂度：实现真正的“能力插件化”二、统一接口规范设计（RESTful/gRPC 适配）我们定义一个通用的工具接口协议，让所有外部调用都遵循统一结构。1. 统一接口协议（伪代码）{ "tool_name": "string", "action": "string", "params": { "key": "value" } } 工具返回结构统一为：{ "status": "success|error", "data": {}, "message": "" } 这样 Agent 内核只需要处理：传什么工具？工具的输入是什么？工具的输出是什么？不关心 REST/gRPC 或者其他协议。2. Adapter 标准接口（Python）from abc import ABC, abstractmethod class BaseToolAdapter(ABC): @abstractmethod def call(self, action: str, params: dict) -> dict: pass 所有工具的接入只需继承这个基类。三、RESTful 工具适配器实现（代码实战）RESTful API 是企业最常见的服务形式，我们实现一个 RESTfulToolAdapter。1. 实战代码：RESTful Tool 适配器import requests class RESTfulToolAdapter(BaseToolAdapter): def __init__(self, base_url): self.base_url = base_url def call(self, action: str, params: dict) -> dict: url = f"{self.base_url}/{action}" try: resp = requests.post(url, json=params, timeout=5) resp.raise_for_status() return { "status": "success", "data": resp.json(), "message": "" } except Exception as e: return { "status": "error", "data": None, "message": str(e) } 2. RESTful API 示例服务（Flask）这里构建一个简单的 RESTful 工具：做加法计算。# math_tool_rest.py from flask import Flask, request, jsonify app = Flask(__name__) @app.route("/add", methods=["POST"]) def add(): data = request.get_json() a = data.get("a") b = data.get("b") return jsonify({"result": a + b}) if __name__ == "__main__": app.run(port=5001) 3. Agent 调用示例rest_tool = RESTfulToolAdapter("http://localhost:5001") agent_input = {"a": 3, "b": 5} result = rest_tool.call("add", agent_input) print(result) 输出：{ "status": "success", "data": {"result": 8}, "message": "" } 四、gRPC 工具适配器实现（代码实战）gRPC 常用于高性能、强类型、跨语言通信场景。1. gRPC 服务定义（math.proto）syntax = "proto3"; service MathTool { rpc Add (AddRequest) returns (AddReply) {} } message AddRequest { int32 a = 1; int32 b = 2; } message AddReply { int32 result = 1; } 生成 Python 代码：python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. math.proto2. gRPC 服务端实现# math_tool_grpc_server.py import grpc from concurrent import futures import math_pb2 import math_pb2_grpc class MathToolServicer(math_pb2_grpc.MathToolServicer): def Add(self, request, context): return math_pb2.AddReply(result=request.a + request.b) server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) math_pb2_grpc.add_MathToolServicer_to_server(MathToolServicer(), server) server.add_insecure_port("[::]:50052") server.start() server.wait_for_termination() 3. gRPC Tool 适配器实现import grpc import math_pb2 import math_pb2_grpc class GRPCToolAdapter(BaseToolAdapter): def __init__(self, host="localhost", port=50052): channel = grpc.insecure_channel(f"{host}:{port}") self.stub = math_pb2_grpc.MathToolStub(channel) def call(self, action: str, params: dict) -> dict: try: if action == "add": req = math_pb2.AddRequest(a=params["a"], b=params["b"]) reply = self.stub.Add(req) return {"status": "success", "data": {"result": reply.result}} else: return {"status": "error", "message": "unknown action", "data": None} except Exception as e: return {"status": "error", "message": str(e), "data": None} 4. Agent 调用示例grpc_tool = GRPCToolAdapter() result = grpc_tool.call("add", {"a": 10, "b": 20}) print(result) 返回：{ "status": "success", "data": {"result": 30} } 五、Agent 层的通用工具调用入口（抽象化）我们最终可以在 Agent 系统中实现工具的统一注册与调用。class ToolRegistry: def __init__(self): self.tools = {} def register(self, name, adapter): self.tools[name] = adapter def call(self, name, action, params): tool = self.tools.get(name) if not tool: return {"status": "error", "message": "tool not found"} return tool.call(action, params) 使用方式：registry = ToolRegistry() registry.register("math_rest", rest_tool) registry.register("math_grpc", grpc_tool) print(registry.call("math_rest", "add", {"a": 1, "b": 2})) print(registry.call("math_grpc", "add", {"a": 5, "b": 6})) 六、RESTful 与 gRPC 的选型对比项目RESTfulgRPC性能中高协议JSON/HTTP1.1ProtoBuf/HTTP2类型安全弱强跨语言优秀优秀调试难度低中等适用场景通用服务高频快速通信、分布式系统推荐：工具响应慢 / 通用服务 → RESTful高频调用 / 数据量大 / 强类型 → gRPC七、总结本文从 Agent 系统视角，介绍了如何构建 统一的工具交互接口标准化体系，并分别给出了：RESTful 实现gRPC 实现Agent 的统一调用入口这种适配层设计能够极大提升 Agent 在复杂生产系统中的接入能力，使其真正成为 可扩展、可插拔、可维护 的智能能力平台。

智能体知识更新机制：增量式知识图谱构建与融合技术一、引言随着大模型驱动的 智能体（Agent） 在企业业务、搜索问答、自动化运营、RPA 流程中广泛应用，知识更新问题变得尤为突出。传统知识库构建方式往往是“全量构建”，一旦知识发生变化，需要重新爬取、处理、训练或重建索引，成本高且不可实时。而智能体需要：识别新知识是否已存在将新知识增量式注入知识图谱自动融合冲突信息实时影响推理过程不中断系统运行因此，“增量式知识图谱构建与融合（Incremental KG Construction & Fusion）”成为智能体落地的核心能力之一。本文系统介绍智能体的增量知识更新体系，并提供一个 Python 版知识图谱增量构建代码实战。二、智能体知识更新机制概述2.1 三类知识更新模式更新模式特点适用场景全量更新（Full Rebuild）一次性重建全部知识小型知识库、结构稳定批次更新（Batch Update）定期增量、按批次更新中型知识库、更新频率适中实时增量更新（Incremental Update）每条知识都实时更新并融合智能体实时问答、监控、自动决策智能体通常需要 实时增量 模式。三、增量式知识图谱构建流程完整流程如下：新数据流入 → 信息抽取 → 知识对齐/消歧 → 图谱增量更新 → 冲突融合 → 推理索引更新3.1 信息抽取（NER / RE）智能体通过 LLM 或规则抽取：实体（Entity）关系（Relation）属性（Attribute）例如从文本 “张三是华为的软件工程师” 中抽取：{ "entity": ["张三", "华为"], "relation": [ ["张三", "就职于", "华为"], ["张三", "职位", "软件工程师"] ] } 3.2 知识对齐与消歧（Entity Alignment）智能体判断新实体是否已存在：文本相似度语义向量距离属性匹配（公司、邮箱、ID）例如“华为技术有限公司” ≈ “华为”。3.3 图谱增量更新（Graph Update）增量更新关键技术：节点重复检测（Duplicate Check）关系冲突解决（Conflict Resolution）时间戳与版本管理（Versioning）3.4 冲突融合（Knowledge Fusion）三类融合方式：融合类型特点优先级融合权重高的知识覆盖低权重证据投票多来源数据投票决定可信度时间序列融合新知识覆盖旧知识四、系统架构设计一个典型的智能体增量知识图谱系统包含： ┌──────────────┐ Data Source → │ 信息抽取模块 │→ 实体/关系 └──────────────┘ ↓ ┌──────────────┐ │ 对齐/消歧模块 │ └──────────────┘ ↓ ┌────────────────┐ │ 图谱增量更新模块│ └────────────────┘ ↓ ┌──────────────┐ │ 冲突融合模块 │ └──────────────┘ ↓ ┌──────────────┐ │ 查询与推理层 │ └──────────────┘五、代码实战：基于 NetworkX 构建简易增量知识图谱系统本示例将展示：如何增量加入实体与关系如何处理重复实体如何自动融合更新属性使用 Python + NetworkX 进行模拟（真实系统可用 Neo4j / ArangoDB / Nebula / RDF）。5.1 安装依赖pip install networkx5.2 基础图谱骨架import networkx as nx class KnowledgeGraph: def __init__(self): self.graph = nx.MultiDiGraph() # 增量添加实体 def add_entity(self, entity_id, attrs): if entity_id not in self.graph: self.graph.add_node(entity_id, **attrs) else: # 属性融合：新属性覆盖旧属性 existing = self.graph.nodes[entity_id] existing.update(attrs) # 增量添加关系 def add_relation(self, src, relation, dst): self.graph.add_edge(src, dst, relation=relation) # 打印图谱 def show(self): for node in self.graph.nodes(data=True): print("Node:", node) for edge in self.graph.edges(data=True): print("Edge:", edge) 5.3 增量知识注入（含消歧与融合）定义：基于相似度判断是否需要复用节点。实战中通常采用 embedding（如 BERT）计算语义相似度，本示例用简单字符串相似度作演示。from difflib import SequenceMatcher def similar(a, b): return SequenceMatcher(None, a, b).ratio() 智能体增量更新逻辑class IncrementalAgent: def __init__(self, kg: KnowledgeGraph, threshold=0.7): self.kg = kg self.threshold = threshold # 简单的实体对齐 def align_entity(self, new_name): for exist in self.kg.graph.nodes: if similar(exist, new_name) > self.threshold: return exist return new_name # 增量注入一条知识 def inject(self, entity, attrs, relation, target): # 实体消歧 entity_id = self.align_entity(entity) target_id = self.align_entity(target) # 写入实体（自动融合属性） self.kg.add_entity(entity_id, attrs) self.kg.add_entity(target_id, {}) # 写入关系 self.kg.add_relation(entity_id, relation, target_id) 5.4 测试增量更新代码kg = KnowledgeGraph() agent = IncrementalAgent(kg) # 第一条知识 agent.inject("华为", {"type": "Company"}, "雇佣", "张三") agent.inject("张三", {"title": "软件工程师"}, "就职于", "华为") # 第二条知识（同实体不同写法，会自动对齐到“华为”） agent.inject("华为技术有限公司", {"location": "深圳"}, "旗下", "荣耀") # 第三条知识（属性融合：覆盖/新增属性） agent.inject("张三", {"title": "高级工程师"}, "参与项目", "Atlas 900") kg.show() 5.5 运行结果示例（示意）Node: ('华为', {'type': 'Company', 'location': '深圳'}) Node: ('张三', {'title': '高级工程师'}) Node: ('荣耀', {}) Node: ('Atlas 900', {}) Edge: ('华为', '张三', {'relation': '雇佣'}) Edge: ('张三', '华为', {'relation': '就职于'}) Edge: ('华为', '荣耀', {'relation': '旗下'}) Edge: ('张三', 'Atlas 900', {'relation': '参与项目'}) 可以看到：“华为技术有限公司” 自动对齐到 华为张三 的 title 属性被融合并更新为 “高级工程师”图谱结构实现增量式扩展六、增量知识融合中的技术难点6.1 消歧精度与误判需结合：文本相似度Embedding 语义距离RDF 属性匹配图结构上下文6.2 知识冲突处理常见冲突：属性冲突时间冲突数据来源可信度冲突6.3 图谱规模增长与性能百万级节点后需考虑：图数据库索引优化分布式图存储流式写入（Streaming Write）七、典型应用场景场景描述企业知识库实时更新文档、FAQ、规范持续变化实时监控智能体新事件即时写入并触发推理金融风控图谱客户/交易/风险节点实时更新RAG Agent 知识补全外部知识动态扩展推荐系统用户画像用户行为增量更新八、总结增量式知识图谱是智能体从“静态问答”迈向“持续进化”的关键能力。本文从架构、流程、关键技术、冲突融合等角度展开，同时提供了一个可运行的 增量图谱构建与融合代码案例。未来方向包括：融合 RAG 与 KG 的混合检索大模型驱动的自动消歧事件图谱（Event Graph）实时流处理图谱 + 大模型联合推理

  【华为云Versatile智能体平台】产品体验官活动 获奖名单如下：一、高价值建议奖：昵称被评为高价值需求 内容展示建议分值礼品码事漫谈增加Agent发布渠道，如微信小程序等7（高价值需求3票、建议采纳需求1票）500元开发者大礼包1份ddhsaVersatile预置base64转图片插件6（高价值需求2票、建议采纳需求2票）500元开发者大礼包1份福州司马懿工作流应用-代码节点优化，去除预处理代码，提升用户使用效率5（高价值需求2票、建议采纳需求1票）500元开发者大礼包1份评分规则：由产品团队组成的评审团对所有参与者提交的建议进行投票，其中高价值需求(2分)、建议采纳需求(1分)，TOP3分值的建议获选。 二、建议贡献排名奖：昵称积分排名（被采纳建议，每条得1分）礼品鸢尾离夏7200元开发者大礼包1份小草飞上天6200元开发者大礼包1份yd_2709879825200元开发者大礼包1份HDC-Feng3200元开发者大礼包1份林欣3200元开发者大礼包1份 恭喜以上8名获奖用户，礼品会发到活动报名时填写报名问卷的账号和收货地址（小助手将联系提供），请注意查收。感谢大家对华为云Versatile智能体平台的关注和支持~ 欢迎各位开发者们在云声平台提出更多优化建议帮助Versatile产品的优化迭代共同构建易用、好用、开放的一站式Agent平台   传送门活动帖子(指定评论区)：cid:link_2 活动报名页面：cid:link_1 产品体验入口：华为开发者空间--开发平台--Versatile Agent

Pi0☁云端推理部署Pi0 是一个通用机器人策略基础模型，专为解决机器人学习中的数据稀缺、泛化能力差和鲁棒性不足等核心挑战而设计。借鉴大语言模型的训练方法，Pi0 通过大规模预训练掌握广泛的机器人操作技能，并能通过微调快速适应具体任务需求。该模型支持高效的数据利用与快速部署，在有限数据下也能实现良好性能，为机器人智能化提供了一种可扩展的解决方案。1. 环境配置在之前LeRobot安装及使用教程和Pi0 模型训练的基础上，Pi0的推理也建议使用云端部署的方式运行。a. 我租赁的服务器是H20，在服务端（openpi环境）中运行：cd /root/autodl-tmp/openpi_episode1_student uv run scripts/serve_policy.py policy:checkpoint --policy.config=enpei_robot_demo_move_toy_low_mem_finetune --policy.dir=checkpoints/enpei_robot_demo_move_toy_low_mem_finetune/my_experiment/9999启动服务器后，他会暴露一个服务端口（默认是 6006）：warning: The `tool.uv.dev-dependencies` field (used in `packages/openpi-client/pyproject.toml`) is deprecated and will be removed in a future release; use `dependency-groups.dev` instead INFO:root:Loading model... INFO:2025-11-09 14:40:27,523:jax._src.xla_bridge:925: Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig' INFO:jax._src.xla_bridge:Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig' INFO:2025-11-09 14:40:27,524:jax._src.xla_bridge:925: Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory INFO:jax._src.xla_bridge:Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory INFO:absl:orbax-checkpoint version: 0.11.13 INFO:absl:Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7f78fd908250> INFO:absl:Restoring checkpoint from /root/autodl-tmp/openpi_episode1_student/checkpoints/enpei_robot_demo_move_toy_low_mem_finetune/my_experiment/9999/params. INFO:absl:[thread=MainThread] Failed to get flag value for EXPERIMENTAL_ORBAX_USE_DISTRIBUTED_PROCESS_ID. INFO:absl:[process=0] /jax/checkpoint/read/bytes_per_sec: 1.1 GiB/s (total bytes: 6.1 GiB) (time elapsed: 5 seconds) (per-host) INFO:absl:Finished restoring checkpoint in 5.72 seconds from /root/autodl-tmp/openpi_episode1_student/checkpoints/enpei_robot_demo_move_toy_low_mem_finetune/my_experiment/9999/params. INFO:root:Loaded norm stats from /root/autodl-tmp/openpi_episode1_student/assets/enpei_robot_demo_move_toy_low_mem_finetune/hou/demo_move_toy_openpi INFO:root:Loaded norm stats from /root/autodl-tmp/openpi_episode1_student/checkpoints/enpei_robot_demo_move_toy_low_mem_finetune/my_experiment/9999/assets/hou/demo_move_toy_openpi INFO:root:Creating server (host: autodl-container-9af746813b-5ccfe20e, ip: 172.17.0.8) INFO:websockets.server:server listening on 0.0.0.0:6006 INFO:websockets.server:connection open INFO:openpi.serving.websocket_policy_server:Connection from ('127.0.0.1', 53688) opened INFO:openpi.serving.websocket_policy_server:Connection from ('127.0.0.1', 53688) closed INFO:websockets.server:connection open INFO:openpi.serving.websocket_policy_server:Connection from ('127.0.0.1', 56818) opened INFO:openpi.serving.websocket_policy_server:Connection from ('127.0.0.1', 56818) closed在AutoDL中使用它的端口转发服务（自定义服务）：b. 在客户端（lerobot环境）进入Lerobot仓库代码，中安装openpi-client：cd ./packages/openpi-client pip install -e . c. 启动机械臂进行归零，安装腕部相机和夹爪，查看相机ID:(base) hou@hou-Ubuntu:~/workspace/lerobot_single_student$ conda activate lerobot (lerobot) hou@hou-Ubuntu:~/workspace/lerobot_single_student$ python -m lerobot.episode_default_position --ip="localhost" --port=12345 INFO 2025-11-09 14:25:56 _position.py:33 Connected to EnpeiRobot controller at localhost:12345 INFO 2025-11-09 14:26:02 _position.py:46 Moving to default position, estimated time: 1.72s INFO 2025-11-09 14:26:04 _position.py:50 Successfully moved to default position (lerobot) hou@hou-Ubuntu:~/workspace/lerobot_single_student$ python -m lerobot.find_cameras opencv --- Detected Cameras --- Camera #0: Name: OpenCV Camera @ /dev/video0 Type: OpenCV Id: /dev/video0 Backend api: V4L2 Default stream profile: Format: 0.0 Width: 640 Height: 480 Fps: 30.0 -------------------- Camera #1: Name: OpenCV Camera @ /dev/video2 Type: OpenCV Id: /dev/video2 Backend api: V4L2 Default stream profile: Format: 0.0 Width: 640 Height: 480 Fps: 30.0 -------------------- Camera #2: Name: OpenCV Camera @ /dev/video4 Type: OpenCV Id: /dev/video4 Backend api: V4L2 Default stream profile: Format: 0.0 Width: 640 Height: 480 Fps: 30.0 -------------------- Finalizing image saving... Image capture finished. Images saved to outputs/captured_images2. Pi0 推理启动客户端，在终端中运行run.sh：host 服务器地址（这里因为用了 AutoDL 本地转发，所以是 localhost）port 服务器端口instruction 文本指令，保持和采集数据一致enpei_use_radian 需要使用弧度制#!/bin/bash python -m lerobot.test_openpi \ --robot.ip_address="localhost" \ --robot.port=12345 \ --robot.type=enpei_follower \ --robot.id=enpei_follower \ --robot.cameras="{ handeye: {type: opencv, index_or_path: 4, width: 320, height: 240, fps: 30}, fixed: {type: opencv, index_or_path: 0, width: 320, height: 240, fps: 30}}" \ --host=localhost \ --port=6006 \ --instruction="Put the toy to the white box" \ --fps=30 \ --enpei_use_radian=true (lerobot) hou@hou-Ubuntu:~/workspace/lerobot_single_student$ ./run.sh INFO 2025-11-09 14:48:58 nt_policy.py:30 Waiting for server at ws://localhost:6006... Connected to remote policy server at localhost:6006 Server metadata: {} Using instruction: Put the toy to the white box 设置enpei follower机器人角度单位: 弧度 INFO 2025-11-09 14:48:58 follower.py:134 Connected to EnpeiRobot controller INFO 2025-11-09 14:49:03 a_opencv.py:176 OpenCVCamera(4) connected. INFO 2025-11-09 14:49:05 a_opencv.py:176 OpenCVCamera(0) connected. INFO 2025-11-09 14:49:05 follower.py:156 enpei_follower EnpeiFollower connected. Robot connected successfully 运行策略，目标FPS: 30 推理线程启动 等待第一个动作生成... wait for frame: 0.0018558219999249559 wait for frame: 0.0018982179999511573 Loop duration: 1.4765s, Real FPS: 0.68, Action FPS: 33.87, Action count: 50, Action shape: (50, 7) Loop duration: 1.4735s, Real FPS: 0.68, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.4755s, Real FPS: 0.68, Action FPS: 33.89, Action count: 50, Action shape: (50, 7) Loop duration: 1.4973s, Real FPS: 0.67, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5040s, Real FPS: 0.66, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5590s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5400s, Real FPS: 0.65, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.4911s, Real FPS: 0.67, Action FPS: 33.91, Action count: 50, Action shape: (50, 7) Loop duration: 1.4938s, Real FPS: 0.67, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.4853s, Real FPS: 0.67, Action FPS: 33.92, Action count: 50, Action shape: (50, 7) Loop duration: 1.4957s, Real FPS: 0.67, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.4930s, Real FPS: 0.67, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.4914s, Real FPS: 0.67, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.4878s, Real FPS: 0.67, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.4911s, Real FPS: 0.67, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.4914s, Real FPS: 0.67, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5189s, Real FPS: 0.66, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.4915s, Real FPS: 0.67, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5207s, Real FPS: 0.66, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.4865s, Real FPS: 0.67, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5499s, Real FPS: 0.65, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.6458s, Real FPS: 0.61, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.6337s, Real FPS: 0.61, Action FPS: 33.98, Action count: 50, Action shape: (50, 7) Loop duration: 1.5200s, Real FPS: 0.66, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5218s, Real FPS: 0.66, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5184s, Real FPS: 0.66, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5273s, Real FPS: 0.65, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5247s, Real FPS: 0.66, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5285s, Real FPS: 0.65, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5576s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5193s, Real FPS: 0.66, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5519s, Real FPS: 0.64, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5240s, Real FPS: 0.66, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.6051s, Real FPS: 0.62, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5251s, Real FPS: 0.66, Action FPS: 33.92, Action count: 50, Action shape: (50, 7) Loop duration: 1.5533s, Real FPS: 0.64, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5564s, Real FPS: 0.64, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5579s, Real FPS: 0.64, Action FPS: 33.92, Action count: 50, Action shape: (50, 7) Loop duration: 1.5550s, Real FPS: 0.64, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5526s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5507s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5511s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5557s, Real FPS: 0.64, Action FPS: 33.92, Action count: 50, Action shape: (50, 7) Loop duration: 1.5514s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5539s, Real FPS: 0.64, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5573s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5767s, Real FPS: 0.63, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5736s, Real FPS: 0.64, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5553s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5537s, Real FPS: 0.64, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5514s, Real FPS: 0.64, Action FPS: 33.98, Action count: 50, Action shape: (50, 7) Loop duration: 1.6261s, Real FPS: 0.61, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.6378s, Real FPS: 0.61, Action FPS: 33.94, Action count: 50, Action shape: (50, 7) Loop duration: 1.5520s, Real FPS: 0.64, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) Loop duration: 1.5588s, Real FPS: 0.64, Action FPS: 33.98, Action count: 50, Action shape: (50, 7) Loop duration: 1.5567s, Real FPS: 0.64, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.5534s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5481s, Real FPS: 0.65, Action FPS: 33.97, Action count: 50, Action shape: (50, 7) Loop duration: 1.6520s, Real FPS: 0.61, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5514s, Real FPS: 0.64, Action FPS: 33.96, Action count: 50, Action shape: (50, 7) Loop duration: 1.5552s, Real FPS: 0.64, Action FPS: 33.95, Action count: 50, Action shape: (50, 7) Loop duration: 1.5494s, Real FPS: 0.65, Action FPS: 33.93, Action count: 50, Action shape: (50, 7) ^C Client stopped by user 推理线程结束测试视频：3. 小结本文介绍了Pi0云端推理部署的完整流程。首先配置好服务器端环境并启动策略服务，然后在客户端安装所需依赖并连接机械臂与摄像头。实测下来，即使对于没见过的云宝机械臂的抓取放置成功率也能达到100%，体现了模型强大的泛化能力！

11月15日，CANN Meetup 北京站，邀您共赴一场技术盛宴

Pi0 模型训练Pi0 是一个通用机器人策略基础模型，专为解决机器人学习中的数据稀缺、泛化能力差和鲁棒性不足等核心挑战而设计。借鉴大语言模型的训练方法，Pi0 通过大规模预训练掌握广泛的机器人操作技能，并能通过微调快速适应具体任务需求。该模型支持高效的数据利用与快速部署，在有限数据下也能实现良好性能，为机器人智能化提供了一种可扩展的解决方案。1. 环境安装在LeRobot安装及使用教程的基础上，训练Pi0需要显存较大的显卡，我租赁的实例是 H20，使用VS Code SSH远程连接服务器：# ssh -p 17825 root@region-42.seetacloud.com Host region-42 HostName region-42.seetacloud.com User root Port 17825 在安装Pi0的过程中需要访问Github和HuggingFace等国外网站，建议开启AutoDL内置的学术加速服务，由于该加速服务可能对正常网络造成一定影响，安装过程中如果失败可以多次尝试，当不再需要时建议取消学术加速。# 启用学术加速 source /etc/network_turbo cd /root/autodl-tmp git clone https://github.com/enpeizhao/openpi_episode1_student.git cd openpi_episode1_student/ pip install uv GIT_LFS_SKIP_SMUDGE=1 uv sync GIT_LFS_SKIP_SMUDGE=1 uv pip install -e . # 取消学术加速 unset http_proxy && unset https_proxy2. 转换数据Pi0训练必须是弧度制数据，安装ffmpeg转换为openpi格式：sudo apt update sudo apt install ffmpeg uv run ./examples/libero/lerobot2oppi.py \ --source-repo-id=hou/demo_move_toy \ --target-repo-id=hou/demo_move_toy_openpi \ --output-path=./demo_move_toy_openpi \ --source-dataset-root=/root/autodl-tmp/demo_move_toy \ --max-episodes=100 3. 修改配置以下是我的配置文件，为了节省训练时间num_train_steps设置为10_000：# 单臂配置：openpi_episode1_student/src/openpi/training/config.py TrainConfig( name="enpei_robot_demo_move_toy_low_mem_finetune", # Here is an example of loading a pi0 model for LoRA fine-tuning. model=pi0.Pi0Config(paligemma_variant="gemma_2b_lora", action_expert_variant="gemma_300m_lora"), data=LeRobotLiberoDataConfig( repo_id="hou/demo_move_toy_openpi", # 数据集repo_od root="./demo_move_toy_openpi", # 数据集路径 base_config=DataConfig(prompt_from_task=True), ), weight_loader=weight_loaders.CheckpointWeightLoader("/root/autodl-tmp/pi0_base/params"), num_train_steps=10_000, # The freeze filter defines which parameters should be frozen during training. # We have a convenience function in the model config that returns the default freeze filter # for the given model config for LoRA finetuning. Just make sure it matches the model config # you chose above. freeze_filter=pi0.Pi0Config( paligemma_variant="gemma_2b_lora", action_expert_variant="gemma_300m_lora" ).get_freeze_filter(), # Turn off EMA for LoRA finetuning. ema_decay=None, ), 4. 模型训练# 计算 normalization uv run scripts/compute_norm_stats.py --config-name enpei_robot_demo_move_toy_low_mem_finetune # 开启训练 XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py enpei_robot_demo_move_toy_low_mem_finetune --exp-name=my_experiment --overwrite在训练过程中可以看到模型的LOSS在逐渐降低，也可以在终端中查看GPU的利用率，采集100个episodes数据使用H20迭代10_000步大概需要花费9个小时：训练过程中Pi0会保存多个时间点的权重checkpoint，每次约消耗磁盘12G，因此建议将代码目录放到/root/autodl-tmp数据盘目录下，这样训练过程中产生的权重文件会自动保存在/root/autodl-tmp/checkpoints下，避免系统盘因空间不足导致训练失败。5. 本文小结本文介绍了在H20实例上安装和训练Pi0模型的完整流程，包括环境配置、数据转换、修改训练配置以及模型训练等关键步骤，并提供了相关命令和注意事项，帮助读者快速上手Pi0模型训练。

Addcdiv算子的核心公式是 y = input_data + value * (x1 / x2)。这个看似简单的组合操作，在机器学习和科学计算中其实是一个相当基础且重要的模式。它通常不会作为一个独立的层出现，而是作为构建更复杂模型或算法的基本砖块。其最主要的应用场景集中在以下几个方面：1. 优化算法与参数更新这是Addcdiv最经典和广泛的应用。在许多先进的优化器中，我们都能看到它的影子。最典型的代表是Adam和RMSProp等自适应学习率算法。以Adam为例：在更新模型参数时，Adam会计算梯度的一阶矩（均值）和二阶矩（未中心化的方差）估计。参数更新步骤大致为：参数 = 参数 - 学习率 * (一阶矩估计 / (sqrt(二阶矩估计) + epsilon))。映射到Addcdiv：在这里，input_data就是待更新的参数，value是负的学习率，x1是一阶矩估计，x2是sqrt(二阶矩估计) + epsilon。这个操作在每个训练步骤中为数百万甚至数十亿的参数执行一次，因此其计算效率至关重要。用高度优化的Addcdiv算子来实现，能极大提升整个训练过程的吞吐量。2. 注意力机制中的归一化操作在Transformer架构中，注意力权重的计算通常涉及一个缩放操作。标准的点积注意力公式为：Attention(Q, K, V) = softmax(Q * K^T / sqrt(d_k)) * V。映射到Addcdiv：在计算Q * K^T之后，我们需要将结果除以一个标量sqrt(d_k)。这个过程可以看作是一个广播操作：x1是注意力得分矩阵，x2是一个标量sqrt(d_k)，而input_data和value在这里可以视为0和1（即纯除法）。虽然在最终实现时可能被更专用的内核融合，但从计算模式上看，它符合Addcdiv的范畴。3. 物理模拟与科学计算在这些领域，许多物理量的更新遵循类似的模式。例如，在计算流体动力学或分子动力学模拟中，一个粒子的新位置可能等于旧位置加上速度与时间的乘积（这本身是加法与乘法的组合），而速度又可能是力与质量的比值。当多个这样的基本操作组合在一起时，就可能抽象出Addcdiv的模式。虽然这些领域可能使用双精度浮点数，但其计算图与Addcdiv是相通的。4. 自定义梯度与反向传播当研究人员或工程师在开发新的模型结构时，可能会设计出自定义的数学操作。如果这个自定义操作的前向传播恰好符合a + b * (c / d)的形式，那么在编写其反向传播（求导）时，其梯度计算也极有可能包含类似的结构。拥有一个高效的Addcdiv原语，可以方便地将这些自定义操作在底层高效实现，而无需手动组合多个基础算子，从而避免了多次启动内核和中间结果读写的开销。总结来说，Addcdiv算子的用武之地在于它封装了一个频繁出现且计算密集的复合操作模式。它的价值不仅在于其数学功能，更在于其性能优势。在AI芯片上，将多个基础操作（除法、乘法、加法）融合成一个单一的Addcdiv内核，可以显著减少：内核启动开销：从启动三个内核减少到一个。内存带宽压力：中间结果（如x1/x2）可以直接在芯片的高速缓存（如UB）中进行计算，而无需写回至外部DRAM再读回。因此，尽管用户在日常编程中可能不会直接调用它，但Addcdiv作为底层支撑算子，在训练大规模神经网络、运行复杂优化算法时，默默地发挥着提升整体计算效率的关键作用。

如何在Python中调用C++版本的ByteTrack跟踪算法这个项目提供了基于ByteTrack-TensorRT的Python插件，并在原有算法基础上提供了跟踪目标的类别信息，Jetson Orin Nano在之前YOLOv5插件的基础上实现高达83 FPS的实时检测跟踪性能。⚡ 极致性能: 基于TensorRT优化，充分利用硬件加速📦 开箱即用：构建过程简单，快速部署您的跟踪应用🐍 Python 友好: 使用Pybind11提供简洁Python接口📱 边缘设备优化: 特别针对Jetson边缘设备进行适配Build plugin首先安装必要的库克隆仓库构建项目，注意JetPack 5.x版本才能正常运行：sudo apt update sudo apt install ffmpeg sudo apt install pybind11-dev sudo apt install libeigen3-dev git cone https://github.com/HouYanSong/bytetrack_pybind11.git cd bytetrack_pybind11 pip install pybind11 rm -fr build cmake -S . -B build cmake --build build[ 12%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/BYTETracker.cpp.o [ 25%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/STrack.cpp.o [ 37%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/kalmanFilter.cpp.o [ 50%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/lapjv.cpp.o [ 62%] Building CXX object CMakeFiles/bytetrack.dir/bytetrack/src/utils.cpp.o [ 75%] Linking CXX shared library libbytetrack.so [ 75%] Built target bytetrack [ 87%] Building CXX object CMakeFiles/bytetrack_trt.dir/bytetrack_trt.cpp.o [100%] Linking CXX shared module bytetrack_trt.cpython-38-aarch64-linux-gnu.so [100%] Built target bytetrack_trtRun demo我们提供了一个简单的Python示例，只需要导入C++构建的Python动态链接库就可以非常方便的调用ByteTrack跟踪算法，返回目标位置、跟踪ID和类别信息。import cv2 import time import ctypes # 加载依赖库 ctypes.CDLL("./yolov5_trt_plugin/libyolo_plugin.so", mode=ctypes.RTLD_GLOBAL) ctypes.CDLL("./yolov5_trt_plugin/libyolo_utils.so", mode=ctypes.RTLD_GLOBAL) ctypes.CDLL("./build/libbytetrack.so", mode=ctypes.RTLD_GLOBAL) # 导入YOLOv5检测器和ByteTrack跟踪器 from yolov5_trt_plugin import yolov5_trt from build import bytetrack_trt def draw_image(image, detections, tracks, fps): for track in tracks: x, y, w, h = track.tlwh track_id = track.track_id class_id = track.label x1, y1, x2, y2 = int(x), int(y), int(x+w), int(y+h) cv2.rectangle(image, (x1, y1), (x2, y2), (0, 255, 0), 2) cv2.putText(image, f"C:{class_id} T:{track_id}", (x1, y1 - 10), cv2.FONT_HERSHEY_PLAIN, 1.2, (0, 0, 255), 2) cv2.putText(image, f"FPS: {fps:.2f}", (10, 30), cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 0, 255), 2) return image def main(input_path, output_path): cap = cv2.VideoCapture(input_path) fps_value = int(cap.get(cv2.CAP_PROP_FPS)) width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) writer = cv2.VideoWriter(output_path, cv2.VideoWriter_fourcc(*'MJPG'), fps_value, (width, height)) detector = yolov5_trt.YOLOv5Detector("./yolov5_trt_plugin/yolov5s.engine", width, height) tracker = bytetrack_trt.BYTETracker(frame_rate = fps_value, track_buffer = 30) fps_list = [] frame_count = 0 total_time = 0.0 while cap.isOpened(): ret, frame = cap.read() if not ret: break start_time = time.time() # 目标检测 detections = detector.detect(input_image=frame, input_w=640, input_h=640, conf_thresh=0.45, nms_thresh=0.55) objects = [] for det in detections: x1, y1, x2, y2 = det['bbox'] rect = bytetrack_trt.RectFloat(x1, y1, x2-x1, y2-y1) # x, y, width, height obj = bytetrack_trt.Object() obj.rect = rect obj.label = det['class_id'] obj.prob = det['confidence'] objects.append(obj) # 目标跟踪 tracks = tracker.update(objects) process_time = time.time() - start_time current_fps = 1.0 / process_time if process_time > 0 else 0 frame_count += 1 total_time += process_time fps_list.append(current_fps) # 图像绘制 image = draw_image(frame, detections, tracks, current_fps) writer.write(image) cap.release() writer.release() if frame_count > 0: avg_fps = frame_count / total_time if total_time > 0 else 0 print(f"Processed {frame_count} frames") print(f"Average FPS: {avg_fps:.2f}") print(f"Min FPS: {min(fps_list):.2f}") print(f"Max FPS: {max(fps_list):.2f}") if __name__ == "__main__": input_video = "./media/sample_720p.mp4" output_video = "./result.avi" main(input_video, output_video) 仅需在终端中运行yolov5_bytetrack.py脚本：python yolov5_bytetrack.py[11/07/2025-17:13:10] [I] [TRT] Loaded engine size: 8 MiB Deserialize yoloLayer plugin: YoloLayer [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +536, GPU +702, now: CPU 841, GPU 3927 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +83, GPU +94, now: CPU 924, GPU 4021 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +7, now: CPU 0, GPU 7 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 924, GPU 4021 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +1, now: CPU 924, GPU 4022 (MiB) [11/07/2025-17:13:12] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +11, now: CPU 0, GPU 18 (MiB) Init ByteTrack! Processed 1442 frames Average FPS: 83.78 Min FPS: 68.31 Max FPS: 113.35 Conclusion Remarks本文实现了ByteTrack-TensorRT跟踪算法的Python插件，并在原有算法基础上提供了跟踪目标的类别信息，Jetson Orin Nano (8GB)上的YOLOv5实时目标检测和跟踪速度高达80FPS，满足对快速运动目标的跟踪需求。

如何在Jetson上将YOLOv5实时检测速度提升至120+FPS这个项目提供了基于 Pybind11 的 TensorRT YOLOv5 插件 Python 绑定，实现了令人难以置信的实时目标检测性能！⚡ 超100FPS性能: 在 Jetson Orin Nano 上轻松实现超过 120 帧/秒的检测速度🎯 高精度检测: 基于成熟的 YOLOv5 架构，准确识别COCO数据集上的80类目标🔌 即插即用: 简单的 Python 接口，无需复杂的配置🛠️ 工业级优化: 采用 TensorRT 进行模型优化和加速1. Building the plugin首先安装必要的库克隆仓库构建项目，注意JetPack 5.x版本才能正常运行：sudo apt update sudo apt install ffmpeg sudo apt install pybind11-dev git clone https://github.com/HouYanSong/yolov5_trt_pybind11.git cd yolov5_trt_pybind11 pip install pybind11 rm -fr build cmake -S . -B build cmake --build build2. Model quantization生成量化图片对YOLOv5s模型进行Int8量化，保存量化后的模型：./media/gen_calib.sh ./build/build weights/yolov5s.onnx 1 ./media/ ./media/filelist.txt weights/yolov5s.engine[11/06/2025-11:57:36] [I] [TRT] [MemUsageChange] Init CUDA: CPU +221, GPU +0, now: CPU 249, GPU 4229 (MiB) [11/06/2025-11:57:39] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +302, GPU +277, now: CPU 574, GPU 4529 (MiB) [11/06/2025-11:57:39] [I] [TRT] ---------------------------------------------------------------- [11/06/2025-11:57:39] [I] [TRT] Input filename: weights/yolov5s.onnx [11/06/2025-11:57:39] [I] [TRT] ONNX IR version: 0.0.7 [11/06/2025-11:57:39] [I] [TRT] Opset version: 12 [11/06/2025-11:57:39] [I] [TRT] Producer name: [11/06/2025-11:57:39] [I] [TRT] Producer version: [11/06/2025-11:57:39] [I] [TRT] Domain: [11/06/2025-11:57:39] [I] [TRT] Model version: 0 [11/06/2025-11:57:39] [I] [TRT] Doc string: [11/06/2025-11:57:39] [I] [TRT] ---------------------------------------------------------------- [11/06/2025-11:57:39] [I] [TRT] No importer registered for op: YoloLayer_TRT. Attempting to import as plugin. [11/06/2025-11:57:39] [I] [TRT] Searching for plugin: YoloLayer_TRT, plugin_version: 1, plugin_namespace: [11/06/2025-11:57:39] [I] [TRT] Successfully created plugin: YoloLayer_TRT [11/06/2025-11:57:39] [I] sample0001.png [11/06/2025-11:57:39] [I] sample0002.png [11/06/2025-11:57:39] [I] sample0003.png [11/06/2025-11:57:39] [I] sample0004.png [11/06/2025-11:57:39] [I] sample0005.png [11/06/2025-11:57:39] [I] sample0006.png [11/06/2025-11:57:39] [I] sample0007.png [11/06/2025-11:57:39] [I] sample0008.png [11/06/2025-11:57:39] [I] sample0009.png [11/06/2025-11:57:39] [I] sample0010.png [11/06/2025-11:57:39] [I] sample0011.png [11/06/2025-11:57:39] [I] sample0012.png [11/06/2025-11:57:39] [I] sample0013.png [11/06/2025-11:57:39] [I] sample0014.png [11/06/2025-11:57:39] [I] sample0015.png [11/06/2025-11:57:39] [I] sample0016.png [11/06/2025-11:57:39] [I] sample0017.png [11/06/2025-11:57:39] [I] sample0018.png [11/06/2025-11:57:39] [I] sample0019.png [11/06/2025-11:57:39] [I] sample0020.png [11/06/2025-11:57:39] [I] sample0021.png [11/06/2025-11:57:39] [I] sample0022.png [11/06/2025-11:57:39] [I] sample0023.png [11/06/2025-11:57:39] [I] sample0024.png [11/06/2025-11:57:39] [I] sample0025.png [11/06/2025-11:57:39] [I] sample0026.png [11/06/2025-11:57:39] [I] sample0027.png [11/06/2025-11:57:39] [I] sample0028.png [11/06/2025-11:57:39] [I] sample0029.png [11/06/2025-11:57:39] [I] sample0030.png [11/06/2025-11:57:39] [I] sample0031.png [11/06/2025-11:57:39] [I] sample0032.png [11/06/2025-11:57:39] [I] sample0033.png [11/06/2025-11:57:39] [I] sample0034.png [11/06/2025-11:57:39] [I] sample0035.png [11/06/2025-11:57:39] [I] sample0036.png [11/06/2025-11:57:39] [I] sample0037.png [11/06/2025-11:57:39] [I] sample0038.png [11/06/2025-11:57:39] [I] sample0039.png [11/06/2025-11:57:39] [I] sample0040.png [11/06/2025-11:57:39] [I] sample0041.png [11/06/2025-11:57:39] [I] sample0042.png [11/06/2025-11:57:39] [I] sample0043.png [11/06/2025-11:57:39] [I] sample0044.png [11/06/2025-11:57:39] [I] sample0045.png [11/06/2025-11:57:39] [I] sample0046.png [11/06/2025-11:57:39] [I] sample0047.png [11/06/2025-11:57:39] [I] sample0048.png [11/06/2025-11:57:39] [I] sample0049.png [11/06/2025-11:57:39] [I] sample0050.png [11/06/2025-11:57:39] [I] sample0051.png [11/06/2025-11:57:39] [I] sample0052.png [11/06/2025-11:57:39] [I] sample0053.png [11/06/2025-11:57:39] [I] sample0054.png [11/06/2025-11:57:39] [I] sample0055.png [11/06/2025-11:57:39] [I] sample0056.png [11/06/2025-11:57:39] [I] sample0057.png [11/06/2025-11:57:39] [I] sample0058.png [11/06/2025-11:57:39] [I] sample0059.png [11/06/2025-11:57:39] [I] sample0060.png [11/06/2025-11:57:39] [I] sample0061.png [11/06/2025-11:57:39] [I] sample0062.png [11/06/2025-11:57:39] [I] sample0063.png [11/06/2025-11:57:39] [I] sample0064.png [11/06/2025-11:57:39] [I] sample0065.png [11/06/2025-11:57:39] [I] sample0066.png [11/06/2025-11:57:39] [I] sample0067.png [11/06/2025-11:57:39] [I] sample0068.png [11/06/2025-11:57:39] [I] sample0069.png [11/06/2025-11:57:39] [I] sample0070.png [11/06/2025-11:57:39] [I] sample0071.png [11/06/2025-11:57:39] [I] sample0072.png [11/06/2025-11:57:39] [I] sample0073.png [11/06/2025-11:57:39] [I] sample0074.png [11/06/2025-11:57:39] [I] sample0075.png [11/06/2025-11:57:39] [I] sample0076.png [11/06/2025-11:57:39] [I] sample0077.png [11/06/2025-11:57:39] [I] sample0078.png [11/06/2025-11:57:39] [I] sample0079.png [11/06/2025-11:57:39] [I] sample0080.png [11/06/2025-11:57:39] [I] sample0081.png [11/06/2025-11:57:39] [I] sample0082.png [11/06/2025-11:57:39] [I] sample0083.png [11/06/2025-11:57:39] [I] sample0084.png [11/06/2025-11:57:39] [I] sample0085.png [11/06/2025-11:57:39] [I] sample0086.png [11/06/2025-11:57:39] [I] sample0087.png [11/06/2025-11:57:39] [I] sample0088.png [11/06/2025-11:57:39] [I] sample0089.png [11/06/2025-11:57:39] [I] sample0090.png [11/06/2025-11:57:39] [I] sample0091.png [11/06/2025-11:57:39] [I] sample0092.png [11/06/2025-11:57:39] [I] sample0093.png [11/06/2025-11:57:39] [I] sample0094.png [11/06/2025-11:57:39] [I] sample0095.png [11/06/2025-11:57:39] [I] sample0096.png [11/06/2025-11:57:39] [I] sample0097.png [11/06/2025-11:57:39] [I] sample0098.png [11/06/2025-11:57:39] [I] sample0099.png [11/06/2025-11:57:39] [I] sample0100.png [11/06/2025-11:57:39] [I] sample0101.png [11/06/2025-11:57:39] [I] sample0102.png [11/06/2025-11:57:39] [I] sample0103.png [11/06/2025-11:57:39] [I] sample0104.png [11/06/2025-11:57:39] [I] sample0105.png [11/06/2025-11:57:39] [I] sample0106.png [11/06/2025-11:57:39] [I] sample0107.png [11/06/2025-11:57:39] [I] sample0108.png [11/06/2025-11:57:39] [I] sample0109.png [11/06/2025-11:57:39] [I] sample0110.png [11/06/2025-11:57:39] [I] sample0111.png [11/06/2025-11:57:39] [I] sample0112.png [11/06/2025-11:57:39] [I] sample0113.png [11/06/2025-11:57:39] [I] sample0114.png [11/06/2025-11:57:39] [I] sample0115.png [11/06/2025-11:57:39] [I] sample0116.png [11/06/2025-11:57:39] [I] sample0117.png [11/06/2025-11:57:39] [I] sample0118.png [11/06/2025-11:57:39] [I] sample0119.png [11/06/2025-11:57:39] [I] sample0120.png [11/06/2025-11:57:39] [I] sample0121.png [11/06/2025-11:57:39] [I] sample0122.png [11/06/2025-11:57:39] [I] sample0123.png [11/06/2025-11:57:39] [I] sample0124.png [11/06/2025-11:57:39] [I] sample0125.png [11/06/2025-11:57:39] [I] sample0126.png [11/06/2025-11:57:39] [I] sample0127.png [11/06/2025-11:57:39] [I] sample0128.png [11/06/2025-11:57:39] [I] sample0129.png [11/06/2025-11:57:39] [I] sample0130.png [11/06/2025-11:57:39] [I] sample0131.png [11/06/2025-11:57:39] [I] sample0132.png [11/06/2025-11:57:39] [I] sample0133.png [11/06/2025-11:57:39] [I] sample0134.png [11/06/2025-11:57:39] [I] sample0135.png [11/06/2025-11:57:39] [I] sample0136.png [11/06/2025-11:57:39] [I] sample0137.png [11/06/2025-11:57:39] [I] sample0138.png [11/06/2025-11:57:39] [I] sample0139.png [11/06/2025-11:57:39] [I] sample0140.png [11/06/2025-11:57:39] [I] sample0141.png [11/06/2025-11:57:39] [I] sample0142.png [11/06/2025-11:57:39] [I] sample0143.png [11/06/2025-11:57:39] [I] sample0144.png [11/06/2025-11:57:39] [I] sample0145.png CalibrationDataReader: 145 images, 145 batches. [11/06/2025-11:57:39] [I] [TRT] Reading Calibration Cache for calibrator: MinMaxCalibration [11/06/2025-11:57:39] [I] [TRT] Generated calibration scales using calibration cache. Make sure that calibration cache has latest scales. [11/06/2025-11:57:39] [I] [TRT] To regenerate calibration cache, please delete the existing one. TensorRT will generate a new calibration cache. [11/06/2025-11:57:39] [W] [TRT] Missing scale and zero-point for tensor DecodeNumDetection, expect fall back to non-int8 implementation for any layer consuming or producing given tensor [11/06/2025-11:57:39] [W] [TRT] Missing scale and zero-point for tensor DecodeDetectionClasses, expect fall back to non-int8 implementation for any layer consuming or producing given tensor [11/06/2025-11:57:39] [I] [TRT] ---------- Layers Running on DLA ---------- [11/06/2025-11:57:39] [I] [TRT] ---------- Layers Running on GPU ---------- [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.0/conv/Conv + PWN(PWN(/model.0/act/Sigmoid), /model.0/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.1/conv/Conv + PWN(PWN(/model.1/act/Sigmoid), /model.1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/cv1/conv/Conv + PWN(PWN(/model.2/cv1/act/Sigmoid), /model.2/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/cv2/conv/Conv + PWN(PWN(/model.2/cv2/act/Sigmoid), /model.2/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/m/m.0/cv1/conv/Conv + PWN(PWN(/model.2/m/m.0/cv1/act/Sigmoid), /model.2/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/m/m.0/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.2/m/m.0/cv2/act/Sigmoid), /model.2/m/m.0/cv2/act/Mul), /model.2/m/m.0/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.2/cv3/conv/Conv + PWN(PWN(/model.2/cv3/act/Sigmoid), /model.2/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.3/conv/Conv + PWN(PWN(/model.3/act/Sigmoid), /model.3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/cv1/conv/Conv + PWN(PWN(/model.4/cv1/act/Sigmoid), /model.4/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/cv2/conv/Conv + PWN(PWN(/model.4/cv2/act/Sigmoid), /model.4/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/m/m.0/cv1/conv/Conv + PWN(PWN(/model.4/m/m.0/cv1/act/Sigmoid), /model.4/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/m/m.0/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.4/m/m.0/cv2/act/Sigmoid), /model.4/m/m.0/cv2/act/Mul), /model.4/m/m.0/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/m/m.1/cv1/conv/Conv + PWN(PWN(/model.4/m/m.1/cv1/act/Sigmoid), /model.4/m/m.1/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/m/m.1/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.4/m/m.1/cv2/act/Sigmoid), /model.4/m/m.1/cv2/act/Mul), /model.4/m/m.1/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.4/cv3/conv/Conv + PWN(PWN(/model.4/cv3/act/Sigmoid), /model.4/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.5/conv/Conv + PWN(PWN(/model.5/act/Sigmoid), /model.5/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/cv1/conv/Conv + PWN(PWN(/model.6/cv1/act/Sigmoid), /model.6/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/cv2/conv/Conv + PWN(PWN(/model.6/cv2/act/Sigmoid), /model.6/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.0/cv1/conv/Conv + PWN(PWN(/model.6/m/m.0/cv1/act/Sigmoid), /model.6/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.0/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.6/m/m.0/cv2/act/Sigmoid), /model.6/m/m.0/cv2/act/Mul), /model.6/m/m.0/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.1/cv1/conv/Conv + PWN(PWN(/model.6/m/m.1/cv1/act/Sigmoid), /model.6/m/m.1/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.1/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.6/m/m.1/cv2/act/Sigmoid), /model.6/m/m.1/cv2/act/Mul), /model.6/m/m.1/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.2/cv1/conv/Conv + PWN(PWN(/model.6/m/m.2/cv1/act/Sigmoid), /model.6/m/m.2/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/m/m.2/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.6/m/m.2/cv2/act/Sigmoid), /model.6/m/m.2/cv2/act/Mul), /model.6/m/m.2/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.6/cv3/conv/Conv + PWN(PWN(/model.6/cv3/act/Sigmoid), /model.6/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.7/conv/Conv + PWN(PWN(/model.7/act/Sigmoid), /model.7/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/cv1/conv/Conv + PWN(PWN(/model.8/cv1/act/Sigmoid), /model.8/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/cv2/conv/Conv + PWN(PWN(/model.8/cv2/act/Sigmoid), /model.8/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/m/m.0/cv1/conv/Conv + PWN(PWN(/model.8/m/m.0/cv1/act/Sigmoid), /model.8/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/m/m.0/cv2/conv/Conv [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POINTWISE: PWN(PWN(PWN(/model.8/m/m.0/cv2/act/Sigmoid), /model.8/m/m.0/cv2/act/Mul), /model.8/m/m.0/Add) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.8/cv3/conv/Conv + PWN(PWN(/model.8/cv3/act/Sigmoid), /model.8/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.9/cv1/conv/Conv + PWN(PWN(/model.9/cv1/act/Sigmoid), /model.9/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POOLING: /model.9/m/MaxPool [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POOLING: /model.9/m_1/MaxPool [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] POOLING: /model.9/m_2/MaxPool [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.9/cv1/act/Mul_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.9/m/MaxPool_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.9/m_1/MaxPool_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.9/cv2/conv/Conv + PWN(PWN(/model.9/cv2/act/Sigmoid), /model.9/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.10/conv/Conv + PWN(PWN(/model.10/act/Sigmoid), /model.10/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] RESIZE: /model.11/Resize [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.11/Resize_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/cv1/conv/Conv + PWN(PWN(/model.13/cv1/act/Sigmoid), /model.13/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/cv2/conv/Conv + PWN(PWN(/model.13/cv2/act/Sigmoid), /model.13/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/m/m.0/cv1/conv/Conv + PWN(PWN(/model.13/m/m.0/cv1/act/Sigmoid), /model.13/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/m/m.0/cv2/conv/Conv + PWN(PWN(/model.13/m/m.0/cv2/act/Sigmoid), /model.13/m/m.0/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.13/cv3/conv/Conv + PWN(PWN(/model.13/cv3/act/Sigmoid), /model.13/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.14/conv/Conv + PWN(PWN(/model.14/act/Sigmoid), /model.14/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] RESIZE: /model.15/Resize [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.15/Resize_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.4/cv3/act/Mul_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/cv1/conv/Conv + PWN(PWN(/model.17/cv1/act/Sigmoid), /model.17/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/cv2/conv/Conv + PWN(PWN(/model.17/cv2/act/Sigmoid), /model.17/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/m/m.0/cv1/conv/Conv + PWN(PWN(/model.17/m/m.0/cv1/act/Sigmoid), /model.17/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/m/m.0/cv2/conv/Conv + PWN(PWN(/model.17/m/m.0/cv2/act/Sigmoid), /model.17/m/m.0/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.17/cv3/conv/Conv + PWN(PWN(/model.17/cv3/act/Sigmoid), /model.17/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.18/conv/Conv + PWN(PWN(/model.18/act/Sigmoid), /model.18/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.24/m.0/Conv + PWN(/model.24/Sigmoid) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.14/act/Mul_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/cv1/conv/Conv + PWN(PWN(/model.20/cv1/act/Sigmoid), /model.20/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/cv2/conv/Conv + PWN(PWN(/model.20/cv2/act/Sigmoid), /model.20/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/m/m.0/cv1/conv/Conv + PWN(PWN(/model.20/m/m.0/cv1/act/Sigmoid), /model.20/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/m/m.0/cv2/conv/Conv + PWN(PWN(/model.20/m/m.0/cv2/act/Sigmoid), /model.20/m/m.0/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.20/cv3/conv/Conv + PWN(PWN(/model.20/cv3/act/Sigmoid), /model.20/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.21/conv/Conv + PWN(PWN(/model.21/act/Sigmoid), /model.21/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.24/m.1/Conv + PWN(/model.24/Sigmoid_1) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] COPY: /model.10/act/Mul_output_0 copy [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/cv1/conv/Conv + PWN(PWN(/model.23/cv1/act/Sigmoid), /model.23/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/cv2/conv/Conv + PWN(PWN(/model.23/cv2/act/Sigmoid), /model.23/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/m/m.0/cv1/conv/Conv + PWN(PWN(/model.23/m/m.0/cv1/act/Sigmoid), /model.23/m/m.0/cv1/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/m/m.0/cv2/conv/Conv + PWN(PWN(/model.23/m/m.0/cv2/act/Sigmoid), /model.23/m/m.0/cv2/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.23/cv3/conv/Conv + PWN(PWN(/model.23/cv3/act/Sigmoid), /model.23/cv3/act/Mul) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] CONVOLUTION: /model.24/m.2/Conv + PWN(/model.24/Sigmoid_2) [11/06/2025-11:57:39] [I] [TRT] [GpuLayer] PLUGIN_V2: YoloLayer [11/06/2025-11:57:40] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +534, GPU +689, now: CPU 1137, GPU 5200 (MiB) [11/06/2025-11:57:41] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +83, GPU +132, now: CPU 1220, GPU 5332 (MiB) [11/06/2025-11:57:41] [I] [TRT] Local timing cache in use. Profiling results in this builder pass will not be stored. [11/06/2025-12:00:45] [I] [TRT] Some tactics do not have sufficient workspace memory to run. Increasing workspace size will enable more tactics, please check verbose output for requested sizes. [11/06/2025-12:01:03] [I] [TRT] Total Activation Memory: 1115794944 [11/06/2025-12:01:03] [I] [TRT] Detected 1 inputs and 4 output network tensors. [11/06/2025-12:01:03] [I] [TRT] Total Host Persistent Memory: 175984 [11/06/2025-12:01:03] [I] [TRT] Total Device Persistent Memory: 614912 [11/06/2025-12:01:03] [I] [TRT] Total Scratch Memory: 0 [11/06/2025-12:01:03] [I] [TRT] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 7 MiB, GPU 553 MiB [11/06/2025-12:01:03] [I] [TRT] [BlockAssignment] Started assigning block shifts. This will take 67 steps to complete. [11/06/2025-12:01:03] [I] [TRT] [BlockAssignment] Algorithm ShiftNTopDown took 2.77161ms to assign 6 blocks to 67 nodes requiring 10925056 bytes. [11/06/2025-12:01:03] [I] [TRT] Total Activation Memory: 10925056 [11/06/2025-12:01:04] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 1557, GPU 5945 (MiB) [11/06/2025-12:01:04] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +0, now: CPU 1557, GPU 5945 (MiB) [11/06/2025-12:01:04] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +7, GPU +8, now: CPU 7, GPU 8 (MiB) Engine build success! Python call example以下是一个简单Python示例调用C++生成的动态链接库，仅需指定模型文件的路径和视频输入的大小，就能返回视频每一帧的检测结果，并且在视频推理过程中可以动态调整置信度和交并比等参数的阈值。import cv2 import time import ctypes ctypes.CDLL("./build/libyolo_plugin.so", mode=ctypes.RTLD_GLOBAL) ctypes.CDLL("./build/libyolo_utils.so", mode=ctypes.RTLD_GLOBAL) from build import yolov5_trt def draw_detections(image, detections, fps): for detection in detections: class_id = detection['class_id'] x1, y1, x2, y2 = detection['bbox'] confidence = detection['confidence'] cv2.rectangle(image, (x1, y1), (x2, y2), (0x27, 0xC1, 0x36), 2) cv2.putText(image, f"{class_id}:{confidence:.2f}", (x1, y1 - 10), cv2.FONT_HERSHEY_PLAIN, 1.2, (0x27, 0xC1, 0x36), 2) cv2.putText(image, f"FPS: {fps:.2f}", (10, 30), cv2.FONT_HERSHEY_PLAIN, 1.5, (0, 0, 255), 2) return image def main(input_path, output_path): cap = cv2.VideoCapture(input_path) fps = int(cap.get(cv2.CAP_PROP_FPS)) width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)) height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)) detector = yolov5_trt.YOLOv5Detector("./weights/yolov5s.engine", width, height) writer = cv2.VideoWriter(output_path, cv2.VideoWriter_fourcc(*'MJPG'), fps, (width, height)) fps_list = [] frame_count = 0 total_time = 0.0 while cap.isOpened(): ret, frame = cap.read() if not ret: break start_time = time.time() detections = detector.detect(input_image=frame, input_w=640, input_h=640, conf_thresh=0.45, nms_thresh=0.55) process_time = time.time() - start_time current_fps = 1.0 / process_time if process_time > 0 else 0 frame_count += 1 total_time += process_time fps_list.append(current_fps) image = draw_detections(frame, detections, current_fps) writer.write(image) cap.release() writer.release() if frame_count > 0: avg_fps = frame_count / total_time if total_time > 0 else 0 print(f"Processed {frame_count} frames") print(f"Average FPS: {avg_fps:.2f}") print(f"Min FPS: {min(fps_list):.2f}") print(f"Max FPS: {max(fps_list):.2f}") if __name__ == "__main__": input_video = "./media/sample_720p.mp4" output_video = "./result.avi" main(input_video, output_video) 对应的C++推理代码如下：#include "NvInfer.h" #include "logger.h" #include "common.h" #include "buffers.h" #include "utils/preprocess.h" #include "utils/postprocess.h" #include "utils/types.h" #include "utils/utils.h" #include <pybind11/pybind11.h> #include <pybind11/numpy.h> #include <pybind11/stl.h> #include <memory> #include <mutex> namespace py = pybind11; // 将numpy数组转换为cv::Mat cv::Mat numpy_to_mat(py::array_t<unsigned char>& input) { py::buffer_info buf_info = input.request(); if (buf_info.ndim == 3) { // 彩色图像 int height = buf_info.shape[0]; int width = buf_info.shape[1]; int channels = buf_info.shape[2]; cv::Mat mat(height, width, CV_8UC3, (unsigned char*)buf_info.ptr); return mat.clone(); } else if (buf_info.ndim == 2) { // 灰度图像 int height = buf_info.shape[0]; int width = buf_info.shape[1]; cv::Mat mat(height, width, CV_8UC1, (unsigned char*)buf_info.ptr); return mat.clone(); } throw std::runtime_error("Unsupported array dimensions"); } // 将cv::Mat转换为numpy数组 py::array_t<unsigned char> mat_to_numpy(cv::Mat& mat) { if (mat.empty()) { return py::array_t<unsigned char>(); } if (mat.channels() == 1) { // 灰度图像 auto result = py::array_t<unsigned char>({mat.rows, mat.cols}); auto buf = result.request(); memcpy(buf.ptr, mat.data, sizeof(unsigned char) * mat.total()); return result; } else { // 彩色图像 auto result = py::array_t<unsigned char>({mat.rows, mat.cols, mat.channels()}); auto buf = result.request(); memcpy(buf.ptr, mat.data, sizeof(unsigned char) * mat.total() * mat.channels()); return result; } } // 加载模型文件 std::vector<unsigned char> load_engine_file(const std::string &file_name) { std::vector<unsigned char> engine_data; std::ifstream engine_file(file_name, std::ios::binary); assert(engine_file.is_open() && "Unable to load engine file."); engine_file.seekg(0, engine_file.end); int length = engine_file.tellg(); engine_data.resize(length); engine_file.seekg(0, engine_file.beg); engine_file.read(reinterpret_cast<char *>(engine_data.data()), length); return engine_data; } // YOLOv5推理器类 class YOLOv5Detector { private: std::unique_ptr<nvinfer1::IRuntime> runtime; std::shared_ptr<nvinfer1::ICudaEngine> engine; std::unique_ptr<nvinfer1::IExecutionContext> context; std::unique_ptr<samplesCommon::BufferManager> buffers; bool initialized = false; public: YOLOv5Detector(const std::string& engine_file, int frame_width, int frame_height) { initialize(engine_file); int img_size = frame_width * frame_height; cuda_preprocess_init(img_size); // 申请cuda内存 } void initialize(const std::string& engine_file) { // ========== 1. 创建推理运行时runtime ========== runtime = std::unique_ptr<nvinfer1::IRuntime>(nvinfer1::createInferRuntime(sample::gLogger.getTRTLogger())); if (!runtime) { throw std::runtime_error("Failed to create TensorRT runtime"); } // ========== 2. 反序列化生成engine ========== auto plan = load_engine_file(engine_file); engine = std::shared_ptr<nvinfer1::ICudaEngine>(runtime->deserializeCudaEngine(plan.data(), plan.size())); if (!engine) { throw std::runtime_error("Failed to deserialize engine"); } // ========== 3. 创建执行上下文context ========== context = std::unique_ptr<nvinfer1::IExecutionContext>(engine->createExecutionContext()); if (!context) { throw std::runtime_error("Failed to create execution context"); } // ========== 4. 创建输入输出缓冲区 ========== buffers = std::make_unique<samplesCommon::BufferManager>(engine); initialized = true; } py::list detect(py::array_t<unsigned char>& input_image, int input_w=kInputW, int input_h=kInputH, float conf_thresh=kConfThresh, float nms_thresh=kNmsThresh) { if (!initialized) { throw std::runtime_error("Detector not initialized"); } // 将numpy数组转换为cv::Mat cv::Mat frame = numpy_to_mat(input_image); if (frame.empty()) { throw std::runtime_error("Invalid input image"); } // CUDA预处理 process_input_gpu(frame, (float *)buffers->getDeviceBuffer(kInputTensorName), input_w, input_h); // ========== 5. 执行推理 ========== context->executeV2(buffers->getDeviceBindings().data()); // 拷贝回host buffers->copyOutputToHost(); // 从buffer manager中获取模型输出 int32_t *num_det = (int32_t *)buffers->getHostBuffer(kOutNumDet); int32_t *cls = (int32_t *)buffers->getHostBuffer(kOutDetCls); float *conf = (float *)buffers->getHostBuffer(kOutDetScores); float *bbox = (float *)buffers->getHostBuffer(kOutDetBBoxes); // 执行nms（非极大值抑制） std::vector<Detection> bboxs; yolo_nms(bboxs, num_det, cls, conf, bbox, conf_thresh, nms_thresh); // 返回检测结果 py::list result_list; for (size_t j = 0; j < bboxs.size(); j++) { cv::Rect r = get_rect(frame, bboxs[j].bbox, input_w, input_h); py::dict detection; detection["class_id"] = (int)bboxs[j].class_id; detection["confidence"] = (float)bboxs[j].conf; detection["bbox"] = py::cast(std::vector<int>{r.x, r.y, r.x + r.width, r.y + r.height}); result_list.append(detection); } return result_list; } }; // Python绑定代码 PYBIND11_MODULE(yolov5_trt, m) { m.doc() = "YOLOv5 TensorRT Python bindings"; py::class_<YOLOv5Detector>(m, "YOLOv5Detector") .def(py::init<const std::string&, int, int>(), "Initialize detector with engine file", py::arg("engine_file"), py::arg("frame_width"), py::arg("frame_height")) .def("detect", &YOLOv5Detector::detect, "Perform detection on input image", py::arg("input_image"), py::arg("input_w") = kInputW, py::arg("input_h") = kInputH, py::arg("conf_thresh") = kConfThresh, py::arg("nms_thresh") = kNmsThresh); } 实际在Jetson Oron Nano (8GB)上对720P输入大小的视频进行目标检测，平均帧率稳定在120+ FPS，满足工业场景下对实时性的要求。python yolov5_infer.py[11/06/2025-15:23:26] [I] [TRT] Loaded engine size: 7 MiB Deserialize yoloLayer plugin: YoloLayer [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +536, GPU +955, now: CPU 830, GPU 4470 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +83, GPU +149, now: CPU 913, GPU 4619 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +7, now: CPU 0, GPU 7 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +0, now: CPU 913, GPU 4620 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] Init cuDNN: CPU +0, GPU +3, now: CPU 913, GPU 4623 (MiB) [11/06/2025-15:23:28] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +11, now: CPU 0, GPU 18 (MiB) Processed 1442 frames Average FPS: 127.51 Min FPS: 75.75 Max FPS: 134.67 Conclusion Remarks最后我们还提供了ByteTrack跟踪算法的Python绑定，基于Pybind11实现，并在原有算法基础上提供了跟踪目标的类别信息，Jetson Orin Nano也能在此基础上也能实现高达83 FPS的实时目标检测和跟踪性能：ByteTrack-Pybind11: 高性能实时目标跟踪解决方案 🚀

Janus是一个多模态AI模型系列，其名称源于古罗马的双面神雅努斯，象征模型能同时处理和理解两种不同模态的数据。以deepseek-ai的Janus-1.3B和Janus-Pro-7B为例，这些模型专攻图像与文本的联合理解，其核心架构通常基于改进的视觉语言模型框架。Janus系列采用视觉编码器提取图像特征，再通过跨模态融合模块与文本特征交互。其中Janus-1.3B参数量为13亿，适用于轻量级多模态任务；Janus-Pro-7B则通过70亿参数实现更强的表征能力。这类模型在训练时使用大规模图文对数据集，通过对比学习、掩码重建等目标函数建立模态对齐。与专用单模态模型不同，Janus类模型的核心优势在于跨模态推理。例如它能理解“图像中红色物体的位置”这类需视觉定位与语义解析复合的问题。这种能力源于其双重编码器设计与交叉注意力机制，使模型能在特征层面实现视觉与语言信号的深度融合。当前多模态模型发展呈现参数规模与模态广度同步扩展的趋势。Janus系列可视为OpenAI CLIP、BLIP等模型的演进，其技术路线与Flamingo、GPT-4V等保持架构相关性。这类模型面临的挑战包括模态对齐偏差、计算复杂度优化，以及细粒度语义 grounding 等问题。在实际应用中，Janus类模型可支撑智能客服的图文问答、电商产品的跨模态检索、无障碍技术的图像描述生成等场景。其发展直接影响着具身智能、内容审核、教育科技等领域的技术演进路径。

Qwen2-VL模型将模型逻辑拆分为Qwen2VLModel和Qwen2VLForConditionalGeneration两个类是一种经典的设计模式。这种设计体现了单一职责原则和模块化思想，具有清晰的层次结构。Qwen2VLModel作为基础模型，其核心职责是实现多模态Transformer的编码器-解码器架构。它专注于接收融合后的多模态嵌入（文本、图像、视频），通过多层Transformer块进行特征变换，最终输出隐藏状态。这个类不包含任务特定的输出头，其输出是模型的核心表示，可以作为多种下游任务的通用特征提取器。从接口角度看，它的forward方法返回BaseModelOutputWithPast，包含最后的隐藏状态、过去键值对等中间结果，为更复杂的任务流水线提供基础数据。Qwen2VLForConditionalGeneration作为任务特定模型，在基础模型之上增加了语言建模头。它的核心扩展是将Qwen2VLModel输出的隐藏状态通过一个线性变换层（lm_head）映射到词汇表空间，生成每个位置的下一个词元预测概率。这个设计支持自回归生成任务，能够处理图像描述、视觉问答、多模态对话等需要文本生成的应用场景。它的forward方法返回Qwen2VLCausalLMOutputWithPast，除了基础模型的输出外，还包含语言建模的logits和可选的损失值。这种分离设计的优势体现在多个层面。在工程上，它实现了计算逻辑与任务逻辑的解耦，基础模型可以独立优化和测试，无需依赖特定任务头。在功能上，它支持灵活的模型复用，同一个Qwen2VLModel可以搭配不同的任务头用于多种下游任务。在资源效率上，当仅需要特征提取时，可以单独加载基础模型，节省内存占用。从Hugging Face Transformers库的设计哲学来看，这种模式是标准实践。类似的分离也见于BERT（BertModel/BertForSequenceClassification）、T5（T5Model/T5ForConditionalGeneration）等架构。这种一致性使得开发者能够快速理解新模型的结构，并利用熟悉的API模式进行开发。具体到Qwen2-VL的场景，因为多模态输入的处理已经相当复杂，将核心变换与生成任务分离有助于管理复杂性。基础模型处理多模态融合和表示学习，而条件生成模型专注于利用这些表示进行连贯的文本生成。在实际应用中，如果用户只需要获取多模态输入的联合表示（例如用于检索或分类），可以使用Qwen2VLModel；如果需要模型根据多模态输入生成文本回复，则使用Qwen2VLForConditionalGeneration。

Hugging Face 构建了一个以开源、协作和可访问性为核心理念的机器学习生态系统。这个生态系统并非单一工具，而是一个紧密相连的工具、库、模型和社区平台组成的综合体，其根本目标是降低现代机器学习，尤其是自然语言处理的门槛。其生态系统的基石是 Transformers 库。这个库不仅仅提供了数千个预训练模型的统一接口，更重要的是它定义了一套标准的模型架构、训练和推理流程。这使得研究人员和开发者能够以几行代码加载和使用最先进的模型，如BERT、GPT等，而无需关心底层复杂的实现细节。这种标准化极大地加速了模型的迭代与应用落地。第二个关键组成部分是 Hugging Face Hub，这是一个模型、数据集和演示应用的共享平台。你可以将其理解为机器学习的“Github”。它托管了数十万个模型和数万个数据集，覆盖了从文本、图像到音频的多种模态。这种集中式托管和社区贡献模式，彻底改变了以往模型和数据分散、难以寻找和复现的局面，形成了强大的网络效应和知识聚集。第三个层面是围绕核心库构建的工具链。这包括：Datasets 库：提供了高效、标准化的数据加载和后处理功能，尤其擅长处理大规模数据集。Tokenizers 库：提供了各类分词算法的高速实现，是文本处理的前置关键环节。Accelerate 库：简化了在多GPU、TPU等硬件上的分布式训练和推理代码的编写。Gradio/Spaces：允许用户快速为任何模型构建图形化交互界面，并直接部署在HF平台上，极大地简化了模型的演示和分享。这个生态系统的运作模式形成了一个高效的闭环：研究者可以在Hub上发布新的模型架构和训练方法；开发者使用Transformers库快速集成这些模型到自己的应用中，或使用Datasets库的数据进行微调；最后通过Gradio创建应用进行展示和测试。整个过程都建立在开源和协作的基础上。从更宏观的视角看，Hugging Face生态系统实际上是在为机器学习领域建立事实上的标准。它通过提供一套优秀的、统一的工具，减少了重复劳动，促进了成果的复用和比较，从而推动了整个领域的快速发展。它使得算力或数据资源有限的中小团队甚至个人，也能站在巨人的肩膀上，接触到业界最前沿的技术，这在一定程度上民主化了人工智能的开发能力。Hugging Face生态系统的核心是三位法国籍的联合创始人：Clément Delangue、Julien Chaumond和Thomas Wolf。他们并没有一开始就试图打造一个庞大的平台，而是从一个非常具体的痛点切入：创建一个标准化的、开源的模型库。这就是后来的Transformers库。这个库的成功在于它提供了一个极其友好的API（如from_pretrained），让开发者能够用几行代码就加载和使用最先进的模型。这种低门槛、高价值的特性，使其迅速在开发者社区中获得了快速传播。生态的扩张遵循了“飞轮效应”。当Transformers库吸引了大量开发者后，他们顺势推出了Hugging Face Hub。这个平台解决了模型分享和发现的难题，进一步巩固了其作为NLP中心的地位。开发者不仅来这里找工具，更来这里贡献和协作。随着用户和影响力的增长，他们又将这套模式复制到数据集（Datasets库）、演示应用（Gradio/Spaces）和分布式训练（Accelerate库）等领域，逐步构建起一个完整的工具链。关键的推动力还包括其坚定的开源与商业化结合的独特路径。公司核心的库和平台始终保持着开源和免费，这为其建立了极高的信任和极广的开发者基础。而其商业模式则建立在为企业提供托管、推理和企业级解决方案上（如Inference Endpoints和Trainer SaaS服务）。这种“开源引流，服务变现”的模式，既避免了与传统云厂商的正面竞争，又使其生态保持了活力和中立性。值得一提的是，Thomas Wolf作为首席科学官，在技术愿景和社区互动上起到了关键作用。他本人就是一位活跃的研究员和布道者，深度参与核心库的开发，并与学术社区保持着紧密联系，这确保了生态系统在技术上的前瞻性和权威性。总结：它是由一小群洞察力深刻的创始人，通过解决一个真实且迫切的行业问题起步，以出色的开源项目为杠杆，撬动了庞大的开发者社区，并巧妙地设计了一套可持续的商业模式，最终推动其从一个聊天机器人初创公司演变为今天机器学习领域不可或缺的基础设施。

PyTorch版本的Hugging Face transformers 库中的生成框架成功地将各类语言模型的文本生成能力标准化、民主化。其核心设计哲学是提供一个统一的接口，无论模型架构是编码器-解码器（如T5、BART）还是仅解码器（如GPT系列），用户都能通过简单的 model.generate() 方法调用复杂的生成策略。这背后是一套高度模块化且可扩展的架构。生成过程可以被解构为几个关键组件的协同工作：LogitsProcessor、StoppingCriteria、BeamScorer 以及 Cache。LogitsProcessor 负责在每一步生成时对模型的原始输出（logits）进行加工，例如施加重复惩罚（RepetitionPenaltyLogitsProcessor）、确保最小生成长度（MinLengthLogitsProcessor）或进行核采样（TopPLogitsWarper）。这种设计使得生成内容的控制策略可以像乐高积木一样自由组合。StoppingCriteria 则决定了生成的终止条件，除了常见的最大长度和结束符（EOS）外，用户可以实现自定义的停止逻辑。在解码策略方面，库内实现了从经典到前沿的多种算法。贪婪解码和束搜索（Beam Search）是确定性方法的代表，后者通过维护多个候选序列来寻找局部最优解，广泛应用于机器翻译等任务。而采样类方法，如多项式采样、Top-K采样和Top-P（核）采样，则通过引入随机性来生成更多样、更富有创造性的文本。对比搜索（Contrastive Search）则是一种较新的方法，它通过惩罚与上文高相似的候选token来有效缓解生成中的重复和退化问题，在开放域生成中表现出色。辅助生成（Assisted Generation）或推测解码（Speculative Decoding）代表了生成加速的前沿方向。其核心思想是使用一个更快但更小的“助手模型”来草拟几个未来的token，然后由原始“大模型”并行地验证这些token。如果草拟的token被接受，生成步骤就被大幅加速；如果不被接受，大模型只需纠正第一个出错的token。这种方法能在完全不改变输出质量的前提下，显著提升大模型的推理速度。缓存（Cache）系统是生成效率的关键。为了加速自回归生成中重复的注意力计算，Transformer模型广泛使用了键值缓存（KV Cache）。transformers 库抽象了 Cache 类，并提供了 DynamicCache（动态缓存）和 StaticCache（静态缓存）等实现。动态缓存灵活通用，而静态缓存则针对特定硬件（如GPU）进行了优化，通过预分配和固定形状的内存来减少开销，尤其在长序列生成和批处理场景下能带来显著的性能提升。该生成框架与PyTorch的生态深度集成，兼容其自动微分、动态图特性以及各种设备（CPU、CUDA）。同时，它也考虑了与 accelerate 库的配合，以支持在多个GPU甚至CPU上进行大模型推理（ZeRO阶段3）。其输出被设计为丰富的 ModelOutput 子类，不仅包含生成的序列，还可以根据需要返回每一步的分数、注意力权重、隐藏状态以及过去的键值缓存，为分析和调试提供了极大的便利。总而言之，PyTorch版 transformers 的生成模块通过精心的抽象和模块化设计，在保持接口简洁性的同时，内部实现了极其复杂和多样化的生成逻辑。

  【华为云Versatile智能体平台】产品体验官活动 获奖名单如下： 一、高价值建议奖：昵称被评为高价值需求 内容展示建议分值礼品码事漫谈增加Agent发布渠道，如微信小程序等7（高价值需求3票、建议采纳需求1票）500元开发者大礼包1份ddhsaVersatile预置base64转图片插件6（高价值需求2票、建议采纳需求2票）500元开发者大礼包1份福州司马懿工作流应用-代码节点优化，去除预处理代码，提升用户使用效率5（高价值需求2票、建议采纳需求1票）500元开发者大礼包1份评分规则：由产品团队组成的评审团对所有参与者提交的建议进行投票，其中高价值需求(2分)、建议采纳需求(1分)，TOP3分值的建议获选。 二、建议贡献排名奖：昵称积分排名（被采纳建议，每条得1分）礼品鸢尾离夏7200元开发者大礼包1份小草飞上天6200元开发者大礼包1份yd_2709879825200元开发者大礼包1份HDC-Feng3200元开发者大礼包1份林欣3200元开发者大礼包1份 恭喜以上8名获奖用户，礼品会发到活动报名时填写报名问卷的账号和收货地址（小助手将联系提供），请注意查收。感谢大家对华为云Versatile智能体平台的关注和支持~ 欢迎各位开发者们在云声平台提出更多优化建议帮助Versatile产品的优化迭代共同构建易用、好用、开放的一站式Agent平台  活动介绍：时下大热的AI Agent智能体，正推动千行万业进入AI数字生产力的革新快轨。Versatile智能体平台是华为云在本年度重磅推出的一站式企业级智能体构建平台，面向企业AI+转型、软件及解决方案提供商、独立开发者等多角色多场景，使能高效开发Agent，将“人人都能构建自己的企业级智能体”产品愿景照进现实。为了让Versatile产品能力更贴合用户应用场景、进一步满足用户需求、提升体验，我们发起本次产品体验官招募活动，邀请各位有热情、有想法的开发者们加入Versatile智能体平台体验官阵营，提出真实的产品改进意见，以帮助产品开发和优化迭代。期待联接广大开发者们的力量，合力构建易用、好用、开放的AI Agent平台。图：产品架构图图：产品界面图 活动时间：2025.11.4-2025.12.15 活动流程：1、 在活动页点击报名后，进入官方微信群开通服务（下图），访问产品体验环境：华为开发者空间--开发平台--Versatile Agent2、 参照官网帮助文档（内含实践案例指导）或收录进案例中心的实操案例，进行产品深度使用体验；同时需创建并发布任一类型的智能体应用（至少一个）;3、 将体验完成截图发送至活动指定评论区（本帖下方），跟帖评论体验感受，可以围绕体验流程、操作步骤、问题建议（搭配产品界面截图说明）等方向进行展开说明；4、在官网云声·建议平台，反馈产品优化建议。（注：以上四步需全部完成）  服务开通方式：报名成功后，请参与者扫描二维码进入官方微信群，进群后请发送“我要开通Versatile，账号ID是xxxxxx”，技术支持将辅助在24小时内完成服务开通；同时欢迎在群聊内开展技术交流，反馈操作疑问。（Versatile智能体平台产品体验交流群）TIPS:账号ID获取路径：华为云控制台-右上角账号名下拉窗口-账号ID（红框内，可复制）示意图：  产品体验官职责：为了构建更好的用户体验，各体验官可从以下要素考虑，反馈体验感受，包括：交互体验层面、感官体验层面、bug类、需求类、能力创新类等等。 奖项设置：奖项设置获奖要求获奖名额激励礼品高价值建议奖被评选为高价值需求3每人价值500元开发者礼包1份建议贡献排名奖被采纳建议数≥3条，且根据积分排名TOP55每人价值200元开发者礼包1份说明1、每条被采纳建议累计1积分，被采纳建议数统计截止时间为2025年12月15日24点；2、若出现积分相同且排名一致的情况，因奖品数量有限，根据先到先得原则进行发放；3、同一用户仅限获评一个奖项；4、 如礼品库存不足将存在替换成等价值礼品的可能；5、激励礼品如下：序号礼包名称介绍1500元开发者礼包华为手环8NFC版（黑色）华为12000mh移动电源充电宝（白色）开发者定制不锈钢水杯   开发者空间定制鼠标垫（大号）2200元开发者礼包华为FreeBuds SE 2无线耳机（白色）U型按摩枕开发者空间定制鼠标垫（大号）(奖品示意图）  活动说明：1、 完成体验请进入活动指定评论区（本帖下方），截图体验完成的实际截图，并说明体验感受。提交建议内容需表述清晰，有操作截图、链接等详细描述。2、 您对Versatile智能体平台有任何改进建议，请提交至云声·建议平台（填写时，关联产品/功能 选择“智能体平台Versatile”) 并标明以【Versatile智能体平台体验】为开头，让我们听到你的声音，为产品改进贡献一份力量。示例：【Versatile智能体平台体验】整体上手体验不错，建议考虑XXX场景，实现AI应用一键分享的能力等等。3、所有参与活动的开发者需要完成实际体验，有任何问题欢迎进入Versatile论坛阵地发帖交流，我们将在48小时内解答。  关于华为云Versatile智能体平台产品官网主页：cid:link_5帮助文档：cid:link_1Versatile论坛交流阵地：cid:link_4产品体验入口：华为开发者空间--开发平台--Versatile Agent