-
MindX SDK2.0.4怎么下载,社区版有没有,下载不了
-
已经将 PC 和 MDC 设备在同一网段(192.168.30.42),MTB 310和MDC已经连接,12V供电。电脑和MTB通过网口相连。192.168.30.42可以ping通,但是SSH登录不上MDC 210,截图如下。请问是哪里出的问题
-
安装toolkit安装包时,出现No space left on device的error,该怎么解决?
-
1.用的设备是mdc300f平台2.cann版本是5.0mdc3003.mindstudio版本是2.0.0(bete3)4.onnx模型链接:cid:link_05.在cann6.0可以成功转出om模型,但是最终需要部署到mdc300平台上,必须要用自带的cann转换。自带的5.0mdc300转换报错flatten算子axis必须改为1,于是将flatten算子的axis=3强制转为1,出现了如下图的错误,经过初步分析是flatten算子后gather算子没有起作用。想请教一下如何在改了flatten算子axis值的情况下,还能保持给gather算子正确的输入。
-
参考Mindpore 官方自定义算子文档,修改算子定义,构造正向及反向函数,执行报错,有哪位大神帮忙看下咩靴靴!Mindpore版本1.9.0
-
同一个基于AscendIndexIVFSQT检索方法的demo,在100000底库小数据集上可成功运行:Load data set successfull. index start index init train start Training IVF quantizer on 100000 vectors in 256D to 1024 cluster training computeCompressIndex on correlation matrix Clustering 100000 points in 256D to 1024 clusters, redo 1 times, 16 iterations Preprocessing in 0.03 s training computeCompressValue on 100000 vectors in 256D Iteration 15 (2.73 s, search 2.59 s): objective=87058.3 imbalance=1.155 nsplit=0 training scalar quantizer on 100000 vectors in 64D add start AscendIndex::add: adding 0:100000 / 100000 update start search start Search finished successfully Write result successfully但是在9000000底库规模的数据集上,执行到update时报错:Load data set successfull. index start index init train start Training IVF quantizer on 9147767 vectors in 256D to 1024 cluster training computeCompressIndex on correlation matrix Sampling a subset of 524288 / 9147767 for training Clustering 524288 points in 256D to 1024 clusters, redo 1 times, 16 iterations Preprocessing in 3.41 s training computeCompressValue on 9147767 vectors in 256D Iteration 15 (114.84 s, search 114.24 s): objective=472094 imbalance=1.097 nsplit=0 training scalar quantizer on 9147767 vectors in 64D add start AscendIndex::add: adding 0:882984 / 9147767 AscendIndex::add: adding 882984:1765968 / 9147767 AscendIndex::add: adding 1765968:2648952 / 9147767 AscendIndex::add: adding 2648952:3531936 / 9147767 AscendIndex::add: adding 3531936:4414920 / 9147767 AscendIndex::add: adding 4414920:5297904 / 9147767 AscendIndex::add: adding 5297904:6180888 / 9147767 AscendIndex::add: adding 6180888:7063872 / 9147767 AscendIndex::add: adding 7063872:7946856 / 9147767 AscendIndex::add: adding 7946856:8829840 / 9147767 AscendIndex::add: adding 8829840:9147767 / 9147767 update start terminate called after throwing an instance of 'faiss::FaissException' what(): Error in void faiss::ascend::AscendClusteringImpl::subClusExecInt8(int, uint16_t*, float*, int, size_t) at ascend/custom/impl/AscendClusteringImpl.cpp:117: Error: 'ret == RPC_ERROR_NONE' failed: RpcIndexClusteringExecInt8 failed(1).以下是一些参数配置: const size_t dim = 256; const size_t dimOut = 64; const size_t nlist = 1024; const size_t k = 10; const size_t nprobe = 24; const size_t lowerBound = 1000; const size_t mergeThres = 1000; const float memoryLimit = 100000000; const size_t addTotal = 100000000;求大佬帮忙解答下~~
-
参照https://gitee.com/ascend/modelzoo-GPL/tree/master/built-in/ACL_Pytorch/Yolov5_for_Pytorch进行环境配置,执行pip3 install -r requirements.txt显示源也换了好几次,但是还是报相同错误,执行sudo python3 -m pip install torch===1.8.1 torchvision===0.9.1 -f https://download.pytorch.org/whl/torch_stable.html -i https://pypi.douban.com/simple依旧报相同错误。还请专家帮忙指点!
-
根据文档制作dd镜像https://bbs.huaweicloud.com/forum/thread-139685-1-1.html,使用几天后,再给200dk上电,就不能启动了。无法ssh连接到200dk。重新制作卡,设备又可以启动。无法查明问题。
-
华为云主机驱动从20.1.rc1升级到6.0.rc1,固件未升级(虚拟机无法安装固件),nnae、toolkit未安装,在22.0.RC3版本ascend-tensorflow镜像中进行推理,推理脚本来自于华为官方文档样例:https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/60RC1alpha003/moddevg/onlineinfer1/atlasoiug_26_0004.html,推理执行用户HwHiAiUser报错信息如下:WARNING:tensorflow:From infer_from_pb.py:195: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.########NOW Start Preprocess!!!#########WARNING:tensorflow:From infer_from_pb.py:75: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.2023-01-16 08:59:08.096267: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency2023-01-16 08:59:08.097916: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0xaaaad1c3f4f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:2023-01-16 08:59:08.097962: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default VersionWARNING:tensorflow:From infer_from_pb.py:79: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.Instructions for updating:Use tf.gfile.GFile.WARNING:tensorflow:From infer_from_pb.py:82: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.Instructions for updating:box_ind is deprecated, use box_indices instead########NOW Start Batch!!!#########WARNING:tensorflow:From infer_from_pb.py:106: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.WARNING:tensorflow:From infer_from_pb.py:133: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.WARNING:tensorflow:From infer_from_pb.py:134: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.########NOW Start inference!!!#########2023-01-16 08:59:09.008236: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.008535: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.008665: W tf_adapter/util/npu_attrs.cc:1077] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.008685: W tf_adapter/util/npu_attrs.cc:1207] The op_select_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-16 08:59:09.008694: W tf_adapter/util/npu_attrs.cc:1211] The optypelist_for_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-16 08:59:09.060358: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.060464: W tf_adapter/util/npu_attrs.cc:1077] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.060487: W tf_adapter/util/npu_attrs.cc:1207] The op_select_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-16 08:59:09.060503: W tf_adapter/util/npu_attrs.cc:1211] The optypelist_for_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-16 08:59:09.064144: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.064227: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.064315: W tf_adapter/util/npu_attrs.cc:1077] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.064335: W tf_adapter/util/npu_attrs.cc:1207] The op_select_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-16 08:59:09.064348: W tf_adapter/util/npu_attrs.cc:1211] The optypelist_for_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-16 08:59:09.064495: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.064573: W tf_adapter/util/npu_attrs.cc:1077] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.064590: W tf_adapter/util/npu_attrs.cc:1207] The op_select_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-16 08:59:09.064602: W tf_adapter/util/npu_attrs.cc:1211] The optypelist_for_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-16 08:59:09.064669: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-16 08:59:09.065074: W tf_adapter/util/npu_attrs.cc:148] [GePlugin] DEVICE_ID and ASCEND_DEVICE_ID is none, use default device id : 0, if set session_device_id, session_device_id has a higher priority2023-01-16 08:59:14.059804: I tf_adapter/kernels/geop_npu.cc:825] The model has been compiled on the Ascend AI processor, current graph id is: 1Traceback (most recent call last): File "/usr/local/python37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/usr/local/python37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/usr/local/python37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata)tensorflow.python.framework.errors_impl.InternalError: GeOp3_0GEOP::::DoRunAsync FailedError Message is : E49999: Inner Error!E49999 Compile op[trans_Cast_169,] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/cast.py], optype[Cast], taskID[5]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/dense/MatMul,[fp32_vars/dense/bias,fp32_vars/dense/bias,fp32_vars/dense/BiasAdd,fp32_vars/dense/MatMul]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/mat_mul.py], optype[MatMulV2], taskID[7]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[trans_TransData_163,] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/trans_data.py], optype[TransData], taskID[132]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_14/Conv2D,[fp32_vars/BatchNorm_14/FusedBatchNormV3,fp32_vars/BatchNorm_14/FusedBatchNormV3,fp32_vars/conv2d_14/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[136]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[138]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Mean,[fp32_vars/Mean/reduction_indices,fp32_vars/Mean/reduction_indices,fp32_vars/Mean]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/reduce_mean_d.py], optype[ReduceMeanD], taskID[133]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_1/Conv2D,[fp32_vars/BatchNorm_1/FusedBatchNormV3,fp32_vars/BatchNorm_1/FusedBatchNormV3,fp32_vars/conv2d_1/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[137]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_46/Conv2D,[fp32_vars/BatchNorm_46/FusedBatchNormV3,fp32_vars/BatchNorm_46/FusedBatchNormV3,fp32_vars/conv2d_46/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[134]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_27/Conv2D,[fp32_vars/BatchNorm_27/FusedBatchNormV3,fp32_vars/BatchNorm_27/FusedBatchNormV3,fp32_vars/conv2d_27/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[135]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[trans_TransData_168,] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/trans_data.py], optype[TransData], taskID[6]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[trans_Cast_0,] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/cast.py], optype[Cast], taskID[142]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[trans_TransData_164,] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/trans_data.py], optype[TransData], taskID[8]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464415437296] recompile single op[fp32_vars/dense/MatMul] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464415437296] recompile single op[trans_Cast_169] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464415437296] recompile single op[trans_TransData_168] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464415437296] recompile single op[trans_TransData_164] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ParalCompOp] Thread[281464415437296] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:867] [SubGraphOpt][Compile][CompOpOnly] CompileOp failed.[FUNC:CompileOpOnly][FILE:op_compiler.cc][LINE:994] [GraphOpt][FusedGraph][RunCompile] Failed to compile graph with compiler Normal mode Op Compiler[FUNC:SubGraphCompile][FILE:fe_graph_optimizer.cc][LINE:1467] Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank3_new_sub_graph110[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:130] Compile op[fp32_vars/conv2d/Conv2D,[fp32_vars/BatchNorm/FusedBatchNormV3,fp32_vars/BatchNorm/FusedBatchNormV3,fp32_vars/conv2d/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[140]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[trans_TransData_1,] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/trans_data.py], optype[TransData], taskID[141]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/MaxPoolWithArgmax] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/max_pool_with_argmax.py], optype[MaxPoolWithArgmax], taskID[139]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_46] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[202]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/add_15] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/add.py], optype[Add], taskID[196]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/add_12] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/add.py], optype[Add], taskID[212]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_48] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[197]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_39] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[213]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_50/Conv2D,[fp32_vars/BatchNorm_50/FusedBatchNormV3,fp32_vars/BatchNorm_50/FusedBatchNormV3,fp32_vars/conv2d_50/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[201]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_45/Conv2D,[fp32_vars/BatchNorm_45/FusedBatchNormV3,fp32_vars/BatchNorm_45/FusedBatchNormV3,fp32_vars/conv2d_45/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[205]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_22] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[227]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_40] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[229]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_38] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[231]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_25/Conv2D,[fp32_vars/BatchNorm_25/FusedBatchNormV3,fp32_vars/BatchNorm_25/FusedBatchNormV3,fp32_vars/conv2d_25/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[226]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_44/Conv2D,[fp32_vars/BatchNorm_44/FusedBatchNormV3,fp32_vars/BatchNorm_44/FusedBatchNormV3,fp32_vars/conv2d_44/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[228]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/add] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/add.py], optype[Add], taskID[248]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_3] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[249]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/add_13] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/add.py], optype[Add], taskID[252]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_40/Conv2D,[fp32_vars/BatchNorm_40/FusedBatchNormV3,fp32_vars/BatchNorm_40/FusedBatchNormV3,fp32_vars/conv2d_40/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[243]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_51/Conv2D,[fp32_vars/BatchNorm_51/FusedBatchNormV3,fp32_vars/BatchNorm_51/FusedBatchNormV3,fp32_vars/conv2d_51/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[207]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_41/Conv2D,[fp32_vars/BatchNorm_41/FusedBatchNormV3,fp32_vars/BatchNorm_41/FusedBatchNormV3,fp32_vars/conv2d_41/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[230]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_43/Conv2D,[fp32_vars/BatchNorm_43/FusedBatchNormV3,fp32_vars/BatchNorm_43/FusedBatchNormV3,fp32_vars/conv2d_43/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[251]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_26/Conv2D,[fp32_vars/BatchNorm_26/FusedBatchNormV3,fp32_vars/BatchNorm_26/FusedBatchNormV3,fp32_vars/conv2d_26/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[241]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/add_3] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/add.py], optype[Add], taskID[257]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/add_7] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/add.py], optype[Add], taskID[260]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_12] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[258]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_2/Conv2D,[fp32_vars/BatchNorm_2/FusedBatchNormV3,fp32_vars/BatchNorm_2/FusedBatchNormV3,fp32_vars/conv2d_2/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[254]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_10] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[279]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/add_1] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/add.py], optype[Add], taskID[282]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_24/Conv2D,[fp32_vars/BatchNorm_24/FusedBatchNormV3,fp32_vars/BatchNorm_24/FusedBatchNormV3,fp32_vars/conv2d_24/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[259]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_12/Conv2D,[fp32_vars/BatchNorm_12/FusedBatchNormV3,fp32_vars/BatchNorm_12/FusedBatchNormV3,fp32_vars/conv2d_12/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[278]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_11/Conv2D,[fp32_vars/BatchNorm_11/FusedBatchNormV3,fp32_vars/BatchNorm_11/FusedBatchNormV3,fp32_vars/conv2d_11/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[256]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/Relu_13] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/relu.py], optype[Relu], taskID[294]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_8/Conv2D,[fp32_vars/BatchNorm_8/FusedBatchNormV3,fp32_vars/BatchNorm_8/FusedBatchNormV3,fp32_vars/conv2d_8/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[263]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/add_4] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/add.py], optype[Add], taskID[308]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_15/Conv2D,[fp32_vars/BatchNorm_15/FusedBatchNormV3,fp32_vars/BatchNorm_15/FusedBatchNormV3,fp32_vars/conv2d_15/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[293]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_3/Conv2D,[fp32_vars/BatchNorm_3/FusedBatchNormV3,fp32_vars/BatchNorm_3/FusedBatchNormV3,fp32_vars/conv2d_3/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[287]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_13/Conv2D,[fp32_vars/BatchNorm_13/FusedBatchNormV3,fp32_vars/BatchNorm_13/FusedBatchNormV3,fp32_vars/conv2d_13/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[299]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] Compile op[fp32_vars/conv2d_16/Conv2D,[fp32_vars/BatchNorm_16/FusedBatchNormV3,fp32_vars/BatchNorm_16/FusedBatchNormV3,fp32_vars/conv2d_16/Conv2D]] failed, oppath[/usr/local/Ascend/nnae/6.0.RC1/opp/op_impl/built-in/ai_core/tbe/impl/conv2d.py], optype[Conv2D], taskID[301]. Please check op's compilation error message.[FUNC:ReportBuildErrMessage][FILE:fusion_op.cc][LINE:856] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_22/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_19/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_16/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_13/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_9/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_6/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_3/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_21/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_18/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_15/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_6] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_5] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_4] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_5/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_8/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_20] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_17] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_14] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_11] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_19] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_16] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_13] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_11/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_12/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_24/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_2] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_10] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_2/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_21] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_18] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_15] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_12] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_7] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_3] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_26/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_43/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_29/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_32/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_35/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_38/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_41/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_48/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_51/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_28/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_52/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_31] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_49/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_28] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_42/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_5] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_39/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_36/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_33/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_30/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/MaxPoolWithArgmax] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_4/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[trans_Cast_0] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_7] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_27/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_14/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_4] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[trans_TransData_163] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Mean] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_1/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_46/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_15] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[trans_TransData_1] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_50/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_7/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_46] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_10/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_45/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_2] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_8] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_39] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_17/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_44/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_20/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_38] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_23/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_43] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_6] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_41] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_47] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_44] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_14] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_12] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_11] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_10] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_9] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_8] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_48] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_31/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_45] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_42] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_36] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_33] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_30] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_27] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_24] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_47/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_22] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_40] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_35] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_32] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_29] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_26] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_23] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_37] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_34] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_25] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_25/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_3] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/Relu_9] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/add_13] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_40/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_37/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ProcFailedCompTask] Thread[281464432222704] recompile single op[fp32_vars/conv2d_34/Conv2D] failed[FUNC:ProcessAllFailedCompileTasks][FILE:tbe_op_store_adapter.cc][LINE:827] [SubGraphOpt][Compile][ParalCompOp] Thread[281464432222704] process fail task failed[FUNC:ParallelCompileOp][FILE:tbe_op_store_adapter.cc][LINE:867] Call OptimizeFusedGraph failed, ret:-1, engine_name:AIcoreEngine, graph_name:partition0_rank1_new_sub_graph1[FUNC:OptimizeSubGraph][FILE:graph_optimize.cc][LINE:130] subgraph 0 optimize failed[FUNC:OptimizeSubGraphWithMultiThreads][FILE:graph_manager.cc][LINE:714] Solution: Please contact support engineer. [[{{node GeOp3_0}}]]使用root用户进行推理,无报错
-
一、问题现象(附报错日志上下文):[EVENT] FMK(2277,python3):2023-01-14-03:04:11.647.742 [tf_adapter/common/adp_logger.cc:27][TF_ADAPTER] [tf_adapter/optimizers/om_partition_subgraphs_pass.cc:2136] OMPartition subgraph_3 begin. mix_compile_mode is False, iterations_per_loop is 1, input_shape: , dynamic_dims:[EVENT] FMK(2277,python3):2023-01-14-03:04:11.691.310 [tf_adapter/common/adp_logger.cc:27][TF_ADAPTER] [tf_adapter/optimizers/om_partition_subgraphs_pass.cc:2247] OMPartition subgraph_3 markForPartition success.[EVENT] FMK(2277,python3):2023-01-14-03:04:11.719.649 [tf_adapter/common/adp_logger.cc:27][TF_ADAPTER] [tf_adapter/optimizers/om_partition_subgraphs_pass.cc:2260] OMPartition subgraph_3 SubgraphsInFunctions succeed.[EVENT] FMK(2277,python3):2023-01-14-03:04:11.719.746 [tf_adapter/common/adp_logger.cc:27][TF_ADAPTER] [tf_adapter/optimizers/om_partition_subgraphs_pass.cc:2271] OMPartition subgraph_3 succeed. [72 ms]2023-01-14 03:04:11.719841: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-14 03:04:11.719925: W tf_adapter/util/npu_attrs.cc:1077] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-14 03:04:11.719943: W tf_adapter/util/npu_attrs.cc:1207] The op_select_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-14 03:04:11.719955: W tf_adapter/util/npu_attrs.cc:1211] The optypelist_for_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-14 03:04:11.723206: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-14 03:04:11.723289: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-14 03:04:11.723375: W tf_adapter/util/npu_attrs.cc:1077] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-14 03:04:11.723392: W tf_adapter/util/npu_attrs.cc:1207] The op_select_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-14 03:04:11.723406: W tf_adapter/util/npu_attrs.cc:1211] The optypelist_for_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-14 03:04:11.723567: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-14 03:04:11.723643: W tf_adapter/util/npu_attrs.cc:1077] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-14 03:04:11.723657: W tf_adapter/util/npu_attrs.cc:1207] The op_select_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-14 03:04:11.723668: W tf_adapter/util/npu_attrs.cc:1211] The optypelist_for_implmode option IS DEPRECATED. It will be removed in a future version. Please use op_precision_mode instead2023-01-14 03:04:11.723742: W tf_adapter/util/npu_attrs.cc:861] The enable_data_pre_proc option IS DEPRECATED. It will be removed in a future version.2023-01-14 03:04:11.724170: W tf_adapter/util/npu_attrs.cc:148] [GePlugin] DEVICE_ID and ASCEND_DEVICE_ID is none, use default device id : 0, if set session_device_id, session_device_id has a higher priority[ERROR] TDT(2277,python3):2023-01-14-03:06:11.740.978 [log.cpp:143]deviceId=0: drvtype=8 drv hdc server create failed, ret = 31,[hdc_server.cpp:136:Init]2277 Msg: hdc server create error[ERROR] TDT(2277,python3):2023-01-14-03:06:11.741.025 [log.cpp:143]TuningDataReceive hdc server init failed, deviceId=0,[tuning_data_receive.cpp:81:InitDirectly]2277 Msg: call member function before instance initialed[ERROR] TDT(2277,python3):2023-01-14-03:06:11.741.051 [log.cpp:143]TuningDataReceive InitDirectly failed, logicId=0,[tuning_data_receive.cpp:102:Init]2277 Msg: hdc server create error[ERROR] TDT(2277,python3):2023-01-14-03:06:11.741.073 [log.cpp:143]log summary receive init failed,[tdt_host.cpp:683:TdtOutFeedInit]2277 Msg: hdc server create error[TRACE] GE(2277,python3):2023-01-14-03:06:11.742.422 [status:INIT] [ge_api.cc:194]2277 GEInitializeImpl:GEInitialize start[EVENT] PROFILING(2277,python3):2023-01-14-03:06:11.770.387 [msprof_callback_impl.cpp:78] >>> (tid:2277) MsprofCtrlCallback called, type: 255[EVENT] PROFILING(2277,python3):2023-01-14-03:06:11.770.491 [prof_acl_mgr.cpp:1190] >>> (tid:2277) Init profiling for dynamic profiling[TRACE] GE(2277,python3):2023-01-14-03:06:11.803.836 [status:RUNNING] [ge_api.cc:248]2277 GEInitializeImpl:Initializing environment[EVENT] GE(2277,python3):2023-01-14-03:06:11.803.914 [gelib.cc:55]2277 Initialize:[GEPERFTRACE] GE Init Start[ERROR] RUNTIME(2277,python3):2023-01-14-03:06:11.804.931 [runtime.cc:1051]2277 CheckHaveDevice:[INIT][SYS]Call halGetDeviceInfo failed: drvRet=4, module type=0, info type=1.[ERROR] RUNTIME(2277,python3):2023-01-14-03:06:11.805.800 [runtime.cc:1003]2277 InitSocType:[INIT][SYS]Call halGetDeviceInfo failed: drvRetCode=4, module type=0, info type=1.[ERROR] RUNTIME(2277,python3):2023-01-14-03:06:11.805.835 [runtime.cc:1075]2277 Init:[INIT][SYS]Init SocType failed.[ERROR] RUNTIME(2277,python3):2023-01-14-03:06:11.805.892 [api_c.cc:2967]2277 rtGetSocVersion:[INIT][SYS]ErrCode=507000, desc=[instance pointer null], InnerCode=0x70b0001[ERROR] GE(2277,python3):2023-01-14-03:06:11.805.925 [gelib.cc:276]2277 SetRTSocVersion: ErrorNo: 507000() [INIT][SYS][Get][SocVersion]rtGetSocVersion failed[ERROR] GE(2277,python3):2023-01-14-03:06:11.805.947 [gelib.cc:68]2277 Initialize: ErrorNo: 4294967295(failed) [INIT][SYS][Set][RTSocVersion]GeLib initial: SetRTSocVersion failed.[ERROR] GE(2277,python3):2023-01-14-03:06:11.805.985 [ge_api.cc:254]2277 GEInitializeImpl: ErrorNo: 1343229953(GEInitialize Failed.) [INIT][SYS][Init][GELib]Failed, error code = 4294967295[ERROR] FMK(2277,python3):2023-01-14-03:06:14.806.114 [tf_adapter/common/adp_logger.cc:24][TF_ADAPTER] [tf_adapter/util/ge_plugin.cc:251] [GePlugin] Initialize ge failed, ret : failed2023-01-14 03:06:14.807266: F tf_adapter/util/ge_plugin.cc:253] [GePlugin] Initialize ge failed, ret : failedError Message is :EL0003: The argument is invalid.Solution: Try again with a valid argument.TraceBack (most recent call last):Check param failed, rtInstance can not be NULL![FUNC:rtGetSocVersion][FILE:api_c.cc][LINE:2967]rtGetSocVersion failed.[FUNC:SetRTSocVersion][FILE:gelib.cc][LINE:276]SetRTSocVersion failed.[FUNC:Initialize][FILE:gelib.cc][LINE:69]Aborted (core dumped)二、软件版本:-- CANN 版本 (e.g., CANN 3.0.x,5.x.x): CANN 6.0.rc1--Tensorflow/Pytorch/MindSpore 版本: Tensorflow 1.15.0--Python 版本 (e.g., Python 3.7.5): Python 3.7.5--操作系统版本 (e.g., Ubuntu 18.04): Ubuntu 18.04使用ascend-tensorflow 22.0.RC3镜像请问有人遇到过类似问题吗?
-
网上查到的资料显示要想在昇腾服务器上使用cuda训练的pytorch模型,需要将模型转为om格式,但是没找到推理脚本改造的文档,请问在昇腾服务器上用pytorch推理有没有官方资料可以学习?另外还有TensorFlow的,请大神们指点,谢谢
-
使用 MindStudio 将一个在 GPU 正常运行的项目迁移至 NPU,AICore 的利用率一直上不去。在平时的 RTX 3090 显卡训练,GPU的利用率能够一直维持在 99% 左右,训练时长大约为一个小时。但是,在 NPU 上一直不出结果,训练推进缓慢。是不是我训练的设置有问题?还是项目相应的 API 需要修改?
-
目前查到的资料都是将推理脚本部署到基础镜像内进行测试,是否可以在docker外进行推理?
上滑加载中
推荐直播
-
TinyEngine低代码引擎系列.第1讲——低代码浪潮之下,带你走进TinyEngine
2024/11/11 周一 16:00-18:00
李老师 高级前端开发工程师
低代码浪潮之下,带你走进TinyEngine。李旭宏老师将从低代码的发展趋势、TinyEngine的项目介绍,三方物料组件的使用、跨技术栈的使用、源码生成能力的差异性对比等多个方面带大家对TinyEngine低代码引擎有一个更清晰的认知和了解。
即将直播 -
0代码智能构建AI Agent——华为云AI原生应用引擎的架构与实践
2024/11/13 周三 16:30-18:00
苏秦 华为云aPaaS DTSE技术布道师
大模型及生成式AI对应用和软件产业带来了哪些影响?从企业场景及应用开发视角,面向AI原生应用需要什么样的工具及平台能力?企业要如何选好、用好、管好大模型,使能AI原生应用快速创新?本期直播,华为云aPaaS DTSE技术布道师苏秦将基于华为云自身实践出发,深入浅出地介绍华为云AI原生应用引擎,通过分钟级智能生成Agent应用的方式帮助企业完成从传统应用到智能应用的竞争力转型,使能千行万业智能应用创新。
去报名
热门标签