- 资产集市
- 教学
- 实践
- AI说
- 案例库
- 生态合作
- 专区
中国站
简体中文本案例采用TensorFlow object_detection框架与疲劳/分神驾驶样例数据集数据集实现一个简单的人脸手机检测模型训练。
注意:本案例必须使用GPU运行,请查看《ModelArts JupyterLab 硬件规格使用指南》了解切换硬件规格的方法
import os
import moxing as mox
INFO:root:Using MoXing-v2.1.0.5d9c87c8-5d9c87c8 INFO:root:Using OBS-Python-SDK-3.20.9.1
if not os.path.exists("object_detection.zip"):
mox.file.copy("obs://houyansong/object_detection.zip", "object_detection.zip")
if not os.path.exists("object_detection"):
os.system("unzip -q object_detection.zip")
首先通过conda创建一个python3.6的虚拟环境py36:
!/home/ma-user/anaconda3/bin/conda create -n py36 python=3.6.10 -y
/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Collecting package metadata (current_repodata.json): done Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source. Collecting package metadata (repodata.json): done Solving environment: done ==> WARNING: A newer version of conda exists. <== current version: 4.8.2 latest version: 23.3.1 Please update conda by running $ conda update -n base -c defaults conda ## Package Plan ## environment location: /home/ma-user/anaconda3/envs/py36 added / updated specs: - python=3.6.10 The following packages will be downloaded: package | build ---------------------------|----------------- _libgcc_mutex-0.1 | main 3 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main _openmp_mutex-5.1 | 1_gnu 21 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ca-certificates-2023.01.10 | h06a4308_0 120 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main certifi-2021.5.30 | py36h06a4308_0 139 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ld_impl_linux-64-2.38 | h1181459_1 654 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libffi-3.3 | he6710b0_2 50 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgcc-ng-11.2.0 | h1234567_1 5.3 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgomp-11.2.0 | h1234567_1 474 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libstdcxx-ng-11.2.0 | h1234567_1 4.7 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ncurses-6.4 | h6a678d5_0 914 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main openssl-1.1.1t | h7f8727e_0 3.7 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main pip-21.2.2 | py36h06a4308_0 1.8 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main python-3.6.10 | h7579374_2 29.7 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main readline-8.2 | h5eee18b_0 357 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main setuptools-58.0.4 | py36h06a4308_0 788 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main sqlite-3.41.2 | h5eee18b_0 1.2 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tk-8.6.12 | h1ccaba5_0 3.0 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main wheel-0.37.1 | pyhd3eb1b0_0 33 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main xz-5.2.10 | h5eee18b_1 429 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main zlib-1.2.13 | h5eee18b_0 103 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ------------------------------------------------------------ Total: 53.5 MB The following NEW packages will be INSTALLED: _libgcc_mutex anaconda/pkgs/main/linux-64::_libgcc_mutex-0.1-main _openmp_mutex anaconda/pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu ca-certificates anaconda/pkgs/main/linux-64::ca-certificates-2023.01.10-h06a4308_0 certifi anaconda/pkgs/main/linux-64::certifi-2021.5.30-py36h06a4308_0 ld_impl_linux-64 anaconda/pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1 libffi anaconda/pkgs/main/linux-64::libffi-3.3-he6710b0_2 libgcc-ng anaconda/pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1 libgomp anaconda/pkgs/main/linux-64::libgomp-11.2.0-h1234567_1 libstdcxx-ng anaconda/pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1 ncurses anaconda/pkgs/main/linux-64::ncurses-6.4-h6a678d5_0 openssl anaconda/pkgs/main/linux-64::openssl-1.1.1t-h7f8727e_0 pip anaconda/pkgs/main/linux-64::pip-21.2.2-py36h06a4308_0 python anaconda/pkgs/main/linux-64::python-3.6.10-h7579374_2 readline anaconda/pkgs/main/linux-64::readline-8.2-h5eee18b_0 setuptools anaconda/pkgs/main/linux-64::setuptools-58.0.4-py36h06a4308_0 sqlite anaconda/pkgs/main/linux-64::sqlite-3.41.2-h5eee18b_0 tk anaconda/pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0 wheel anaconda/pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0 xz anaconda/pkgs/main/linux-64::xz-5.2.10-h5eee18b_1 zlib anaconda/pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0 Downloading and Extracting Packages ca-certificates-2023 | 120 KB | ##################################### | 100% python-3.6.10 | 29.7 MB | ##################################### | 100% wheel-0.37.1 | 33 KB | ##################################### | 100% tk-8.6.12 | 3.0 MB | ##################################### | 100% ld_impl_linux-64-2.3 | 654 KB | ##################################### | 100% zlib-1.2.13 | 103 KB | ##################################### | 100% certifi-2021.5.30 | 139 KB | ##################################### | 100% pip-21.2.2 | 1.8 MB | ##################################### | 100% libgcc-ng-11.2.0 | 5.3 MB | ##################################### | 100% libffi-3.3 | 50 KB | ##################################### | 100% readline-8.2 | 357 KB | ##################################### | 100% xz-5.2.10 | 429 KB | ##################################### | 100% libgomp-11.2.0 | 474 KB | ##################################### | 100% _libgcc_mutex-0.1 | 3 KB | ##################################### | 100% libstdcxx-ng-11.2.0 | 4.7 MB | ##################################### | 100% sqlite-3.41.2 | 1.2 MB | ##################################### | 100% setuptools-58.0.4 | 788 KB | ##################################### | 100% _openmp_mutex-5.1 | 21 KB | ##################################### | 100% openssl-1.1.1t | 3.7 MB | ##################################### | 100% ncurses-6.4 | 914 KB | ##################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done # # To activate this environment, use # # $ conda activate py36 # # To deactivate an active environment, use # # $ conda deactivate
接下来安装依赖包:
!/home/ma-user/anaconda3/envs/py36/bin/pip install ipykernel
Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple Collecting ipykernel Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/e9/ad/9101e0ab5e84dd117462bb3a1379d31728a849b6886458452e3d97dc6bba/ipykernel-5.5.6-py3-none-any.whl (121 kB) |████████████████████████████████| 121 kB 82.5 MB/s eta 0:00:01 Collecting ipython-genutils Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB) Collecting tornado>=4.2 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/85/26/e710295dcb4aac62b08f22d07efc899574476db37532159a7f71713cdaf2/tornado-6.1-cp36-cp36m-manylinux2010_x86_64.whl (427 kB) |████████████████████████████████| 427 kB 83.2 MB/s eta 0:00:01 Collecting jupyter-client Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/56/a7/f4d3790ce7bb925d3ffe299244501a264f23ee7ec401914f7d788881ea31/jupyter_client-7.1.2-py3-none-any.whl (130 kB) |████████████████████████████████| 130 kB 14.9 MB/s eta 0:00:01 Collecting traitlets>=4.1.0 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ca/ab/872a23e29cec3cf2594af7e857f18b687ad21039c1f9b922fac5b9b142d5/traitlets-4.3.3-py2.py3-none-any.whl (75 kB) |████████████████████████████████| 75 kB 67.6 MB/s eta 0:00:01 Collecting ipython>=5.0.0 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/53/09/958a4802489d28b2484114ee6414c7502ef57de6f2dbc9095b718640060c/ipython-7.16.3-py3-none-any.whl (783 kB) |████████████████████████████████| 783 kB 15.5 MB/s eta 0:00:01 Collecting jedi<=0.17.2,>=0.10 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/c3/d4/36136b18daae06ad798966735f6c3fb96869c1be9f8245d2a8f556e40c36/jedi-0.17.2-py2.py3-none-any.whl (1.4 MB) |████████████████████████████████| 1.4 MB 79.5 MB/s eta 0:00:01kB 79.5 MB/s eta 0:00:01 Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/58/87/cac418cef18781a9081cb2075cc2cf08c77e0679c1f9b474587d71bbf777/prompt_toolkit-3.0.33-py3-none-any.whl (383 kB) |████████████████████████████████| 383 kB 16.0 MB/s eta 0:00:01 Collecting pexpect Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/39/7b/88dbb785881c28a102619d46423cb853b46dbccc70d3ac362d99773a78ce/pexpect-4.8.0-py2.py3-none-any.whl (59 kB) |████████████████████████████████| 59 kB 25.9 MB/s eta 0:00:01 Requirement already satisfied: setuptools>=18.5 in /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages (from ipython>=5.0.0->ipykernel) (58.0.4) Collecting backcall Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/4c/1c/ff6546b6c12603d8dd1070aa3c3d273ad4c07f5771689a7b69a550e8c951/backcall-0.2.0-py2.py3-none-any.whl (11 kB) Collecting pickleshare Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/9a/41/220f49aaea88bc6fa6cba8d05ecf24676326156c23b991e80b3f2fc24c77/pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB) Collecting pygments Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/0b/42/d9d95cc461f098f204cd20c85642ae40fbff81f74c300341b8d0e0df14e0/Pygments-2.14.0-py3-none-any.whl (1.1 MB) |████████████████████████████████| 1.1 MB 19.0 MB/s eta 0:00:01 Collecting decorator Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/d5/50/83c593b07763e1161326b3b8c6686f0f4b0f24d5526546bee538c89837d6/decorator-5.1.1-py3-none-any.whl (9.1 kB) Collecting parso<0.8.0,>=0.7.0 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/93/d1/e635bdde32890db5aeb2ffbde17e74f68986305a4466b0aa373b861e3f00/parso-0.7.1-py2.py3-none-any.whl (109 kB) |████████████████████████████████| 109 kB 87.7 MB/s eta 0:00:01 Collecting wcwidth Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/59/7c/e39aca596badaf1b78e8f547c807b04dae603a433d3e7a7e04d67f2ef3e5/wcwidth-0.2.5-py2.py3-none-any.whl (30 kB) Collecting six Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting jupyter-core>=4.6.0 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/60/7d/bee50351fe3ff6979e949b9c4c00c556a7a9732ba39b547d07d93450de23/jupyter_core-4.9.2-py3-none-any.whl (86 kB) |████████████████████████████████| 86 kB 17.3 MB/s eta 0:00:01 Collecting entrypoints Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/35/a8/365059bbcd4572cbc41de17fd5b682be5868b218c3c5479071865cab9078/entrypoints-0.4-py3-none-any.whl (5.3 kB) Collecting nest-asyncio>=1.5 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/06/e0/93453ebab12f5ce9a9ceda2ff71648b30e5f2ce5bba19ee3c95cbd0aaa67/nest_asyncio-1.5.4-py3-none-any.whl (5.1 kB) Collecting pyzmq>=13 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6c/33/d68be963793bced811ed51903eacb719ed40387d31bf2dd7abf409390107/pyzmq-24.0.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.1 MB) |████████████████████████████████| 1.1 MB 55.4 MB/s eta 0:00:01 Collecting python-dateutil>=2.1 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) |████████████████████████████████| 247 kB 21.1 MB/s eta 0:00:01 Collecting ptyprocess>=0.5 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB) Installing collected packages: six, ipython-genutils, decorator, wcwidth, traitlets, ptyprocess, parso, tornado, pyzmq, python-dateutil, pygments, prompt-toolkit, pickleshare, pexpect, nest-asyncio, jupyter-core, jedi, entrypoints, backcall, jupyter-client, ipython, ipykernel Successfully installed backcall-0.2.0 decorator-5.1.1 entrypoints-0.4 ipykernel-5.5.6 ipython-7.16.3 ipython-genutils-0.2.0 jedi-0.17.2 jupyter-client-7.1.2 jupyter-core-4.9.2 nest-asyncio-1.5.4 parso-0.7.1 pexpect-4.8.0 pickleshare-0.7.5 prompt-toolkit-3.0.33 ptyprocess-0.7.0 pygments-2.14.0 python-dateutil-2.8.2 pyzmq-24.0.1 six-1.16.0 tornado-6.1 traitlets-4.3.3 wcwidth-0.2.5
添加kernel配置文件,使虚拟环境可以在notebook中被识别:
import json
import os
data = {
"display_name": "Python36",
"env": {
"PATH": "/home/ma-user/anaconda3/envs/py36/bin:/home/ma-user/anaconda3/envs/python-3.7.10/bin:/modelarts/authoring/notebook-conda/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/anaconda3/envs/PyTorch-1.4/bin"
},
"language": "python",
"argv": [
"/home/ma-user/anaconda3/envs/py36/bin/python",
"-m",
"ipykernel",
"-f",
"{connection_file}"
]
}
if not os.path.exists("/home/ma-user/anaconda3/share/jupyter/kernels/py36/"):
os.mkdir("/home/ma-user/anaconda3/share/jupyter/kernels/py36/")
with open('/home/ma-user/anaconda3/share/jupyter/kernels/py36/kernel.json', 'w') as f:
json.dump(data, f, indent=4)
conda env list
/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) # conda environments: # base * /home/ma-user/anaconda3 TensorFlow-2.1 /home/ma-user/anaconda3/envs/TensorFlow-2.1 py36 /home/ma-user/anaconda3/envs/py36 python-3.7.10 /home/ma-user/anaconda3/envs/python-3.7.10 Note: you may need to restart the kernel to use updated packages.
选择该kernel(py36),验证一下python版本和pip版本:
!python -V
Python 3.6.10 :: Anaconda, Inc.
!pip -V
pip 21.2.2 from /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/pip (python 3.6)
安装TensorFlow
conda install tensorflow-gpu==1.15 -y
/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Collecting package metadata (current_repodata.json): done Solving environment: failed with initial frozen solve. Retrying with flexible solve. Collecting package metadata (repodata.json): done Solving environment: \ The environment is inconsistent, please check the package plan carefully The following packages are causing the inconsistency: - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::libstdcxx-ng==11.2.0=h1234567_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::libgcc-ng==11.2.0=h1234567_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::libffi==3.3=he6710b0_2 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::ncurses==6.4=h6a678d5_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::openssl==1.1.1t=h7f8727e_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::xz==5.2.10=h5eee18b_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::zlib==1.2.13=h5eee18b_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::readline==8.2=h5eee18b_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::tk==8.6.12=h1ccaba5_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::sqlite==3.41.2=h5eee18b_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::python==3.6.10=h7579374_2 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::certifi==2021.5.30=py36h06a4308_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::wheel==0.37.1=pyhd3eb1b0_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::setuptools==58.0.4=py36h06a4308_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::pip==21.2.2=py36h06a4308_0 done ==> WARNING: A newer version of conda exists. <== current version: 4.8.2 latest version: 23.3.1 Please update conda by running $ conda update -n base -c defaults conda ## Package Plan ## environment location: /home/ma-user/anaconda3/envs/py36 added / updated specs: - tensorflow-gpu==1.15 The following packages will be downloaded: package | build ---------------------------|----------------- _tflow_select-2.1.0 | gpu 2 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main absl-py-0.15.0 | pyhd3eb1b0_0 103 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main astor-0.8.1 | py36h06a4308_0 47 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main blas-1.0 | mkl 6 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main c-ares-1.19.0 | h5eee18b_0 118 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cudatoolkit-10.0.130 | 0 261.2 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cudnn-7.6.5 | cuda10.0_0 165.0 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main cupti-10.0.130 | 0 1.5 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main gast-0.2.2 | py36_0 155 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main google-pasta-0.2.0 | pyhd3eb1b0_0 46 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main grpcio-1.31.0 | py36hf8bcb03_0 2.0 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main h5py-2.10.0 | py36hd6299e0_1 901 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main hdf5-1.10.6 | hb1b8bf9_0 3.7 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main intel-openmp-2022.1.0 | h9e868ea_3769 4.5 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main keras-applications-1.0.8 | py_1 29 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main keras-preprocessing-1.1.2 | pyhd3eb1b0_0 35 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgfortran-ng-7.5.0 | ha8ba4b0_17 22 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libgfortran4-7.5.0 | ha8ba4b0_17 995 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main libprotobuf-3.17.2 | h4ff587b_1 2.0 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main markdown-3.1.1 | py36_0 116 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl-2020.2 | 256 138.3 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl-service-2.3.0 | py36he8ac12f_0 52 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_fft-1.3.0 | py36h54f3939_0 170 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main mkl_random-1.1.1 | py36h0573a6f_0 327 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main numpy-1.19.2 | py36h54aff64_0 22 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main numpy-base-1.19.2 | py36hfa32c7d_0 4.1 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main opt_einsum-3.3.0 | pyhd3eb1b0_1 57 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main protobuf-3.17.2 | py36h295c915_0 319 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main scipy-1.5.2 | py36h0b6359f_0 14.4 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main six-1.16.0 | pyhd3eb1b0_1 18 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tensorboard-1.15.0 | pyhb230dea_0 3.2 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tensorflow-1.15.0 |gpu_py36h5a509aa_0 4 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tensorflow-base-1.15.0 |gpu_py36h9dcbed7_0 156.5 MB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tensorflow-estimator-1.15.1| pyh2649769_0 271 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main tensorflow-gpu-1.15.0 | h0d30ee6_0 3 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main termcolor-1.1.0 | py36h06a4308_1 9 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main webencodings-0.5.1 | py36_1 19 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main werkzeug-0.16.1 | py_0 258 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main wrapt-1.12.1 | py36h7b6447c_1 49 KB https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main ------------------------------------------------------------ Total: 760.6 MB The following NEW packages will be INSTALLED: _tflow_select anaconda/pkgs/main/linux-64::_tflow_select-2.1.0-gpu absl-py anaconda/pkgs/main/noarch::absl-py-0.15.0-pyhd3eb1b0_0 astor anaconda/pkgs/main/linux-64::astor-0.8.1-py36h06a4308_0 blas anaconda/pkgs/main/linux-64::blas-1.0-mkl c-ares anaconda/pkgs/main/linux-64::c-ares-1.19.0-h5eee18b_0 cudatoolkit anaconda/pkgs/main/linux-64::cudatoolkit-10.0.130-0 cudnn anaconda/pkgs/main/linux-64::cudnn-7.6.5-cuda10.0_0 cupti anaconda/pkgs/main/linux-64::cupti-10.0.130-0 gast anaconda/pkgs/main/linux-64::gast-0.2.2-py36_0 google-pasta anaconda/pkgs/main/noarch::google-pasta-0.2.0-pyhd3eb1b0_0 grpcio anaconda/pkgs/main/linux-64::grpcio-1.31.0-py36hf8bcb03_0 h5py anaconda/pkgs/main/linux-64::h5py-2.10.0-py36hd6299e0_1 hdf5 anaconda/pkgs/main/linux-64::hdf5-1.10.6-hb1b8bf9_0 intel-openmp anaconda/pkgs/main/linux-64::intel-openmp-2022.1.0-h9e868ea_3769 keras-applications anaconda/pkgs/main/noarch::keras-applications-1.0.8-py_1 keras-preprocessi~ anaconda/pkgs/main/noarch::keras-preprocessing-1.1.2-pyhd3eb1b0_0 libgfortran-ng anaconda/pkgs/main/linux-64::libgfortran-ng-7.5.0-ha8ba4b0_17 libgfortran4 anaconda/pkgs/main/linux-64::libgfortran4-7.5.0-ha8ba4b0_17 libprotobuf anaconda/pkgs/main/linux-64::libprotobuf-3.17.2-h4ff587b_1 markdown anaconda/pkgs/main/linux-64::markdown-3.1.1-py36_0 mkl anaconda/pkgs/main/linux-64::mkl-2020.2-256 mkl-service anaconda/pkgs/main/linux-64::mkl-service-2.3.0-py36he8ac12f_0 mkl_fft anaconda/pkgs/main/linux-64::mkl_fft-1.3.0-py36h54f3939_0 mkl_random anaconda/pkgs/main/linux-64::mkl_random-1.1.1-py36h0573a6f_0 numpy anaconda/pkgs/main/linux-64::numpy-1.19.2-py36h54aff64_0 numpy-base anaconda/pkgs/main/linux-64::numpy-base-1.19.2-py36hfa32c7d_0 opt_einsum anaconda/pkgs/main/noarch::opt_einsum-3.3.0-pyhd3eb1b0_1 protobuf anaconda/pkgs/main/linux-64::protobuf-3.17.2-py36h295c915_0 scipy anaconda/pkgs/main/linux-64::scipy-1.5.2-py36h0b6359f_0 six anaconda/pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1 tensorboard anaconda/pkgs/main/noarch::tensorboard-1.15.0-pyhb230dea_0 tensorflow anaconda/pkgs/main/linux-64::tensorflow-1.15.0-gpu_py36h5a509aa_0 tensorflow-base anaconda/pkgs/main/linux-64::tensorflow-base-1.15.0-gpu_py36h9dcbed7_0 tensorflow-estima~ anaconda/pkgs/main/noarch::tensorflow-estimator-1.15.1-pyh2649769_0 tensorflow-gpu anaconda/pkgs/main/linux-64::tensorflow-gpu-1.15.0-h0d30ee6_0 termcolor anaconda/pkgs/main/linux-64::termcolor-1.1.0-py36h06a4308_1 webencodings anaconda/pkgs/main/linux-64::webencodings-0.5.1-py36_1 werkzeug anaconda/pkgs/main/noarch::werkzeug-0.16.1-py_0 wrapt anaconda/pkgs/main/linux-64::wrapt-1.12.1-py36h7b6447c_1 Downloading and Extracting Packages c-ares-1.19.0 | 118 KB | ##################################### | 100% grpcio-1.31.0 | 2.0 MB | ##################################### | 100% mkl_fft-1.3.0 | 170 KB | ##################################### | 100% tensorflow-gpu-1.15. | 3 KB | ##################################### | 100% mkl-service-2.3.0 | 52 KB | ##################################### | 100% keras-preprocessing- | 35 KB | ##################################### | 100% cudnn-7.6.5 | 165.0 MB | ##################################### | 100% webencodings-0.5.1 | 19 KB | ##################################### | 100% protobuf-3.17.2 | 319 KB | ##################################### | 100% blas-1.0 | 6 KB | ##################################### | 100% h5py-2.10.0 | 901 KB | ##################################### | 100% wrapt-1.12.1 | 49 KB | ##################################### | 100% scipy-1.5.2 | 14.4 MB | ##################################### | 100% libprotobuf-3.17.2 | 2.0 MB | ##################################### | 100% opt_einsum-3.3.0 | 57 KB | ##################################### | 100% cudatoolkit-10.0.130 | 261.2 MB | ##################################### | 100% markdown-3.1.1 | 116 KB | ##################################### | 100% werkzeug-0.16.1 | 258 KB | ##################################### | 100% absl-py-0.15.0 | 103 KB | ##################################### | 100% astor-0.8.1 | 47 KB | ##################################### | 100% numpy-base-1.19.2 | 4.1 MB | ##################################### | 100% mkl-2020.2 | 138.3 MB | ##################################### | 100% intel-openmp-2022.1. | 4.5 MB | ##################################### | 100% gast-0.2.2 | 155 KB | ##################################### | 100% libgfortran-ng-7.5.0 | 22 KB | ##################################### | 100% hdf5-1.10.6 | 3.7 MB | ##################################### | 100% numpy-1.19.2 | 22 KB | ##################################### | 100% keras-applications-1 | 29 KB | ##################################### | 100% google-pasta-0.2.0 | 46 KB | ##################################### | 100% tensorflow-base-1.15 | 156.5 MB | ##################################### | 100% tensorflow-estimator | 271 KB | ##################################### | 100% libgfortran4-7.5.0 | 995 KB | ##################################### | 100% _tflow_select-2.1.0 | 2 KB | ##################################### | 100% termcolor-1.1.0 | 9 KB | ##################################### | 100% mkl_random-1.1.1 | 327 KB | ##################################### | 100% tensorflow-1.15.0 | 4 KB | ##################################### | 100% tensorboard-1.15.0 | 3.2 MB | ##################################### | 100% cupti-10.0.130 | 1.5 MB | ##################################### | 100% six-1.16.0 | 18 KB | ##################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done Note: you may need to restart the kernel to use updated packages.
安装protobuf
conda install -c anaconda protobuf
/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version! RequestsDependencyWarning) Collecting package metadata (current_repodata.json): done Solving environment: - The environment is inconsistent, please check the package plan carefully The following packages are causing the inconsistency: - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::libstdcxx-ng==11.2.0=h1234567_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::libgcc-ng==11.2.0=h1234567_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::libffi==3.3=he6710b0_2 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::ncurses==6.4=h6a678d5_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::openssl==1.1.1t=h7f8727e_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::xz==5.2.10=h5eee18b_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::zlib==1.2.13=h5eee18b_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::readline==8.2=h5eee18b_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::tk==8.6.12=h1ccaba5_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::sqlite==3.41.2=h5eee18b_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::python==3.6.10=h7579374_2 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::certifi==2021.5.30=py36h06a4308_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::wheel==0.37.1=pyhd3eb1b0_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::setuptools==58.0.4=py36h06a4308_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::pip==21.2.2=py36h06a4308_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::libgfortran-ng==7.5.0=ha8ba4b0_17 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::c-ares==1.19.0=h5eee18b_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::hdf5==1.10.6=hb1b8bf9_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::intel-openmp==2022.1.0=h9e868ea_3769 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::libprotobuf==3.17.2=h4ff587b_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::mkl==2020.2=256 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::astor==0.8.1=py36h06a4308_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::gast==0.2.2=py36_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::six==1.16.0=pyhd3eb1b0_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::termcolor==1.1.0=py36h06a4308_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::webencodings==0.5.1=py36_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::werkzeug==0.16.1=py_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::wrapt==1.12.1=py36h7b6447c_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::absl-py==0.15.0=pyhd3eb1b0_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::google-pasta==0.2.0=pyhd3eb1b0_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::mkl-service==2.3.0=py36he8ac12f_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::grpcio==1.31.0=py36hf8bcb03_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::markdown==3.1.1=py36_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::numpy-base==1.19.2=py36hfa32c7d_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::protobuf==3.17.2=py36h295c915_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::h5py==2.10.0=py36hd6299e0_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::mkl_fft==1.3.0=py36h54f3939_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::mkl_random==1.1.1=py36h0573a6f_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::numpy==1.19.2=py36h54aff64_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::keras-applications==1.0.8=py_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::opt_einsum==3.3.0=pyhd3eb1b0_1 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::scipy==1.5.2=py36h0b6359f_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::tensorboard==1.15.0=pyhb230dea_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::keras-preprocessing==1.1.2=pyhd3eb1b0_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::tensorflow-base==1.15.0=gpu_py36h9dcbed7_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/noarch::tensorflow-estimator==1.15.1=pyh2649769_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::tensorflow==1.15.0=gpu_py36h5a509aa_0 - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/linux-64::tensorflow-gpu==1.15.0=h0d30ee6_0 done ==> WARNING: A newer version of conda exists. <== current version: 4.8.2 latest version: 23.3.1 Please update conda by running $ conda update -n base -c defaults conda ## Package Plan ## environment location: /home/ma-user/anaconda3/envs/py36 added / updated specs: - protobuf The following packages will be downloaded: package | build ---------------------------|----------------- ca-certificates-2023.01.10 | h06a4308_0 127 KB anaconda ------------------------------------------------------------ Total: 127 KB The following packages will be SUPERSEDED by a higher-priority channel: ca-certificates anaconda/pkgs/main --> anaconda Downloading and Extracting Packages ca-certificates-2023 | 127 KB | ##################################### | 100% Preparing transaction: done Verifying transaction: done Executing transaction: done Note: you may need to restart the kernel to use updated packages.
安装object detection
%cd object_detection/models/research
/home/ma-user/work/object_detection/models/research
%ll
total 92 -rw-r----- 1 ma-user 7130 Jun 12 2022 README.md drwxr-x--- 3 ma-user 4096 Apr 19 15:06 adversarial_text/ drwxr-x--- 3 ma-user 4096 Apr 19 15:06 attention_ocr/ drwxr-x--- 4 ma-user 4096 Apr 19 15:06 audioset/ drwxr-x--- 2 ma-user 4096 Apr 19 15:06 autoaugment/ drwxr-x--- 4 ma-user 4096 Apr 19 15:06 cognitive_planning/ drwxr-x--- 7 ma-user 4096 Apr 19 15:06 cvt_text/ drwxr-x--- 3 ma-user 4096 Apr 19 15:06 deep_speech/ drwxr-x--- 9 ma-user 4096 Apr 19 15:06 deeplab/ drwxr-x--- 3 ma-user 4096 Apr 19 15:06 delf/ drwxr-x--- 8 ma-user 4096 Apr 19 15:06 efficient-hrl/ drwxr-x--- 3 ma-user 4096 Apr 19 15:06 lfads/ drwxr-x--- 13 ma-user 4096 Apr 19 15:06 lstm_object_detection/ drwxr-x--- 2 ma-user 4096 Apr 19 15:06 marco/ drwxr-x--- 2 ma-user 4096 Apr 19 15:06 nst_blogpost/ drwxr-x--- 27 ma-user 4096 Apr 19 15:06 object_detection/ drwxr-x--- 2 ma-user 4096 Apr 19 15:06 pcl_rl/ drwxr-x--- 2 ma-user 4096 Apr 19 15:06 rebar/ drwxr-x--- 10 ma-user 4096 Apr 19 15:06 seq_flow_lite/ -rw-r----- 1 ma-user 978 Apr 17 22:06 setup.py drwxr-x--- 7 ma-user 4096 Apr 19 15:06 slim/ drwxr-x--- 5 ma-user 4096 Apr 19 15:06 vid2depth/
编译
!protoc object_detection/protos/*.proto --python_out=.
安装
!cp object_detection/packages/tf1/setup.py .
!python -m pip install --use-feature=2020-resolver .
WARNING: --use-feature=2020-resolver no longer has any effect, since it is now the default dependency resolver in pip. This will become an error in pip 21.0. Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple Processing /home/ma-user/work/object_detection/models/research DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default. pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555. Collecting pillow Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ea/0f/2fa195c2d8c6fe0b3dc2df5fc6ac6b8dbd005ea30aaa0fa43eca88b8c664/Pillow-8.4.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB) |████████████████████████████████| 3.1 MB 16.8 MB/s eta 0:00:01 Collecting lxml Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/f0/43/79e6b44ca079ae67eaee9e1d92406a6a2df4f44788dc776570f8da47888c/lxml-4.8.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.4 MB) |████████████████████████████████| 6.4 MB 57.2 MB/s eta 0:00:01��█▉ | 778 kB 57.2 MB/s eta 0:00:01MB/s eta 0:00:01 Collecting matplotlib Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/09/03/b7b30fa81cb687d1178e085d0f01111ceaea3bf81f9330c937fb6f6c8ca0/matplotlib-3.3.4-cp36-cp36m-manylinux1_x86_64.whl (11.5 MB) |████████████████████████████████| 11.5 MB 16.4 MB/s eta 0:00:0100:01 Collecting Cython Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/45/f3/badccbc14f7c2735d606cebf93cc8536a0a6b56d08ea546441583048e17d/Cython-0.29.32-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.0 MB) |████████████████████████████████| 2.0 MB 52.4 MB/s eta 0:00:01 Collecting contextlib2 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/76/56/6d6872f79d14c0cb02f1646cbb4592eef935857c0951a105874b7b62a0c3/contextlib2-21.6.0-py2.py3-none-any.whl (13 kB) Collecting tf-slim Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/02/97/b0f4a64df018ca018cc035d44f2ef08f91e2e8aa67271f6f19633a015ff7/tf_slim-1.1.0-py2.py3-none-any.whl (352 kB) |████████████████████████████████| 352 kB 16.7 MB/s eta 0:00:01 Requirement already satisfied: six in /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages (from object-detection==0.1) (1.16.0) Collecting pycocotools Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/ef/c6/90220be3b39fbc4cbd203775ca47dd8dc97fae06fbd2b500637395621b7c/pycocotools-2.0.6.tar.gz (24 kB) Installing build dependencies ... done Getting requirements to build wheel ... done Preparing wheel metadata ... done Collecting lvis Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/72/b6/1992240ab48310b5360bfdd1d53163f43bb97d90dc5dc723c67d41c38e78/lvis-0.5.3-py3-none-any.whl (14 kB) Requirement already satisfied: scipy in /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages (from object-detection==0.1) (1.5.2) Collecting pandas Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/c3/e2/00cacecafbab071c787019f00ad84ca3185952f6bb9bca9550ed83870d4d/pandas-1.1.5-cp36-cp36m-manylinux1_x86_64.whl (9.5 MB) |████████████████████████████████| 9.5 MB 68.1 MB/s eta 0:00:010:01�████████████ | 7.4 MB 68.1 MB/s eta 0:00:01 Requirement already satisfied: python-dateutil>=2.8.0 in /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages (from lvis->object-detection==0.1) (2.8.2) Collecting opencv-python>=4.1.0.25 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/af/bf/8d189a5c43460f6b5c8eb81ead8732e94b9f73ef8d9abba9e8f5a61a6531/opencv_python-4.6.0.66-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.9 MB) |████████████████████████████████| 60.9 MB 68.7 MB/s eta 0:00:01 | 3.3 MB 68.7 MB/s eta 0:00:01 | 8.1 MB 68.7 MB/s eta 0:00:0101 | 17.9 MB 68.7 MB/s eta 0:00:01��███████ | 22.7 MB 68.7 MB/s eta 0:00:01███▌ | 27.5 MB 68.7 MB/s eta 0:00:01 | 32.3 MB 68.7 MB/s eta 0:00:016.5 MB 68.7 MB/s eta 0:00:01��████████████████████▌ | 41.0 MB 68.7 MB/s eta 0:00:01██████ | 45.5 MB 68.7 MB/s eta 0:00:01 | 50.1 MB 68.7 MB/s eta 0:00:01��███████████████████▊ | 54.6 MB 68.7 MB/s eta 0:00:01 |███████████████████████████████ | 59.1 MB 68.7 MB/s eta 0:00:01 Collecting kiwisolver>=1.1.0 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/a7/1b/cbd8ae738719b5f41592a12057ef5442e2ed5f5cb5451f8fc7e9f8875a1a/kiwisolver-1.3.1-cp36-cp36m-manylinux1_x86_64.whl (1.1 MB) |████████████████████████████████| 1.1 MB 75.6 MB/s eta 0:00:01 Collecting pyparsing>=2.4.0 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/6c/10/a7d0fa5baea8fe7b50f448ab742f26f52b80bfca85ac2be9d35cdd9a3246/pyparsing-3.0.9-py3-none-any.whl (98 kB) |████████████████████████████████| 98 kB 18.8 MB/s eta 0:00:01 Collecting cycler>=0.10.0 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/5c/f9/695d6bedebd747e5eb0fe8fad57b72fdf25411273a39791cde838d5a8f51/cycler-0.11.0-py3-none-any.whl (6.4 kB) Requirement already satisfied: numpy>=1.18.2 in /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages (from lvis->object-detection==0.1) (1.19.2) Collecting pytz>=2017.2 Downloading http://repo.myhuaweicloud.com/repository/pypi/packages/85/ac/92f998fc52a70afd7f6b788142632afb27cd60c8c782d1452b7466603332/pytz-2022.6-py2.py3-none-any.whl (498 kB) |████████████████████████████████| 498 kB 16.5 MB/s eta 0:00:01 Requirement already satisfied: absl-py>=0.2.2 in /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages (from tf-slim->object-detection==0.1) (0.15.0) Building wheels for collected packages: object-detection, pycocotools Building wheel for object-detection (setup.py) ... done Created wheel for object-detection: filename=object_detection-0.1-py3-none-any.whl size=1694056 sha256=f75692c2ad67e84d0d2f2260e6834a5013d412e2498d0d3c9149d4be04032dd5 Stored in directory: /tmp/pip-ephem-wheel-cache-cju3rnds/wheels/12/13/38/e12d2a5e3323ed1b1d9131cf197e7a937a73f040c5e00340c7 Building wheel for pycocotools (PEP 517) ... done Created wheel for pycocotools: filename=pycocotools-2.0.6-cp36-cp36m-linux_x86_64.whl size=276422 sha256=e9b68f50857ed53c4737b8610fa1f334480463a2c7447507b73c81a8b726a139 Stored in directory: /home/ma-user/.cache/pip/wheels/42/6a/4a/f778c1c0cfe0369513eb1d2fb4ad160e4b1753cd905b2ea0fe Successfully built object-detection pycocotools Installing collected packages: pyparsing, pillow, kiwisolver, cycler, pytz, opencv-python, matplotlib, Cython, tf-slim, pycocotools, pandas, lxml, lvis, contextlib2, object-detection Successfully installed Cython-0.29.32 contextlib2-21.6.0 cycler-0.11.0 kiwisolver-1.3.1 lvis-0.5.3 lxml-4.8.0 matplotlib-3.3.4 object-detection-0.1 opencv-python-4.6.0.66 pandas-1.1.5 pillow-8.4.0 pycocotools-2.0.6 pyparsing-3.0.9 pytz-2022.6 tf-slim-1.1.0
验证
!python object_detection/builders/model_builder_tf1_test.py
Running tests under Python 3.6.10: /home/ma-user/anaconda3/envs/py36/bin/python [ RUN ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params0 (True) [ OK ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params0 (True) [ RUN ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params1 (False) [ OK ] ModelBuilderTF1Test.test_create_context_rcnn_from_config_with_params1 (False) [ RUN ] ModelBuilderTF1Test.test_create_experimental_model [ OK ] ModelBuilderTF1Test.test_create_experimental_model [ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature0 (True) [ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature0 (True) [ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature1 (False) [ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_from_config_with_crop_feature1 (False) [ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_model_from_config_with_example_miner [ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_model_from_config_with_example_miner [ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul [ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_with_matmul [ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul [ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_faster_rcnn_without_matmul [ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul [ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_with_matmul [ RUN ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul [ OK ] ModelBuilderTF1Test.test_create_faster_rcnn_models_from_config_mask_rcnn_without_matmul [ RUN ] ModelBuilderTF1Test.test_create_rfcn_model_from_config [ OK ] ModelBuilderTF1Test.test_create_rfcn_model_from_config [ RUN ] ModelBuilderTF1Test.test_create_ssd_fpn_model_from_config [ OK ] ModelBuilderTF1Test.test_create_ssd_fpn_model_from_config [ RUN ] ModelBuilderTF1Test.test_create_ssd_models_from_config [ OK ] ModelBuilderTF1Test.test_create_ssd_models_from_config [ RUN ] ModelBuilderTF1Test.test_invalid_faster_rcnn_batchnorm_update [ OK ] ModelBuilderTF1Test.test_invalid_faster_rcnn_batchnorm_update [ RUN ] ModelBuilderTF1Test.test_invalid_first_stage_nms_iou_threshold [ OK ] ModelBuilderTF1Test.test_invalid_first_stage_nms_iou_threshold [ RUN ] ModelBuilderTF1Test.test_invalid_model_config_proto [ OK ] ModelBuilderTF1Test.test_invalid_model_config_proto [ RUN ] ModelBuilderTF1Test.test_invalid_second_stage_batch_size [ OK ] ModelBuilderTF1Test.test_invalid_second_stage_batch_size [ RUN ] ModelBuilderTF1Test.test_session [ SKIPPED ] ModelBuilderTF1Test.test_session [ RUN ] ModelBuilderTF1Test.test_unknown_faster_rcnn_feature_extractor [ OK ] ModelBuilderTF1Test.test_unknown_faster_rcnn_feature_extractor [ RUN ] ModelBuilderTF1Test.test_unknown_meta_architecture [ OK ] ModelBuilderTF1Test.test_unknown_meta_architecture [ RUN ] ModelBuilderTF1Test.test_unknown_ssd_feature_extractor [ OK ] ModelBuilderTF1Test.test_unknown_ssd_feature_extractor ---------------------------------------------------------------------- Ran 21 tests in 0.170s OK (skipped=1)
%cd ../..
/home/ma-user/work/object_detection
!python ./model_main.py --pipeline_config_path=./data/ssdlite_mobilenet_v2_coco.config --model_dir=./my_model_dir --num_train_steps=20000 --sample_1_of_n_eval_examples=1 --alsologtostderr
WARNING:tensorflow:Forced number of epochs for all eval validations to be 1. W0419 22:36:39.569898 140176207116096 model_lib.py:839] Forced number of epochs for all eval validations to be 1. INFO:tensorflow:Maybe overwriting train_steps: 20000 I0419 22:36:39.570197 140176207116096 config_util.py:552] Maybe overwriting train_steps: 20000 INFO:tensorflow:Maybe overwriting use_bfloat16: False I0419 22:36:39.570283 140176207116096 config_util.py:552] Maybe overwriting use_bfloat16: False INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1 I0419 22:36:39.570350 140176207116096 config_util.py:552] Maybe overwriting sample_1_of_n_eval_examples: 1 INFO:tensorflow:Maybe overwriting eval_num_epochs: 1 I0419 22:36:39.570466 140176207116096 config_util.py:552] Maybe overwriting eval_num_epochs: 1 WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1. W0419 22:36:39.570559 140176207116096 model_lib.py:855] Expected number of evaluation epochs is 1, but instead encountered `eval_on_train_input_config.num_epochs` = 0. Overwriting `num_epochs` to 1. INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu None I0419 22:36:39.570642 140176207116096 model_lib.py:892] create_estimator_and_inputs: use_tpu False, export_to_tpu None INFO:tensorflow:Using config: {'_model_dir': './my_model_dir', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f7cd49ae7b8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} I0419 22:36:39.571040 140176207116096 estimator.py:212] Using config: {'_model_dir': './my_model_dir', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f7cd49ae7b8>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7f7cd49bb1e0>) includes params argument, but params are not passed to Estimator. W0419 22:36:39.572054 140176207116096 model_fn.py:630] Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7f7cd49bb1e0>) includes params argument, but params are not passed to Estimator. INFO:tensorflow:Not using Distribute Coordinator. I0419 22:36:39.572829 140176207116096 estimator_training.py:186] Not using Distribute Coordinator. INFO:tensorflow:Running training and evaluation locally (non-distributed). I0419 22:36:39.572988 140176207116096 training.py:612] Running training and evaluation locally (non-distributed). INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. I0419 22:36:39.573206 140176207116096 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. W0419 22:36:39.580518 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/train.record'] I0419 22:36:39.617939 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/train.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/train.record'] I0419 22:36:39.618871 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/train.record'] INFO:tensorflow:Number of filenames to read: 1 I0419 22:36:39.619092 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards. W0419 22:36:39.619165 140176207116096 dataset_builder.py:87] num_readers has been reduced to 1 to match input file shards. WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/builders/dataset_builder.py:104: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`. W0419 22:36:39.625500 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/builders/dataset_builder.py:104: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.experimental_determinstic`. WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/builders/dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.map() W0419 22:36:39.650899 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/builders/dataset_builder.py:236: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.data.Dataset.map() WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/inputs.py:113: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W0419 22:36:54.531403 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/inputs.py:113: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/inputs.py:97: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead. W0419 22:36:54.719357 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/inputs.py:97: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead. WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/autograph/operators/control_flow.py:1004: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: `seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead. W0419 22:37:03.222925 140176207116096 api.py:332] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/autograph/operators/control_flow.py:1004: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version. Instructions for updating: `seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead. WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/inputs.py:288: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. W0419 22:37:07.202532 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/inputs.py:288: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. INFO:tensorflow:Calling model_fn. I0419 22:37:11.676310 140176207116096 estimator.py:1148] Calling model_fn. WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. W0419 22:37:12.180777 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:37:14.857838 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:37:14.955950 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:37:15.061872 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:37:15.161028 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:37:15.258508 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:37:15.356826 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor W0419 22:37:22.344094 140176207116096 deprecation.py:506] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/training/rmsprop.py:119: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version. Instructions for updating: Call initializer instance with the dtype argument instead of passing it to the constructor INFO:tensorflow:Done calling model_fn. I0419 22:37:30.257880 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Create CheckpointSaverHook. I0419 22:37:30.259304 140176207116096 basic_session_run_hooks.py:541] Create CheckpointSaverHook. INFO:tensorflow:Graph was finalized. I0419 22:37:34.441008 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-19 22:37:34.441465: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2023-04-19 22:37:34.459669: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz 2023-04-19 22:37:34.460780: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55f41cbf89d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2023-04-19 22:37:34.460836: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2023-04-19 22:37:34.463494: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2023-04-19 22:37:34.591864: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:37:34.592953: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55f41c8c3940 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2023-04-19 22:37:34.593010: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0 2023-04-19 22:37:34.593393: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:37:34.594340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-19 22:37:34.594730: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 22:37:34.596248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-19 22:37:34.597698: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-19 22:37:34.597982: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-19 22:37:34.599795: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-19 22:37:34.601182: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-19 22:37:34.605275: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 22:37:34.605482: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:37:34.606483: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:37:34.607338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-19 22:37:34.607406: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 22:37:34.609150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-19 22:37:34.609166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-19 22:37:34.609175: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-19 22:37:34.609291: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:37:34.610209: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:37:34.611086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Running local_init_op. I0419 22:37:41.656421 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0419 22:37:42.248819 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Saving checkpoints for 0 into ./my_model_dir/model.ckpt. I0419 22:37:55.287573 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 0 into ./my_model_dir/model.ckpt. 2023-04-19 22:38:13.062418: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 22:38:18.433741: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 INFO:tensorflow:loss = 15.413351, step = 0 I0419 22:38:19.525510 140176207116096 basic_session_run_hooks.py:262] loss = 15.413351, step = 0 INFO:tensorflow:global_step/sec: 1.61398 I0419 22:39:21.483655 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.61398 INFO:tensorflow:loss = 7.804327, step = 100 (61.960 sec) I0419 22:39:21.485480 140176207116096 basic_session_run_hooks.py:260] loss = 7.804327, step = 100 (61.960 sec) INFO:tensorflow:global_step/sec: 1.86404 I0419 22:40:15.131113 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86404 INFO:tensorflow:loss = 5.690725, step = 200 (53.648 sec) I0419 22:40:15.133066 140176207116096 basic_session_run_hooks.py:260] loss = 5.690725, step = 200 (53.648 sec) INFO:tensorflow:global_step/sec: 1.85364 I0419 22:41:09.078472 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85364 INFO:tensorflow:loss = 5.501807, step = 300 (53.947 sec) I0419 22:41:09.079810 140176207116096 basic_session_run_hooks.py:260] loss = 5.501807, step = 300 (53.947 sec) INFO:tensorflow:global_step/sec: 1.8629 I0419 22:42:02.758059 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8629 INFO:tensorflow:loss = 4.371992, step = 400 (53.680 sec) I0419 22:42:02.759510 140176207116096 basic_session_run_hooks.py:260] loss = 4.371992, step = 400 (53.680 sec) INFO:tensorflow:global_step/sec: 1.87014 I0419 22:42:56.230299 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87014 INFO:tensorflow:loss = 4.7281284, step = 500 (53.473 sec) I0419 22:42:56.232190 140176207116096 basic_session_run_hooks.py:260] loss = 4.7281284, step = 500 (53.473 sec) INFO:tensorflow:global_step/sec: 1.86738 I0419 22:43:49.781025 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86738 INFO:tensorflow:loss = 4.5236554, step = 600 (53.550 sec) I0419 22:43:49.782305 140176207116096 basic_session_run_hooks.py:260] loss = 4.5236554, step = 600 (53.550 sec) INFO:tensorflow:global_step/sec: 1.86465 I0419 22:44:43.410333 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86465 INFO:tensorflow:loss = 3.3881083, step = 700 (53.630 sec) I0419 22:44:43.412017 140176207116096 basic_session_run_hooks.py:260] loss = 3.3881083, step = 700 (53.630 sec) INFO:tensorflow:global_step/sec: 1.87522 I0419 22:45:36.737486 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87522 INFO:tensorflow:loss = 4.1427884, step = 800 (53.327 sec) I0419 22:45:36.738705 140176207116096 basic_session_run_hooks.py:260] loss = 4.1427884, step = 800 (53.327 sec) INFO:tensorflow:global_step/sec: 1.88112 I0419 22:46:29.897989 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88112 INFO:tensorflow:loss = 4.3989077, step = 900 (53.161 sec) I0419 22:46:29.899549 140176207116096 basic_session_run_hooks.py:260] loss = 4.3989077, step = 900 (53.161 sec) INFO:tensorflow:global_step/sec: 1.88558 I0419 22:47:22.931572 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88558 INFO:tensorflow:loss = 3.7907445, step = 1000 (53.033 sec) I0419 22:47:22.933047 140176207116096 basic_session_run_hooks.py:260] loss = 3.7907445, step = 1000 (53.033 sec) INFO:tensorflow:Saving checkpoints for 1068 into ./my_model_dir/model.ckpt. I0419 22:47:58.641153 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 1068 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0419 22:48:00.395913 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0419 22:48:00.396692 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0419 22:48:00.396918 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0419 22:48:01.727814 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:48:04.050972 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:48:04.135640 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:48:04.230169 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:48:04.313097 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:48:04.400281 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:48:04.485882 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/eval_util.py:929: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. W0419 22:48:05.264171 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/eval_util.py:929: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.cast` instead. WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2. - tf.py_function takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means `tf.py_function`s can use accelerators such as GPUs as well as being differentiable using a gradient tape. - tf.numpy_function maintains the semantics of the deprecated tf.py_func (it is not differentiable, and manipulates numpy arrays). It drops the stateful argument making all functions stateful. W0419 22:48:05.464002 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/utils/visualization_utils.py:618: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version. Instructions for updating: tf.py_func is deprecated in TF V2. Instead, there are two options available in V2. - tf.py_function takes a python function which manipulates tf eager tensors instead of numpy arrays. It's easy to convert a tf eager tensor to an ndarray (just call tensor.numpy()) but having access to eager tensors means `tf.py_function`s can use accelerators such as GPUs as well as being differentiable using a gradient tape. - tf.numpy_function maintains the semantics of the deprecated tf.py_func (it is not differentiable, and manipulates numpy arrays). It drops the stateful argument making all functions stateful. INFO:tensorflow:Done calling model_fn. I0419 22:48:06.062914 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-19T22:48:06Z I0419 22:48:06.086663 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-19T22:48:06Z INFO:tensorflow:Graph was finalized. I0419 22:48:06.611279 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-19 22:48:06.612909: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:48:06.613397: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-19 22:48:06.613629: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 22:48:06.613666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-19 22:48:06.613693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-19 22:48:06.613719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-19 22:48:06.613743: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-19 22:48:06.613768: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-19 22:48:06.613792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 22:48:06.613861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:48:06.614300: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:48:06.614676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-19 22:48:06.614724: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-19 22:48:06.614734: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-19 22:48:06.614741: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-19 22:48:06.614834: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:48:06.615514: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:48:06.616223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-1068 I0419 22:48:06.617246 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-1068 INFO:tensorflow:Running local_init_op. I0419 22:48:07.859357 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0419 22:48:08.010254 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0419 22:48:28.011782 140169121203968 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0419 22:48:28.090736 140169121203968 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0419 22:48:28.114509 140169121203968 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=0.67s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.004 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.004 INFO:tensorflow:Finished evaluation at 2023-04-19-22:48:29 I0419 22:48:29.046900 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-19-22:48:29 INFO:tensorflow:Saving dict for global step 1068: DetectionBoxes_Precision/mAP = 9.715056e-07, DetectionBoxes_Precision/mAP (large) = 9.715056e-07, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 2.8815455e-06, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.0038461538, DetectionBoxes_Recall/AR@100 (large) = 0.0040983604, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.8561096, Loss/localization_loss = 2.6189196, Loss/regularization_loss = 0.28472114, Loss/total_loss = 10.759753, global_step = 1068, learning_rate = 0.004, loss = 10.759753 I0419 22:48:29.047364 140176207116096 estimator.py:2049] Saving dict for global step 1068: DetectionBoxes_Precision/mAP = 9.715056e-07, DetectionBoxes_Precision/mAP (large) = 9.715056e-07, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 2.8815455e-06, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.0038461538, DetectionBoxes_Recall/AR@100 (large) = 0.0040983604, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.8561096, Loss/localization_loss = 2.6189196, Loss/regularization_loss = 0.28472114, Loss/total_loss = 10.759753, global_step = 1068, learning_rate = 0.004, loss = 10.759753 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1068: ./my_model_dir/model.ckpt-1068 I0419 22:48:29.781107 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 1068: ./my_model_dir/model.ckpt-1068 INFO:tensorflow:global_step/sec: 1.17533 I0419 22:48:48.014041 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.17533 INFO:tensorflow:loss = 3.0477262, step = 1100 (85.082 sec) I0419 22:48:48.015497 140176207116096 basic_session_run_hooks.py:260] loss = 3.0477262, step = 1100 (85.082 sec) INFO:tensorflow:global_step/sec: 1.85929 I0419 22:49:41.798031 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85929 INFO:tensorflow:loss = 3.3657098, step = 1200 (53.784 sec) I0419 22:49:41.799298 140176207116096 basic_session_run_hooks.py:260] loss = 3.3657098, step = 1200 (53.784 sec) INFO:tensorflow:global_step/sec: 1.86595 I0419 22:50:35.390022 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86595 INFO:tensorflow:loss = 2.9763389, step = 1300 (53.592 sec) I0419 22:50:35.391514 140176207116096 basic_session_run_hooks.py:260] loss = 2.9763389, step = 1300 (53.592 sec) INFO:tensorflow:global_step/sec: 1.85686 I0419 22:51:29.244328 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85686 INFO:tensorflow:loss = 3.766046, step = 1400 (53.854 sec) I0419 22:51:29.245539 140176207116096 basic_session_run_hooks.py:260] loss = 3.766046, step = 1400 (53.854 sec) INFO:tensorflow:global_step/sec: 1.84548 I0419 22:52:23.430682 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.84548 INFO:tensorflow:loss = 3.2416878, step = 1500 (54.186 sec) I0419 22:52:23.431988 140176207116096 basic_session_run_hooks.py:260] loss = 3.2416878, step = 1500 (54.186 sec) INFO:tensorflow:global_step/sec: 1.83784 I0419 22:53:17.842534 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.83784 INFO:tensorflow:loss = 2.9348068, step = 1600 (54.412 sec) I0419 22:53:17.844059 140176207116096 basic_session_run_hooks.py:260] loss = 2.9348068, step = 1600 (54.412 sec) INFO:tensorflow:global_step/sec: 1.86218 I0419 22:54:11.543003 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86218 INFO:tensorflow:loss = 3.6011856, step = 1700 (53.700 sec) I0419 22:54:11.544484 140176207116096 basic_session_run_hooks.py:260] loss = 3.6011856, step = 1700 (53.700 sec) INFO:tensorflow:global_step/sec: 1.85784 I0419 22:55:05.368858 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85784 INFO:tensorflow:loss = 2.684142, step = 1800 (53.826 sec) I0419 22:55:05.370124 140176207116096 basic_session_run_hooks.py:260] loss = 2.684142, step = 1800 (53.826 sec) INFO:tensorflow:global_step/sec: 1.86764 I0419 22:55:58.912207 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86764 INFO:tensorflow:loss = 3.250209, step = 1900 (53.544 sec) I0419 22:55:58.913745 140176207116096 basic_session_run_hooks.py:260] loss = 3.250209, step = 1900 (53.544 sec) INFO:tensorflow:global_step/sec: 1.87784 I0419 22:56:52.164996 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87784 INFO:tensorflow:loss = 2.9760566, step = 2000 (53.253 sec) I0419 22:56:52.166257 140176207116096 basic_session_run_hooks.py:260] loss = 2.9760566, step = 2000 (53.253 sec) INFO:tensorflow:global_step/sec: 1.85415 I0419 22:57:46.098258 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85415 INFO:tensorflow:loss = 3.0208983, step = 2100 (53.933 sec) I0419 22:57:46.099697 140176207116096 basic_session_run_hooks.py:260] loss = 3.0208983, step = 2100 (53.933 sec) INFO:tensorflow:Saving checkpoints for 2125 into ./my_model_dir/model.ckpt. I0419 22:57:58.842164 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 2125 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0419 22:58:00.657943 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0419 22:58:00.658629 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0419 22:58:00.658716 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0419 22:58:01.470880 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:58:04.318256 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:58:04.415472 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:58:04.524110 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:58:04.615081 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:58:04.699091 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 22:58:04.785628 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0419 22:58:06.365143 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-19T22:58:06Z I0419 22:58:06.382445 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-19T22:58:06Z INFO:tensorflow:Graph was finalized. I0419 22:58:06.913015 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-19 22:58:06.914360: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:58:06.914885: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-19 22:58:06.915038: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 22:58:06.915069: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-19 22:58:06.915095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-19 22:58:06.915130: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-19 22:58:06.915156: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-19 22:58:06.915180: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-19 22:58:06.915203: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 22:58:06.915267: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:58:06.915690: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:58:06.916030: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-19 22:58:06.916074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-19 22:58:06.916082: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-19 22:58:06.916087: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-19 22:58:06.916223: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:58:06.916642: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 22:58:06.916994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-2125 I0419 22:58:06.918452 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-2125 INFO:tensorflow:Running local_init_op. I0419 22:58:08.146824 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0419 22:58:08.329837 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0419 22:58:28.237354 140169129596672 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0419 22:58:28.241133 140169129596672 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0419 22:58:28.266199 140169129596672 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=0.66s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.002 INFO:tensorflow:Finished evaluation at 2023-04-19-22:58:29 I0419 22:58:29.216242 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-19-22:58:29 INFO:tensorflow:Saving dict for global step 2125: DetectionBoxes_Precision/mAP = 2.3652628e-07, DetectionBoxes_Precision/mAP (large) = 2.3652628e-07, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 1.1826314e-06, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.0015384615, DetectionBoxes_Recall/AR@100 (large) = 0.0016393443, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.3918037, Loss/localization_loss = 2.6018686, Loss/regularization_loss = 0.2854922, Loss/total_loss = 10.279162, global_step = 2125, learning_rate = 0.004, loss = 10.279162 I0419 22:58:29.216688 140176207116096 estimator.py:2049] Saving dict for global step 2125: DetectionBoxes_Precision/mAP = 2.3652628e-07, DetectionBoxes_Precision/mAP (large) = 2.3652628e-07, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 1.1826314e-06, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.0015384615, DetectionBoxes_Recall/AR@100 (large) = 0.0016393443, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.3918037, Loss/localization_loss = 2.6018686, Loss/regularization_loss = 0.2854922, Loss/total_loss = 10.279162, global_step = 2125, learning_rate = 0.004, loss = 10.279162 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 2125: ./my_model_dir/model.ckpt-2125 I0419 22:58:29.223881 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 2125: ./my_model_dir/model.ckpt-2125 INFO:tensorflow:global_step/sec: 1.18644 I0419 22:59:10.383833 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.18644 INFO:tensorflow:loss = 2.556478, step = 2200 (84.286 sec) I0419 22:59:10.385577 140176207116096 basic_session_run_hooks.py:260] loss = 2.556478, step = 2200 (84.286 sec) INFO:tensorflow:global_step/sec: 1.85625 I0419 23:00:04.255761 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85625 INFO:tensorflow:loss = 3.121453, step = 2300 (53.872 sec) I0419 23:00:04.257632 140176207116096 basic_session_run_hooks.py:260] loss = 3.121453, step = 2300 (53.872 sec) INFO:tensorflow:global_step/sec: 1.84558 I0419 23:00:58.439170 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.84558 INFO:tensorflow:loss = 2.829557, step = 2400 (54.183 sec) I0419 23:00:58.440687 140176207116096 basic_session_run_hooks.py:260] loss = 2.829557, step = 2400 (54.183 sec) INFO:tensorflow:global_step/sec: 1.86862 I0419 23:01:51.954531 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86862 INFO:tensorflow:loss = 3.0674396, step = 2500 (53.515 sec) I0419 23:01:51.956020 140176207116096 basic_session_run_hooks.py:260] loss = 3.0674396, step = 2500 (53.515 sec) INFO:tensorflow:global_step/sec: 1.85272 I0419 23:02:45.929410 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85272 INFO:tensorflow:loss = 2.5632863, step = 2600 (53.975 sec) I0419 23:02:45.931153 140176207116096 basic_session_run_hooks.py:260] loss = 2.5632863, step = 2600 (53.975 sec) INFO:tensorflow:global_step/sec: 1.84436 I0419 23:03:40.148807 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.84436 INFO:tensorflow:loss = 2.534719, step = 2700 (54.219 sec) I0419 23:03:40.150202 140176207116096 basic_session_run_hooks.py:260] loss = 2.534719, step = 2700 (54.219 sec) INFO:tensorflow:global_step/sec: 1.83652 I0419 23:04:34.599513 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.83652 INFO:tensorflow:loss = 3.3780591, step = 2800 (54.451 sec) I0419 23:04:34.600934 140176207116096 basic_session_run_hooks.py:260] loss = 3.3780591, step = 2800 (54.451 sec) INFO:tensorflow:global_step/sec: 1.85126 I0419 23:05:28.616744 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85126 INFO:tensorflow:loss = 2.0575783, step = 2900 (54.017 sec) I0419 23:05:28.618347 140176207116096 basic_session_run_hooks.py:260] loss = 2.0575783, step = 2900 (54.017 sec) INFO:tensorflow:global_step/sec: 1.85699 I0419 23:06:22.467274 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85699 INFO:tensorflow:loss = 2.6150322, step = 3000 (53.851 sec) I0419 23:06:22.468864 140176207116096 basic_session_run_hooks.py:260] loss = 2.6150322, step = 3000 (53.851 sec) INFO:tensorflow:global_step/sec: 1.86198 I0419 23:07:16.173726 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86198 INFO:tensorflow:loss = 2.787007, step = 3100 (53.706 sec) I0419 23:07:16.175107 140176207116096 basic_session_run_hooks.py:260] loss = 2.787007, step = 3100 (53.706 sec) INFO:tensorflow:Saving checkpoints for 3181 into ./my_model_dir/model.ckpt. I0419 23:07:59.107681 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 3181 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0419 23:08:00.862425 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0419 23:08:00.863231 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0419 23:08:00.863470 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0419 23:08:01.655987 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:08:03.921623 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:08:04.009291 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:08:04.101646 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:08:04.196213 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:08:04.281378 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:08:04.365588 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0419 23:08:06.342796 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-19T23:08:06Z I0419 23:08:06.359689 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-19T23:08:06Z INFO:tensorflow:Graph was finalized. I0419 23:08:06.886395 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-19 23:08:06.887305: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:08:06.887821: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-19 23:08:06.888007: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 23:08:06.888044: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-19 23:08:06.888071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-19 23:08:06.888095: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-19 23:08:06.888119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-19 23:08:06.888143: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-19 23:08:06.888166: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 23:08:06.888232: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:08:06.888668: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:08:06.889006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-19 23:08:06.889052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-19 23:08:06.889061: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-19 23:08:06.889067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-19 23:08:06.889211: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:08:06.889638: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:08:06.889989: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-3181 I0419 23:08:06.892459 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-3181 INFO:tensorflow:Running local_init_op. I0419 23:08:08.126363 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0419 23:08:08.275604 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0419 23:08:27.639101 140169112811264 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0419 23:08:27.641148 140169112811264 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0419 23:08:27.664784 140169112811264 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=0.71s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001 INFO:tensorflow:Finished evaluation at 2023-04-19-23:08:28 I0419 23:08:28.639946 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-19-23:08:28 INFO:tensorflow:Saving dict for global step 3181: DetectionBoxes_Precision/mAP = 5.559231e-08, DetectionBoxes_Precision/mAP (large) = 5.559231e-08, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 5.5592307e-07, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.00076923077, DetectionBoxes_Recall/AR@100 (large) = 0.00081967213, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.957195, Loss/localization_loss = 2.5863233, Loss/regularization_loss = 0.28602216, Loss/total_loss = 9.829543, global_step = 3181, learning_rate = 0.004, loss = 9.829543 I0419 23:08:28.640524 140176207116096 estimator.py:2049] Saving dict for global step 3181: DetectionBoxes_Precision/mAP = 5.559231e-08, DetectionBoxes_Precision/mAP (large) = 5.559231e-08, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 5.5592307e-07, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.00076923077, DetectionBoxes_Recall/AR@100 (large) = 0.00081967213, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.957195, Loss/localization_loss = 2.5863233, Loss/regularization_loss = 0.28602216, Loss/total_loss = 9.829543, global_step = 3181, learning_rate = 0.004, loss = 9.829543 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 3181: ./my_model_dir/model.ckpt-3181 I0419 23:08:28.647969 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 3181: ./my_model_dir/model.ckpt-3181 INFO:tensorflow:global_step/sec: 1.20106 I0419 23:08:39.433022 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.20106 INFO:tensorflow:loss = 2.4255388, step = 3200 (83.259 sec) I0419 23:08:39.434533 140176207116096 basic_session_run_hooks.py:260] loss = 2.4255388, step = 3200 (83.259 sec) INFO:tensorflow:global_step/sec: 1.84148 I0419 23:09:33.737158 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.84148 INFO:tensorflow:loss = 2.7101407, step = 3300 (54.304 sec) I0419 23:09:33.738431 140176207116096 basic_session_run_hooks.py:260] loss = 2.7101407, step = 3300 (54.304 sec) INFO:tensorflow:global_step/sec: 1.85214 I0419 23:10:27.728914 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85214 INFO:tensorflow:loss = 3.1696527, step = 3400 (53.992 sec) I0419 23:10:27.730705 140176207116096 basic_session_run_hooks.py:260] loss = 3.1696527, step = 3400 (53.992 sec) INFO:tensorflow:global_step/sec: 1.84436 I0419 23:11:21.948203 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.84436 INFO:tensorflow:loss = 2.3644645, step = 3500 (54.219 sec) I0419 23:11:21.949698 140176207116096 basic_session_run_hooks.py:260] loss = 2.3644645, step = 3500 (54.219 sec) INFO:tensorflow:global_step/sec: 1.85056 I0419 23:12:15.986026 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85056 INFO:tensorflow:loss = 1.9033536, step = 3600 (54.038 sec) I0419 23:12:15.987770 140176207116096 basic_session_run_hooks.py:260] loss = 1.9033536, step = 3600 (54.038 sec) INFO:tensorflow:global_step/sec: 1.84353 I0419 23:13:10.229596 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.84353 INFO:tensorflow:loss = 2.3616085, step = 3700 (54.243 sec) I0419 23:13:10.231129 140176207116096 basic_session_run_hooks.py:260] loss = 2.3616085, step = 3700 (54.243 sec) INFO:tensorflow:global_step/sec: 1.86593 I0419 23:14:03.822057 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86593 INFO:tensorflow:loss = 2.0761256, step = 3800 (53.593 sec) I0419 23:14:03.824213 140176207116096 basic_session_run_hooks.py:260] loss = 2.0761256, step = 3800 (53.593 sec) INFO:tensorflow:global_step/sec: 1.85162 I0419 23:14:57.828818 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85162 INFO:tensorflow:loss = 2.1272955, step = 3900 (54.006 sec) I0419 23:14:57.830042 140176207116096 basic_session_run_hooks.py:260] loss = 2.1272955, step = 3900 (54.006 sec) INFO:tensorflow:global_step/sec: 1.85098 I0419 23:15:51.854180 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85098 INFO:tensorflow:loss = 2.43963, step = 4000 (54.026 sec) I0419 23:15:51.855902 140176207116096 basic_session_run_hooks.py:260] loss = 2.43963, step = 4000 (54.026 sec) INFO:tensorflow:global_step/sec: 1.85669 I0419 23:16:45.713515 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85669 INFO:tensorflow:loss = 2.488336, step = 4100 (53.859 sec) I0419 23:16:45.714772 140176207116096 basic_session_run_hooks.py:260] loss = 2.488336, step = 4100 (53.859 sec) INFO:tensorflow:global_step/sec: 1.83801 I0419 23:17:40.120104 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.83801 INFO:tensorflow:loss = 2.3406048, step = 4200 (54.407 sec) I0419 23:17:40.121968 140176207116096 basic_session_run_hooks.py:260] loss = 2.3406048, step = 4200 (54.407 sec) INFO:tensorflow:Saving checkpoints for 4237 into ./my_model_dir/model.ckpt. I0419 23:17:59.342103 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 4237 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0419 23:18:01.079225 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0419 23:18:01.080026 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0419 23:18:01.080237 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0419 23:18:01.878866 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:18:04.173497 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:18:04.257643 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:18:04.352655 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:18:04.436439 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:18:04.519514 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:18:04.607557 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0419 23:18:06.158640 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-19T23:18:06Z I0419 23:18:06.175335 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-19T23:18:06Z INFO:tensorflow:Graph was finalized. I0419 23:18:06.691551 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-19 23:18:06.692628: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:18:06.693127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-19 23:18:06.693284: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 23:18:06.693316: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-19 23:18:06.693340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-19 23:18:06.693363: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-19 23:18:06.693413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-19 23:18:06.693439: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-19 23:18:06.693462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 23:18:06.693527: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:18:06.693927: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:18:06.694275: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-19 23:18:06.694322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-19 23:18:06.694330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-19 23:18:06.694335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-19 23:18:06.694462: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:18:06.694861: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:18:06.695211: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-4237 I0419 23:18:06.697618 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-4237 INFO:tensorflow:Running local_init_op. I0419 23:18:07.916480 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0419 23:18:08.052001 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0419 23:18:27.700734 140170527491840 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0419 23:18:27.704346 140170527491840 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0419 23:18:27.727701 140170527491840 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=0.68s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.002 INFO:tensorflow:Finished evaluation at 2023-04-19-23:18:28 I0419 23:18:28.672679 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-19-23:18:28 INFO:tensorflow:Saving dict for global step 4237: DetectionBoxes_Precision/mAP = 1.6552687e-07, DetectionBoxes_Precision/mAP (large) = 1.6552687e-07, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 8.276344e-07, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.0015384615, DetectionBoxes_Recall/AR@100 (large) = 0.0016393443, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.591464, Loss/localization_loss = 2.5681741, Loss/regularization_loss = 0.28636912, Loss/total_loss = 9.446008, global_step = 4237, learning_rate = 0.004, loss = 9.446008 I0419 23:18:28.673099 140176207116096 estimator.py:2049] Saving dict for global step 4237: DetectionBoxes_Precision/mAP = 1.6552687e-07, DetectionBoxes_Precision/mAP (large) = 1.6552687e-07, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 8.276344e-07, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0, DetectionBoxes_Recall/AR@100 = 0.0015384615, DetectionBoxes_Recall/AR@100 (large) = 0.0016393443, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.591464, Loss/localization_loss = 2.5681741, Loss/regularization_loss = 0.28636912, Loss/total_loss = 9.446008, global_step = 4237, learning_rate = 0.004, loss = 9.446008 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 4237: ./my_model_dir/model.ckpt-4237 I0419 23:18:28.680765 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 4237: ./my_model_dir/model.ckpt-4237 INFO:tensorflow:global_step/sec: 1.20819 I0419 23:19:02.888470 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.20819 INFO:tensorflow:loss = 2.3862898, step = 4300 (82.768 sec) I0419 23:19:02.889758 140176207116096 basic_session_run_hooks.py:260] loss = 2.3862898, step = 4300 (82.768 sec) INFO:tensorflow:global_step/sec: 1.88185 I0419 23:19:56.027779 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88185 INFO:tensorflow:loss = 2.2082098, step = 4400 (53.140 sec) I0419 23:19:56.029338 140176207116096 basic_session_run_hooks.py:260] loss = 2.2082098, step = 4400 (53.140 sec) INFO:tensorflow:global_step/sec: 1.87244 I0419 23:20:49.434065 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87244 INFO:tensorflow:loss = 2.2565105, step = 4500 (53.406 sec) I0419 23:20:49.435487 140176207116096 basic_session_run_hooks.py:260] loss = 2.2565105, step = 4500 (53.406 sec) INFO:tensorflow:global_step/sec: 1.87356 I0419 23:21:42.808512 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87356 INFO:tensorflow:loss = 2.3085456, step = 4600 (53.374 sec) I0419 23:21:42.809695 140176207116096 basic_session_run_hooks.py:260] loss = 2.3085456, step = 4600 (53.374 sec) INFO:tensorflow:global_step/sec: 1.86094 I0419 23:22:36.544759 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86094 INFO:tensorflow:loss = 2.4650106, step = 4700 (53.736 sec) I0419 23:22:36.546065 140176207116096 basic_session_run_hooks.py:260] loss = 2.4650106, step = 4700 (53.736 sec) INFO:tensorflow:global_step/sec: 1.85405 I0419 23:23:30.480745 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85405 INFO:tensorflow:loss = 1.7966045, step = 4800 (53.936 sec) I0419 23:23:30.482062 140176207116096 basic_session_run_hooks.py:260] loss = 1.7966045, step = 4800 (53.936 sec) INFO:tensorflow:global_step/sec: 1.87603 I0419 23:24:23.784695 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87603 INFO:tensorflow:loss = 2.0840428, step = 4900 (53.304 sec) I0419 23:24:23.785921 140176207116096 basic_session_run_hooks.py:260] loss = 2.0840428, step = 4900 (53.304 sec) INFO:tensorflow:global_step/sec: 1.85642 I0419 23:25:17.651826 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85642 INFO:tensorflow:loss = 1.9803698, step = 5000 (53.867 sec) I0419 23:25:17.653072 140176207116096 basic_session_run_hooks.py:260] loss = 1.9803698, step = 5000 (53.867 sec) INFO:tensorflow:global_step/sec: 1.88053 I0419 23:26:10.828334 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88053 INFO:tensorflow:loss = 2.0327735, step = 5100 (53.177 sec) I0419 23:26:10.829739 140176207116096 basic_session_run_hooks.py:260] loss = 2.0327735, step = 5100 (53.177 sec) INFO:tensorflow:global_step/sec: 1.87355 I0419 23:27:04.202998 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87355 INFO:tensorflow:loss = 2.01254, step = 5200 (53.375 sec) I0419 23:27:04.204474 140176207116096 basic_session_run_hooks.py:260] loss = 2.01254, step = 5200 (53.375 sec) INFO:tensorflow:global_step/sec: 1.87392 I0419 23:27:57.567000 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87392 INFO:tensorflow:loss = 2.6373184, step = 5300 (53.364 sec) I0419 23:27:57.568247 140176207116096 basic_session_run_hooks.py:260] loss = 2.6373184, step = 5300 (53.364 sec) INFO:tensorflow:Saving checkpoints for 5305 into ./my_model_dir/model.ckpt. I0419 23:27:59.784726 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 5305 into ./my_model_dir/model.ckpt. WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to delete files with this prefix. W0419 23:27:59.881393 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py:963: remove_checkpoint (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to delete files with this prefix. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0419 23:28:01.526000 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0419 23:28:01.526812 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0419 23:28:01.526906 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0419 23:28:02.325407 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:28:04.578832 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:28:04.662520 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:28:04.754379 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:28:04.841236 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:28:04.926614 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:28:05.008934 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0419 23:28:06.559891 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-19T23:28:06Z I0419 23:28:06.578604 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-19T23:28:06Z INFO:tensorflow:Graph was finalized. I0419 23:28:07.150825 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-19 23:28:07.152239: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:28:07.152803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-19 23:28:07.152952: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 23:28:07.152984: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-19 23:28:07.153021: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-19 23:28:07.153047: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-19 23:28:07.153071: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-19 23:28:07.153105: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-19 23:28:07.153133: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 23:28:07.153202: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:28:07.153643: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:28:07.154005: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-19 23:28:07.154053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-19 23:28:07.154061: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-19 23:28:07.154067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-19 23:28:07.154189: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:28:07.154628: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:28:07.155006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-5305 I0419 23:28:07.157782 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-5305 INFO:tensorflow:Running local_init_op. I0419 23:28:08.426149 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0419 23:28:08.558697 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0419 23:28:28.084888 140170502313728 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0419 23:28:28.088473 140170502313728 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.03s) I0419 23:28:28.120534 140170502313728 coco_tools.py:138] DONE (t=0.03s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=0.77s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.001 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.002 INFO:tensorflow:Finished evaluation at 2023-04-19-23:28:29 I0419 23:28:29.170563 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-19-23:28:29 INFO:tensorflow:Saving dict for global step 5305: DetectionBoxes_Precision/mAP = 4.080948e-06, DetectionBoxes_Precision/mAP (large) = 4.080948e-06, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 1.576075e-05, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0008368201, DetectionBoxes_Recall/AR@100 = 0.0023752816, DetectionBoxes_Recall/AR@100 (large) = 0.0024761644, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.3883076, Loss/localization_loss = 2.5438468, Loss/regularization_loss = 0.2866126, Loss/total_loss = 9.218768, global_step = 5305, learning_rate = 0.004, loss = 9.218768 I0419 23:28:29.170963 140176207116096 estimator.py:2049] Saving dict for global step 5305: DetectionBoxes_Precision/mAP = 4.080948e-06, DetectionBoxes_Precision/mAP (large) = 4.080948e-06, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 1.576075e-05, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0008368201, DetectionBoxes_Recall/AR@100 = 0.0023752816, DetectionBoxes_Recall/AR@100 (large) = 0.0024761644, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.3883076, Loss/localization_loss = 2.5438468, Loss/regularization_loss = 0.2866126, Loss/total_loss = 9.218768, global_step = 5305, learning_rate = 0.004, loss = 9.218768 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 5305: ./my_model_dir/model.ckpt-5305 I0419 23:28:29.177765 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 5305: ./my_model_dir/model.ckpt-5305 INFO:tensorflow:global_step/sec: 1.20987 I0419 23:29:20.220268 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.20987 INFO:tensorflow:loss = 2.5502517, step = 5400 (82.653 sec) I0419 23:29:20.221621 140176207116096 basic_session_run_hooks.py:260] loss = 2.5502517, step = 5400 (82.653 sec) INFO:tensorflow:global_step/sec: 1.89657 I0419 23:30:12.946993 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89657 INFO:tensorflow:loss = 1.6301205, step = 5500 (52.727 sec) I0419 23:30:12.948405 140176207116096 basic_session_run_hooks.py:260] loss = 1.6301205, step = 5500 (52.727 sec) INFO:tensorflow:global_step/sec: 1.88175 I0419 23:31:06.089153 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88175 INFO:tensorflow:loss = 3.1102018, step = 5600 (53.142 sec) I0419 23:31:06.090466 140176207116096 basic_session_run_hooks.py:260] loss = 3.1102018, step = 5600 (53.142 sec) INFO:tensorflow:global_step/sec: 1.87911 I0419 23:31:59.305747 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87911 INFO:tensorflow:loss = 2.386134, step = 5700 (53.217 sec) I0419 23:31:59.307236 140176207116096 basic_session_run_hooks.py:260] loss = 2.386134, step = 5700 (53.217 sec) INFO:tensorflow:global_step/sec: 1.87826 I0419 23:32:52.546614 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87826 INFO:tensorflow:loss = 2.087537, step = 5800 (53.241 sec) I0419 23:32:52.547915 140176207116096 basic_session_run_hooks.py:260] loss = 2.087537, step = 5800 (53.241 sec) INFO:tensorflow:global_step/sec: 1.88597 I0419 23:33:45.569667 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88597 INFO:tensorflow:loss = 2.806619, step = 5900 (53.023 sec) I0419 23:33:45.570889 140176207116096 basic_session_run_hooks.py:260] loss = 2.806619, step = 5900 (53.023 sec) INFO:tensorflow:global_step/sec: 1.9036 I0419 23:34:38.101715 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.9036 INFO:tensorflow:loss = 1.8371814, step = 6000 (52.532 sec) I0419 23:34:38.103190 140176207116096 basic_session_run_hooks.py:260] loss = 1.8371814, step = 6000 (52.532 sec) INFO:tensorflow:global_step/sec: 1.89638 I0419 23:35:30.833801 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89638 INFO:tensorflow:loss = 1.7703884, step = 6100 (52.732 sec) I0419 23:35:30.834982 140176207116096 basic_session_run_hooks.py:260] loss = 1.7703884, step = 6100 (52.732 sec) INFO:tensorflow:global_step/sec: 1.86501 I0419 23:36:24.452745 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86501 INFO:tensorflow:loss = 2.2324133, step = 6200 (53.619 sec) I0419 23:36:24.453965 140176207116096 basic_session_run_hooks.py:260] loss = 2.2324133, step = 6200 (53.619 sec) INFO:tensorflow:global_step/sec: 1.89592 I0419 23:37:17.197700 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89592 INFO:tensorflow:loss = 2.321603, step = 6300 (52.745 sec) I0419 23:37:17.199027 140176207116096 basic_session_run_hooks.py:260] loss = 2.321603, step = 6300 (52.745 sec) INFO:tensorflow:Saving checkpoints for 6382 into ./my_model_dir/model.ckpt. I0419 23:37:59.946388 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 6382 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0419 23:38:01.784521 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0419 23:38:01.785421 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0419 23:38:01.785647 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0419 23:38:02.577707 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:38:05.392861 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:38:05.478450 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:38:05.569944 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:38:05.651607 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:38:05.740188 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:38:05.822724 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0419 23:38:07.353411 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-19T23:38:07Z I0419 23:38:07.370053 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-19T23:38:07Z INFO:tensorflow:Graph was finalized. I0419 23:38:07.880291 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-19 23:38:07.881216: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:38:07.881757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-19 23:38:07.881907: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 23:38:07.881937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-19 23:38:07.881963: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-19 23:38:07.881997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-19 23:38:07.882024: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-19 23:38:07.882049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-19 23:38:07.882072: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 23:38:07.882135: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:38:07.882549: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:38:07.882888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-19 23:38:07.882934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-19 23:38:07.882943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-19 23:38:07.882948: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-19 23:38:07.883065: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:38:07.883478: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:38:07.883826: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-6382 I0419 23:38:07.885110 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-6382 INFO:tensorflow:Running local_init_op. I0419 23:38:09.141309 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0419 23:38:09.288050 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0419 23:38:28.456818 140169104418560 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0419 23:38:28.460231 140169104418560 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0419 23:38:28.484007 140169104418560 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=0.93s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.004 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.004 INFO:tensorflow:Finished evaluation at 2023-04-19-23:38:29 I0419 23:38:29.679768 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-19-23:38:29 INFO:tensorflow:Saving dict for global step 6382: DetectionBoxes_Precision/mAP = 1.6697388e-05, DetectionBoxes_Precision/mAP (large) = 1.6697388e-05, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 9.930834e-05, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0023012552, DetectionBoxes_Recall/AR@100 = 0.0038397168, DetectionBoxes_Recall/AR@100 (large) = 0.0039405995, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.5260773, Loss/localization_loss = 2.5119627, Loss/regularization_loss = 0.28676197, Loss/total_loss = 9.324803, global_step = 6382, learning_rate = 0.004, loss = 9.324803 I0419 23:38:29.680176 140176207116096 estimator.py:2049] Saving dict for global step 6382: DetectionBoxes_Precision/mAP = 1.6697388e-05, DetectionBoxes_Precision/mAP (large) = 1.6697388e-05, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 9.930834e-05, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0023012552, DetectionBoxes_Recall/AR@100 = 0.0038397168, DetectionBoxes_Recall/AR@100 (large) = 0.0039405995, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.5260773, Loss/localization_loss = 2.5119627, Loss/regularization_loss = 0.28676197, Loss/total_loss = 9.324803, global_step = 6382, learning_rate = 0.004, loss = 9.324803 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 6382: ./my_model_dir/model.ckpt-6382 I0419 23:38:29.687180 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 6382: ./my_model_dir/model.ckpt-6382 INFO:tensorflow:global_step/sec: 1.20934 I0419 23:38:39.887707 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.20934 INFO:tensorflow:loss = 2.2973838, step = 6400 (82.690 sec) I0419 23:38:39.888923 140176207116096 basic_session_run_hooks.py:260] loss = 2.2973838, step = 6400 (82.690 sec) INFO:tensorflow:global_step/sec: 1.91387 I0419 23:39:32.137699 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.91387 INFO:tensorflow:loss = 2.186744, step = 6500 (52.250 sec) I0419 23:39:32.139195 140176207116096 basic_session_run_hooks.py:260] loss = 2.186744, step = 6500 (52.250 sec) INFO:tensorflow:global_step/sec: 1.9064 I0419 23:40:24.592782 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.9064 INFO:tensorflow:loss = 2.1141324, step = 6600 (52.455 sec) I0419 23:40:24.593989 140176207116096 basic_session_run_hooks.py:260] loss = 2.1141324, step = 6600 (52.455 sec) INFO:tensorflow:global_step/sec: 1.89193 I0419 23:41:17.448684 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89193 INFO:tensorflow:loss = 2.2692838, step = 6700 (52.856 sec) I0419 23:41:17.450010 140176207116096 basic_session_run_hooks.py:260] loss = 2.2692838, step = 6700 (52.856 sec) INFO:tensorflow:global_step/sec: 1.88287 I0419 23:42:10.559108 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88287 INFO:tensorflow:loss = 1.734136, step = 6800 (53.110 sec) I0419 23:42:10.560498 140176207116096 basic_session_run_hooks.py:260] loss = 1.734136, step = 6800 (53.110 sec) INFO:tensorflow:global_step/sec: 1.88442 I0419 23:43:03.625883 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88442 INFO:tensorflow:loss = 1.8282249, step = 6900 (53.067 sec) I0419 23:43:03.627152 140176207116096 basic_session_run_hooks.py:260] loss = 1.8282249, step = 6900 (53.067 sec) INFO:tensorflow:global_step/sec: 1.9116 I0419 23:43:55.938143 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.9116 INFO:tensorflow:loss = 1.8988017, step = 7000 (52.312 sec) I0419 23:43:55.939393 140176207116096 basic_session_run_hooks.py:260] loss = 1.8988017, step = 7000 (52.312 sec) INFO:tensorflow:global_step/sec: 1.89149 I0419 23:44:48.806454 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89149 INFO:tensorflow:loss = 1.8165348, step = 7100 (52.868 sec) I0419 23:44:48.807829 140176207116096 basic_session_run_hooks.py:260] loss = 1.8165348, step = 7100 (52.868 sec) INFO:tensorflow:global_step/sec: 1.89686 I0419 23:45:41.525053 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89686 INFO:tensorflow:loss = 1.8219991, step = 7200 (52.718 sec) I0419 23:45:41.526290 140176207116096 basic_session_run_hooks.py:260] loss = 1.8219991, step = 7200 (52.718 sec) INFO:tensorflow:global_step/sec: 1.90857 I0419 23:46:33.920267 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90857 INFO:tensorflow:loss = 2.2252913, step = 7300 (52.395 sec) I0419 23:46:33.921509 140176207116096 basic_session_run_hooks.py:260] loss = 2.2252913, step = 7300 (52.395 sec) INFO:tensorflow:global_step/sec: 1.87818 I0419 23:47:27.163299 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87818 INFO:tensorflow:loss = 1.928407, step = 7400 (53.243 sec) I0419 23:47:27.164596 140176207116096 basic_session_run_hooks.py:260] loss = 1.928407, step = 7400 (53.243 sec) INFO:tensorflow:Saving checkpoints for 7464 into ./my_model_dir/model.ckpt. I0419 23:48:00.289097 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 7464 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0419 23:48:02.028231 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0419 23:48:02.029088 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0419 23:48:02.029308 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0419 23:48:02.829450 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:48:05.111389 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:48:05.199854 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:48:05.294550 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:48:05.377722 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:48:05.464595 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:48:05.549768 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0419 23:48:07.685536 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-19T23:48:07Z I0419 23:48:07.703996 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-19T23:48:07Z INFO:tensorflow:Graph was finalized. I0419 23:48:08.236517 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-19 23:48:08.237438: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:48:08.237951: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-19 23:48:08.238102: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 23:48:08.238134: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-19 23:48:08.238160: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-19 23:48:08.238184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-19 23:48:08.238208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-19 23:48:08.238232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-19 23:48:08.238257: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 23:48:08.238322: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:48:08.238737: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:48:08.239089: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-19 23:48:08.239133: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-19 23:48:08.239142: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-19 23:48:08.239148: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-19 23:48:08.239257: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:48:08.239672: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:48:08.240035: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-7464 I0419 23:48:08.242201 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-7464 INFO:tensorflow:Running local_init_op. I0419 23:48:09.493618 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0419 23:48:09.655436 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0419 23:48:29.000952 140168203859712 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0419 23:48:29.004950 140168203859712 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0419 23:48:29.025473 140168203859712 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=0.81s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.002 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.004 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.004 INFO:tensorflow:Finished evaluation at 2023-04-19-23:48:30 I0419 23:48:30.104292 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-19-23:48:30 INFO:tensorflow:Saving dict for global step 7464: DetectionBoxes_Precision/mAP = 2.0770984e-05, DetectionBoxes_Precision/mAP (large) = 2.0770984e-05, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.000114605595, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0023012552, DetectionBoxes_Recall/AR@100 = 0.0038397168, DetectionBoxes_Recall/AR@100 (large) = 0.0039405995, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.7856784, Loss/localization_loss = 2.4915361, Loss/regularization_loss = 0.28686628, Loss/total_loss = 9.564079, global_step = 7464, learning_rate = 0.004, loss = 9.564079 I0419 23:48:30.104649 140176207116096 estimator.py:2049] Saving dict for global step 7464: DetectionBoxes_Precision/mAP = 2.0770984e-05, DetectionBoxes_Precision/mAP (large) = 2.0770984e-05, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.000114605595, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0023012552, DetectionBoxes_Recall/AR@100 = 0.0038397168, DetectionBoxes_Recall/AR@100 (large) = 0.0039405995, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.7856784, Loss/localization_loss = 2.4915361, Loss/regularization_loss = 0.28686628, Loss/total_loss = 9.564079, global_step = 7464, learning_rate = 0.004, loss = 9.564079 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 7464: ./my_model_dir/model.ckpt-7464 I0419 23:48:30.111329 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 7464: ./my_model_dir/model.ckpt-7464 INFO:tensorflow:global_step/sec: 1.20832 I0419 23:48:49.922891 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.20832 INFO:tensorflow:loss = 1.7600393, step = 7500 (82.760 sec) I0419 23:48:49.924305 140176207116096 basic_session_run_hooks.py:260] loss = 1.7600393, step = 7500 (82.760 sec) INFO:tensorflow:global_step/sec: 1.9078 I0419 23:49:42.339248 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.9078 INFO:tensorflow:loss = 1.8422657, step = 7600 (52.416 sec) I0419 23:49:42.340801 140176207116096 basic_session_run_hooks.py:260] loss = 1.8422657, step = 7600 (52.416 sec) INFO:tensorflow:global_step/sec: 1.89836 I0419 23:50:35.016264 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89836 INFO:tensorflow:loss = 2.7143445, step = 7700 (52.677 sec) I0419 23:50:35.017571 140176207116096 basic_session_run_hooks.py:260] loss = 2.7143445, step = 7700 (52.677 sec) INFO:tensorflow:global_step/sec: 1.89925 I0419 23:51:27.668519 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89925 INFO:tensorflow:loss = 1.8896121, step = 7800 (52.652 sec) I0419 23:51:27.669758 140176207116096 basic_session_run_hooks.py:260] loss = 1.8896121, step = 7800 (52.652 sec) INFO:tensorflow:global_step/sec: 1.90774 I0419 23:52:20.086592 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90774 INFO:tensorflow:loss = 2.113993, step = 7900 (52.418 sec) I0419 23:52:20.088003 140176207116096 basic_session_run_hooks.py:260] loss = 2.113993, step = 7900 (52.418 sec) INFO:tensorflow:global_step/sec: 1.90993 I0419 23:53:12.444595 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90993 INFO:tensorflow:loss = 2.8427424, step = 8000 (52.358 sec) I0419 23:53:12.445846 140176207116096 basic_session_run_hooks.py:260] loss = 2.8427424, step = 8000 (52.358 sec) INFO:tensorflow:global_step/sec: 1.90166 I0419 23:54:05.030221 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90166 INFO:tensorflow:loss = 2.197617, step = 8100 (52.586 sec) I0419 23:54:05.031441 140176207116096 basic_session_run_hooks.py:260] loss = 2.197617, step = 8100 (52.586 sec) INFO:tensorflow:global_step/sec: 1.89872 I0419 23:54:57.697280 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89872 INFO:tensorflow:loss = 2.2459867, step = 8200 (52.667 sec) I0419 23:54:57.698790 140176207116096 basic_session_run_hooks.py:260] loss = 2.2459867, step = 8200 (52.667 sec) INFO:tensorflow:global_step/sec: 1.9023 I0419 23:55:50.265051 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.9023 INFO:tensorflow:loss = 1.9537593, step = 8300 (52.568 sec) I0419 23:55:50.266329 140176207116096 basic_session_run_hooks.py:260] loss = 1.9537593, step = 8300 (52.568 sec) INFO:tensorflow:global_step/sec: 1.92775 I0419 23:56:42.139127 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.92775 INFO:tensorflow:loss = 2.0903904, step = 8400 (51.874 sec) I0419 23:56:42.140441 140176207116096 basic_session_run_hooks.py:260] loss = 2.0903904, step = 8400 (51.874 sec) INFO:tensorflow:global_step/sec: 1.91027 I0419 23:57:34.487707 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.91027 INFO:tensorflow:loss = 2.0503836, step = 8500 (52.349 sec) I0419 23:57:34.489065 140176207116096 basic_session_run_hooks.py:260] loss = 2.0503836, step = 8500 (52.349 sec) INFO:tensorflow:Saving checkpoints for 8550 into ./my_model_dir/model.ckpt. I0419 23:58:00.329441 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 8550 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0419 23:58:02.086053 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0419 23:58:02.086994 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0419 23:58:02.087223 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0419 23:58:02.891808 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:58:05.167762 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:58:05.250877 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:58:05.341656 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:58:05.427858 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:58:05.511314 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0419 23:58:05.594381 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0419 23:58:07.170614 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-19T23:58:07Z I0419 23:58:07.187603 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-19T23:58:07Z INFO:tensorflow:Graph was finalized. I0419 23:58:07.706618 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-19 23:58:07.707773: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:58:07.708288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-19 23:58:07.708460: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-19 23:58:07.708493: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-19 23:58:07.708518: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-19 23:58:07.708541: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-19 23:58:07.708564: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-19 23:58:07.708586: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-19 23:58:07.708610: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-19 23:58:07.708686: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:58:07.709113: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:58:07.709486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-19 23:58:07.709532: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-19 23:58:07.709541: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-19 23:58:07.709547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-19 23:58:07.709661: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:58:07.710071: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-19 23:58:07.710446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-8550 I0419 23:58:07.711483 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-8550 INFO:tensorflow:Running local_init_op. I0419 23:58:08.923938 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0419 23:58:09.061475 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0419 23:58:28.151640 140169129596672 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0419 23:58:28.206187 140169129596672 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0419 23:58:28.228164 140169129596672 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=0.95s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.003 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.005 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.005 INFO:tensorflow:Finished evaluation at 2023-04-19-23:58:29 I0419 23:58:29.457402 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-19-23:58:29 INFO:tensorflow:Saving dict for global step 8550: DetectionBoxes_Precision/mAP = 3.3290864e-05, DetectionBoxes_Precision/mAP (large) = 3.3290864e-05, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.00014028576, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0025104603, DetectionBoxes_Recall/AR@100 = 0.0051995493, DetectionBoxes_Recall/AR@100 (large) = 0.005300432, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.102853, Loss/localization_loss = 2.4366343, Loss/regularization_loss = 0.28691784, Loss/total_loss = 9.8264065, global_step = 8550, learning_rate = 0.004, loss = 9.8264065 I0419 23:58:29.457819 140176207116096 estimator.py:2049] Saving dict for global step 8550: DetectionBoxes_Precision/mAP = 3.3290864e-05, DetectionBoxes_Precision/mAP (large) = 3.3290864e-05, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.00014028576, DetectionBoxes_Precision/mAP@.75IOU = 0.0, DetectionBoxes_Recall/AR@1 = 0.0, DetectionBoxes_Recall/AR@10 = 0.0025104603, DetectionBoxes_Recall/AR@100 = 0.0051995493, DetectionBoxes_Recall/AR@100 (large) = 0.005300432, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.102853, Loss/localization_loss = 2.4366343, Loss/regularization_loss = 0.28691784, Loss/total_loss = 9.8264065, global_step = 8550, learning_rate = 0.004, loss = 9.8264065 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 8550: ./my_model_dir/model.ckpt-8550 I0419 23:58:29.465494 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 8550: ./my_model_dir/model.ckpt-8550 INFO:tensorflow:global_step/sec: 1.21693 I0419 23:58:56.661468 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.21693 INFO:tensorflow:loss = 1.752513, step = 8600 (82.174 sec) I0419 23:58:56.662945 140176207116096 basic_session_run_hooks.py:260] loss = 1.752513, step = 8600 (82.174 sec) INFO:tensorflow:global_step/sec: 1.89473 I0419 23:59:49.439280 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89473 INFO:tensorflow:loss = 1.5862393, step = 8700 (52.778 sec) I0419 23:59:49.440647 140176207116096 basic_session_run_hooks.py:260] loss = 1.5862393, step = 8700 (52.778 sec) INFO:tensorflow:global_step/sec: 1.89174 I0420 00:00:42.300601 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89174 INFO:tensorflow:loss = 1.7200961, step = 8800 (52.861 sec) I0420 00:00:42.302026 140176207116096 basic_session_run_hooks.py:260] loss = 1.7200961, step = 8800 (52.861 sec) INFO:tensorflow:global_step/sec: 1.89093 I0420 00:01:35.184620 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89093 INFO:tensorflow:loss = 2.5098932, step = 8900 (52.884 sec) I0420 00:01:35.185884 140176207116096 basic_session_run_hooks.py:260] loss = 2.5098932, step = 8900 (52.884 sec) INFO:tensorflow:global_step/sec: 1.9051 I0420 00:02:27.675213 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.9051 INFO:tensorflow:loss = 1.6835722, step = 9000 (52.491 sec) I0420 00:02:27.676618 140176207116096 basic_session_run_hooks.py:260] loss = 1.6835722, step = 9000 (52.491 sec) INFO:tensorflow:global_step/sec: 1.89434 I0420 00:03:20.464075 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89434 INFO:tensorflow:loss = 2.3483174, step = 9100 (52.789 sec) I0420 00:03:20.465397 140176207116096 basic_session_run_hooks.py:260] loss = 2.3483174, step = 9100 (52.789 sec) INFO:tensorflow:global_step/sec: 1.89118 I0420 00:04:13.341044 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89118 INFO:tensorflow:loss = 2.1451762, step = 9200 (52.877 sec) I0420 00:04:13.342290 140176207116096 basic_session_run_hooks.py:260] loss = 2.1451762, step = 9200 (52.877 sec) INFO:tensorflow:global_step/sec: 1.90514 I0420 00:05:05.830506 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90514 INFO:tensorflow:loss = 2.025448, step = 9300 (52.490 sec) I0420 00:05:05.832380 140176207116096 basic_session_run_hooks.py:260] loss = 2.025448, step = 9300 (52.490 sec) INFO:tensorflow:global_step/sec: 1.90434 I0420 00:05:58.342192 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90434 INFO:tensorflow:loss = 2.2712533, step = 9400 (52.511 sec) I0420 00:05:58.343467 140176207116096 basic_session_run_hooks.py:260] loss = 2.2712533, step = 9400 (52.511 sec) INFO:tensorflow:global_step/sec: 1.8876 I0420 00:06:51.319413 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8876 INFO:tensorflow:loss = 2.2412102, step = 9500 (52.977 sec) I0420 00:06:51.320634 140176207116096 basic_session_run_hooks.py:260] loss = 2.2412102, step = 9500 (52.977 sec) INFO:tensorflow:global_step/sec: 1.89405 I0420 00:07:44.116205 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89405 INFO:tensorflow:loss = 2.1685345, step = 9600 (52.797 sec) I0420 00:07:44.117433 140176207116096 basic_session_run_hooks.py:260] loss = 2.1685345, step = 9600 (52.797 sec) INFO:tensorflow:Saving checkpoints for 9632 into ./my_model_dir/model.ckpt. I0420 00:08:00.412422 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 9632 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 00:08:02.184249 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 00:08:02.185173 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 00:08:02.185410 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 00:08:02.990609 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:08:05.267848 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:08:05.353652 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:08:05.447346 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:08:05.529847 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:08:05.615040 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:08:05.699451 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 00:08:07.268095 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T00:08:07Z I0420 00:08:07.285013 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T00:08:07Z INFO:tensorflow:Graph was finalized. I0420 00:08:07.809017 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 00:08:07.810086: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:08:07.810612: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 00:08:07.810757: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 00:08:07.810787: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 00:08:07.810813: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 00:08:07.810837: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 00:08:07.810859: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 00:08:07.810881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 00:08:07.810906: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 00:08:07.810972: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:08:07.811385: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:08:07.811734: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 00:08:07.811779: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 00:08:07.811788: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 00:08:07.811793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 00:08:07.811895: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:08:07.812545: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:08:07.813263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-9632 I0420 00:08:07.814527 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-9632 INFO:tensorflow:Running local_init_op. I0420 00:08:09.043209 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 00:08:09.189503 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 00:08:28.471762 140170502313728 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 00:08:28.479569 140170502313728 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0420 00:08:28.504474 140170502313728 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.00s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.013 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.024 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.014 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.013 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.011 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.017 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.061 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.061 INFO:tensorflow:Finished evaluation at 2023-04-20-00:08:29 I0420 00:08:29.779731 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-00:08:29 INFO:tensorflow:Saving dict for global step 9632: DetectionBoxes_Precision/mAP = 0.012837523, DetectionBoxes_Precision/mAP (large) = 0.012837523, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.024380332, DetectionBoxes_Precision/mAP@.75IOU = 0.014390765, DetectionBoxes_Recall/AR@1 = 0.010564853, DetectionBoxes_Recall/AR@10 = 0.016527196, DetectionBoxes_Recall/AR@100 = 0.06105729, DetectionBoxes_Recall/AR@100 (large) = 0.061158173, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.058277, Loss/localization_loss = 2.3164737, Loss/regularization_loss = 0.28693745, Loss/total_loss = 9.661688, global_step = 9632, learning_rate = 0.004, loss = 9.661688 I0420 00:08:29.780230 140176207116096 estimator.py:2049] Saving dict for global step 9632: DetectionBoxes_Precision/mAP = 0.012837523, DetectionBoxes_Precision/mAP (large) = 0.012837523, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.024380332, DetectionBoxes_Precision/mAP@.75IOU = 0.014390765, DetectionBoxes_Recall/AR@1 = 0.010564853, DetectionBoxes_Recall/AR@10 = 0.016527196, DetectionBoxes_Recall/AR@100 = 0.06105729, DetectionBoxes_Recall/AR@100 (large) = 0.061158173, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 7.058277, Loss/localization_loss = 2.3164737, Loss/regularization_loss = 0.28693745, Loss/total_loss = 9.661688, global_step = 9632, learning_rate = 0.004, loss = 9.661688 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 9632: ./my_model_dir/model.ckpt-9632 I0420 00:08:29.788120 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 9632: ./my_model_dir/model.ckpt-9632 INFO:tensorflow:global_step/sec: 1.2136 I0420 00:09:06.515363 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.2136 INFO:tensorflow:loss = 1.8332667, step = 9700 (82.399 sec) I0420 00:09:06.516736 140176207116096 basic_session_run_hooks.py:260] loss = 1.8332667, step = 9700 (82.399 sec) INFO:tensorflow:global_step/sec: 1.90763 I0420 00:09:58.936556 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90763 INFO:tensorflow:loss = 2.1878524, step = 9800 (52.421 sec) I0420 00:09:58.937931 140176207116096 basic_session_run_hooks.py:260] loss = 2.1878524, step = 9800 (52.421 sec) INFO:tensorflow:global_step/sec: 1.88932 I0420 00:10:51.865534 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88932 INFO:tensorflow:loss = 1.8619821, step = 9900 (52.929 sec) I0420 00:10:51.866991 140176207116096 basic_session_run_hooks.py:260] loss = 1.8619821, step = 9900 (52.929 sec) INFO:tensorflow:global_step/sec: 1.88795 I0420 00:11:44.833036 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88795 INFO:tensorflow:loss = 2.313528, step = 10000 (52.967 sec) I0420 00:11:44.834460 140176207116096 basic_session_run_hooks.py:260] loss = 2.313528, step = 10000 (52.967 sec) INFO:tensorflow:global_step/sec: 1.87681 I0420 00:12:38.114839 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87681 INFO:tensorflow:loss = 1.8933196, step = 10100 (53.282 sec) I0420 00:12:38.116191 140176207116096 basic_session_run_hooks.py:260] loss = 1.8933196, step = 10100 (53.282 sec) INFO:tensorflow:global_step/sec: 1.89306 I0420 00:13:30.939269 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89306 INFO:tensorflow:loss = 1.9823947, step = 10200 (52.824 sec) I0420 00:13:30.940620 140176207116096 basic_session_run_hooks.py:260] loss = 1.9823947, step = 10200 (52.824 sec) INFO:tensorflow:global_step/sec: 1.89759 I0420 00:14:23.637569 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89759 INFO:tensorflow:loss = 2.5072093, step = 10300 (52.698 sec) I0420 00:14:23.638762 140176207116096 basic_session_run_hooks.py:260] loss = 2.5072093, step = 10300 (52.698 sec) INFO:tensorflow:global_step/sec: 1.8942 I0420 00:15:16.430251 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8942 INFO:tensorflow:loss = 2.0279512, step = 10400 (52.793 sec) I0420 00:15:16.431550 140176207116096 basic_session_run_hooks.py:260] loss = 2.0279512, step = 10400 (52.793 sec) INFO:tensorflow:global_step/sec: 1.90912 I0420 00:16:08.810423 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90912 INFO:tensorflow:loss = 1.8848281, step = 10500 (52.380 sec) I0420 00:16:08.811653 140176207116096 basic_session_run_hooks.py:260] loss = 1.8848281, step = 10500 (52.380 sec) INFO:tensorflow:global_step/sec: 1.88699 I0420 00:17:01.804857 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88699 INFO:tensorflow:loss = 2.0088744, step = 10600 (52.995 sec) I0420 00:17:01.806253 140176207116096 basic_session_run_hooks.py:260] loss = 2.0088744, step = 10600 (52.995 sec) INFO:tensorflow:global_step/sec: 1.89687 I0420 00:17:54.523320 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89687 INFO:tensorflow:loss = 1.767856, step = 10700 (52.718 sec) I0420 00:17:54.524456 140176207116096 basic_session_run_hooks.py:260] loss = 1.767856, step = 10700 (52.718 sec) INFO:tensorflow:Saving checkpoints for 10713 into ./my_model_dir/model.ckpt. I0420 00:18:00.873421 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 10713 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 00:18:02.810395 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 00:18:02.811224 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 00:18:02.811473 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 00:18:03.606454 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:18:06.475554 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:18:06.564357 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:18:06.658094 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:18:06.743751 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:18:06.826506 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:18:06.911632 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 00:18:08.436287 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T00:18:08Z I0420 00:18:08.452999 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T00:18:08Z INFO:tensorflow:Graph was finalized. I0420 00:18:08.962640 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 00:18:08.963501: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:18:08.963993: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 00:18:08.964142: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 00:18:08.964171: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 00:18:08.964196: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 00:18:08.964219: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 00:18:08.964241: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 00:18:08.964263: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 00:18:08.964286: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 00:18:08.964360: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:18:08.964770: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:18:08.965109: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 00:18:08.965153: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 00:18:08.965162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 00:18:08.965167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 00:18:08.965262: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:18:08.965676: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:18:08.966036: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-10713 I0420 00:18:08.967700 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-10713 INFO:tensorflow:Running local_init_op. I0420 00:18:10.210918 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 00:18:10.356184 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 00:18:29.271921 140169129596672 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 00:18:29.275607 140169129596672 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.03s) I0420 00:18:29.301136 140169129596672 coco_tools.py:138] DONE (t=0.03s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.07s). Accumulating evaluation results... DONE (t=0.16s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.064 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.149 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.042 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.064 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.058 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.094 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.175 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.177 INFO:tensorflow:Finished evaluation at 2023-04-20-00:18:30 I0420 00:18:30.696495 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-00:18:30 INFO:tensorflow:Saving dict for global step 10713: DetectionBoxes_Precision/mAP = 0.063934036, DetectionBoxes_Precision/mAP (large) = 0.06395804, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.14859658, DetectionBoxes_Precision/mAP@.75IOU = 0.042059276, DetectionBoxes_Recall/AR@1 = 0.057784036, DetectionBoxes_Recall/AR@10 = 0.093780175, DetectionBoxes_Recall/AR@100 = 0.1751384, DetectionBoxes_Recall/AR@100 (large) = 0.17675252, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.495552, Loss/localization_loss = 2.1084301, Loss/regularization_loss = 0.28689092, Loss/total_loss = 8.890872, global_step = 10713, learning_rate = 0.0037999998, loss = 8.890872 I0420 00:18:30.696881 140176207116096 estimator.py:2049] Saving dict for global step 10713: DetectionBoxes_Precision/mAP = 0.063934036, DetectionBoxes_Precision/mAP (large) = 0.06395804, DetectionBoxes_Precision/mAP (medium) = 0.0, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.14859658, DetectionBoxes_Precision/mAP@.75IOU = 0.042059276, DetectionBoxes_Recall/AR@1 = 0.057784036, DetectionBoxes_Recall/AR@10 = 0.093780175, DetectionBoxes_Recall/AR@100 = 0.1751384, DetectionBoxes_Recall/AR@100 (large) = 0.17675252, DetectionBoxes_Recall/AR@100 (medium) = 0.0, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 6.495552, Loss/localization_loss = 2.1084301, Loss/regularization_loss = 0.28689092, Loss/total_loss = 8.890872, global_step = 10713, learning_rate = 0.0037999998, loss = 8.890872 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 10713: ./my_model_dir/model.ckpt-10713 I0420 00:18:30.704396 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 10713: ./my_model_dir/model.ckpt-10713 INFO:tensorflow:global_step/sec: 1.21067 I0420 00:19:17.122538 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.21067 INFO:tensorflow:loss = 2.0141785, step = 10800 (82.599 sec) I0420 00:19:17.123864 140176207116096 basic_session_run_hooks.py:260] loss = 2.0141785, step = 10800 (82.599 sec) INFO:tensorflow:global_step/sec: 1.87195 I0420 00:20:10.542728 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87195 INFO:tensorflow:loss = 1.7216241, step = 10900 (53.420 sec) I0420 00:20:10.544188 140176207116096 basic_session_run_hooks.py:260] loss = 1.7216241, step = 10900 (53.420 sec) INFO:tensorflow:global_step/sec: 1.90131 I0420 00:21:03.138041 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90131 INFO:tensorflow:loss = 1.7743999, step = 11000 (52.595 sec) I0420 00:21:03.139648 140176207116096 basic_session_run_hooks.py:260] loss = 1.7743999, step = 11000 (52.595 sec) INFO:tensorflow:global_step/sec: 1.90149 I0420 00:21:55.728544 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90149 INFO:tensorflow:loss = 1.6336315, step = 11100 (52.591 sec) I0420 00:21:55.730179 140176207116096 basic_session_run_hooks.py:260] loss = 1.6336315, step = 11100 (52.591 sec) INFO:tensorflow:global_step/sec: 1.87566 I0420 00:22:49.042998 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87566 INFO:tensorflow:loss = 1.3822913, step = 11200 (53.314 sec) I0420 00:22:49.044410 140176207116096 basic_session_run_hooks.py:260] loss = 1.3822913, step = 11200 (53.314 sec) INFO:tensorflow:global_step/sec: 1.90143 I0420 00:23:41.634958 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90143 INFO:tensorflow:loss = 1.8591826, step = 11300 (52.592 sec) I0420 00:23:41.636311 140176207116096 basic_session_run_hooks.py:260] loss = 1.8591826, step = 11300 (52.592 sec) INFO:tensorflow:global_step/sec: 1.88838 I0420 00:24:34.590422 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88838 INFO:tensorflow:loss = 1.5610814, step = 11400 (52.955 sec) I0420 00:24:34.591738 140176207116096 basic_session_run_hooks.py:260] loss = 1.5610814, step = 11400 (52.955 sec) INFO:tensorflow:global_step/sec: 1.88189 I0420 00:25:27.728533 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88189 INFO:tensorflow:loss = 1.6501026, step = 11500 (53.138 sec) I0420 00:25:27.730045 140176207116096 basic_session_run_hooks.py:260] loss = 1.6501026, step = 11500 (53.138 sec) INFO:tensorflow:global_step/sec: 1.87702 I0420 00:26:21.004431 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87702 INFO:tensorflow:loss = 2.4809031, step = 11600 (53.276 sec) I0420 00:26:21.006466 140176207116096 basic_session_run_hooks.py:260] loss = 2.4809031, step = 11600 (53.276 sec) INFO:tensorflow:global_step/sec: 1.88985 I0420 00:27:13.918598 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88985 INFO:tensorflow:loss = 1.8326727, step = 11700 (52.913 sec) I0420 00:27:13.919862 140176207116096 basic_session_run_hooks.py:260] loss = 1.8326727, step = 11700 (52.913 sec) INFO:tensorflow:Saving checkpoints for 11791 into ./my_model_dir/model.ckpt. I0420 00:28:01.081465 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 11791 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 00:28:02.939824 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 00:28:02.940617 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 00:28:02.940810 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 00:28:03.738487 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:28:06.065250 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:28:06.155125 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:28:06.253194 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:28:06.338252 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:28:06.423444 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:28:06.511714 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 00:28:08.522100 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T00:28:08Z I0420 00:28:08.539255 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T00:28:08Z INFO:tensorflow:Graph was finalized. I0420 00:28:09.073571 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 00:28:09.074502: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:28:09.075001: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 00:28:09.075148: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 00:28:09.075180: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 00:28:09.075206: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 00:28:09.075230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 00:28:09.075268: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 00:28:09.075296: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 00:28:09.075320: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 00:28:09.075399: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:28:09.075815: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:28:09.076155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 00:28:09.076210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 00:28:09.076221: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 00:28:09.076227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 00:28:09.076355: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:28:09.076767: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:28:09.077119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-11791 I0420 00:28:09.078654 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-11791 INFO:tensorflow:Running local_init_op. I0420 00:28:10.372432 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 00:28:10.511715 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 00:28:29.968362 140169137989376 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 00:28:30.008084 140169137989376 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.05s) I0420 00:28:30.061391 140169137989376 coco_tools.py:138] DONE (t=0.05s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.21s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.169 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.392 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.111 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.013 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.171 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.136 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.264 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.325 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.025 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.330 INFO:tensorflow:Finished evaluation at 2023-04-20-00:28:31 I0420 00:28:31.542941 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-00:28:31 INFO:tensorflow:Saving dict for global step 11791: DetectionBoxes_Precision/mAP = 0.16860576, DetectionBoxes_Precision/mAP (large) = 0.17081845, DetectionBoxes_Precision/mAP (medium) = 0.012871287, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.39230657, DetectionBoxes_Precision/mAP@.75IOU = 0.11145756, DetectionBoxes_Recall/AR@1 = 0.13647892, DetectionBoxes_Recall/AR@10 = 0.2638864, DetectionBoxes_Recall/AR@100 = 0.3250724, DetectionBoxes_Recall/AR@100 (large) = 0.3300535, DetectionBoxes_Recall/AR@100 (medium) = 0.025, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 4.449712, Loss/localization_loss = 1.6307604, Loss/regularization_loss = 0.2868014, Loss/total_loss = 6.367274, global_step = 11791, learning_rate = 0.0037999998, loss = 6.367274 I0420 00:28:31.543343 140176207116096 estimator.py:2049] Saving dict for global step 11791: DetectionBoxes_Precision/mAP = 0.16860576, DetectionBoxes_Precision/mAP (large) = 0.17081845, DetectionBoxes_Precision/mAP (medium) = 0.012871287, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.39230657, DetectionBoxes_Precision/mAP@.75IOU = 0.11145756, DetectionBoxes_Recall/AR@1 = 0.13647892, DetectionBoxes_Recall/AR@10 = 0.2638864, DetectionBoxes_Recall/AR@100 = 0.3250724, DetectionBoxes_Recall/AR@100 (large) = 0.3300535, DetectionBoxes_Recall/AR@100 (medium) = 0.025, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 4.449712, Loss/localization_loss = 1.6307604, Loss/regularization_loss = 0.2868014, Loss/total_loss = 6.367274, global_step = 11791, learning_rate = 0.0037999998, loss = 6.367274 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 11791: ./my_model_dir/model.ckpt-11791 I0420 00:28:31.550428 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 11791: ./my_model_dir/model.ckpt-11791 INFO:tensorflow:global_step/sec: 1.20438 I0420 00:28:36.949375 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.20438 INFO:tensorflow:loss = 1.3397305, step = 11800 (83.031 sec) I0420 00:28:36.950762 140176207116096 basic_session_run_hooks.py:260] loss = 1.3397305, step = 11800 (83.031 sec) INFO:tensorflow:global_step/sec: 1.90114 I0420 00:29:29.548841 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90114 INFO:tensorflow:loss = 1.7899709, step = 11900 (52.604 sec) I0420 00:29:29.554871 140176207116096 basic_session_run_hooks.py:260] loss = 1.7899709, step = 11900 (52.604 sec) INFO:tensorflow:global_step/sec: 1.89878 I0420 00:30:22.214474 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89878 INFO:tensorflow:loss = 1.8710512, step = 12000 (52.661 sec) I0420 00:30:22.215900 140176207116096 basic_session_run_hooks.py:260] loss = 1.8710512, step = 12000 (52.661 sec) INFO:tensorflow:global_step/sec: 1.90353 I0420 00:31:14.748126 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90353 INFO:tensorflow:loss = 2.0387444, step = 12100 (52.534 sec) I0420 00:31:14.749640 140176207116096 basic_session_run_hooks.py:260] loss = 2.0387444, step = 12100 (52.534 sec) INFO:tensorflow:global_step/sec: 1.87712 I0420 00:32:08.021189 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87712 INFO:tensorflow:loss = 1.6675963, step = 12200 (53.273 sec) I0420 00:32:08.022638 140176207116096 basic_session_run_hooks.py:260] loss = 1.6675963, step = 12200 (53.273 sec) INFO:tensorflow:global_step/sec: 1.90345 I0420 00:33:00.557270 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90345 INFO:tensorflow:loss = 1.7077498, step = 12300 (52.536 sec) I0420 00:33:00.558525 140176207116096 basic_session_run_hooks.py:260] loss = 1.7077498, step = 12300 (52.536 sec) INFO:tensorflow:global_step/sec: 1.89757 I0420 00:33:53.256382 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89757 INFO:tensorflow:loss = 2.012446, step = 12400 (52.699 sec) I0420 00:33:53.257972 140176207116096 basic_session_run_hooks.py:260] loss = 2.012446, step = 12400 (52.699 sec) INFO:tensorflow:global_step/sec: 1.91005 I0420 00:34:45.611059 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.91005 INFO:tensorflow:loss = 1.6907161, step = 12500 (52.355 sec) I0420 00:34:45.612473 140176207116096 basic_session_run_hooks.py:260] loss = 1.6907161, step = 12500 (52.355 sec) INFO:tensorflow:global_step/sec: 1.91192 I0420 00:35:37.914553 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.91192 INFO:tensorflow:loss = 1.8977844, step = 12600 (52.304 sec) I0420 00:35:37.916654 140176207116096 basic_session_run_hooks.py:260] loss = 1.8977844, step = 12600 (52.304 sec) INFO:tensorflow:global_step/sec: 1.90583 I0420 00:36:30.385207 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.90583 INFO:tensorflow:loss = 2.1952918, step = 12700 (52.470 sec) I0420 00:36:30.386567 140176207116096 basic_session_run_hooks.py:260] loss = 2.1952918, step = 12700 (52.470 sec) INFO:tensorflow:global_step/sec: 1.91803 I0420 00:37:22.522066 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.91803 INFO:tensorflow:loss = 1.7286642, step = 12800 (52.137 sec) I0420 00:37:22.523965 140176207116096 basic_session_run_hooks.py:260] loss = 1.7286642, step = 12800 (52.137 sec) INFO:tensorflow:Saving checkpoints for 12875 into ./my_model_dir/model.ckpt. I0420 00:38:01.514931 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 12875 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 00:38:03.472315 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 00:38:03.473251 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 00:38:03.473476 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 00:38:04.280157 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:38:06.649269 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:38:06.734889 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:38:06.828760 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:38:06.917348 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:38:07.002626 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:38:07.090569 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 00:38:08.679599 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T00:38:08Z I0420 00:38:08.696578 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T00:38:08Z INFO:tensorflow:Graph was finalized. I0420 00:38:09.230710 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 00:38:09.232134: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:38:09.232679: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 00:38:09.232851: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 00:38:09.232882: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 00:38:09.232908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 00:38:09.232932: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 00:38:09.232956: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 00:38:09.232981: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 00:38:09.233005: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 00:38:09.233070: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:38:09.233504: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:38:09.233857: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 00:38:09.233904: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 00:38:09.233913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 00:38:09.233919: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 00:38:09.234027: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:38:09.234467: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:38:09.234831: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-12875 I0420 00:38:09.236207 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-12875 INFO:tensorflow:Running local_init_op. I0420 00:38:10.488351 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 00:38:10.629800 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 00:38:31.162389 140170376505088 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 00:38:31.241186 140170376505088 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0420 00:38:31.261003 140170376505088 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.27s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.267 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.647 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.206 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.063 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.271 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.231 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.375 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.396 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.075 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.401 INFO:tensorflow:Finished evaluation at 2023-04-20-00:38:32 I0420 00:38:32.827409 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-00:38:32 INFO:tensorflow:Saving dict for global step 12875: DetectionBoxes_Precision/mAP = 0.26707238, DetectionBoxes_Precision/mAP (large) = 0.27076992, DetectionBoxes_Precision/mAP (medium) = 0.06336634, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.64741296, DetectionBoxes_Precision/mAP@.75IOU = 0.20631252, DetectionBoxes_Recall/AR@1 = 0.23050049, DetectionBoxes_Recall/AR@10 = 0.37505633, DetectionBoxes_Recall/AR@100 = 0.3957258, DetectionBoxes_Recall/AR@100 (large) = 0.4009843, DetectionBoxes_Recall/AR@100 (medium) = 0.075, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 3.0847447, Loss/localization_loss = 1.2215606, Loss/regularization_loss = 0.28669825, Loss/total_loss = 4.593003, global_step = 12875, learning_rate = 0.0037999998, loss = 4.593003 I0420 00:38:32.828007 140176207116096 estimator.py:2049] Saving dict for global step 12875: DetectionBoxes_Precision/mAP = 0.26707238, DetectionBoxes_Precision/mAP (large) = 0.27076992, DetectionBoxes_Precision/mAP (medium) = 0.06336634, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.64741296, DetectionBoxes_Precision/mAP@.75IOU = 0.20631252, DetectionBoxes_Recall/AR@1 = 0.23050049, DetectionBoxes_Recall/AR@10 = 0.37505633, DetectionBoxes_Recall/AR@100 = 0.3957258, DetectionBoxes_Recall/AR@100 (large) = 0.4009843, DetectionBoxes_Recall/AR@100 (medium) = 0.075, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 3.0847447, Loss/localization_loss = 1.2215606, Loss/regularization_loss = 0.28669825, Loss/total_loss = 4.593003, global_step = 12875, learning_rate = 0.0037999998, loss = 4.593003 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 12875: ./my_model_dir/model.ckpt-12875 I0420 00:38:32.834943 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 12875: ./my_model_dir/model.ckpt-12875 INFO:tensorflow:global_step/sec: 1.18348 I0420 00:38:47.018766 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.18348 INFO:tensorflow:loss = 1.8142129, step = 12900 (84.496 sec) I0420 00:38:47.020342 140176207116096 basic_session_run_hooks.py:260] loss = 1.8142129, step = 12900 (84.496 sec) INFO:tensorflow:global_step/sec: 1.88975 I0420 00:39:39.936173 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88975 INFO:tensorflow:loss = 1.614255, step = 13000 (52.917 sec) I0420 00:39:39.937661 140176207116096 basic_session_run_hooks.py:260] loss = 1.614255, step = 13000 (52.917 sec) INFO:tensorflow:global_step/sec: 1.8934 I0420 00:40:32.751713 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8934 INFO:tensorflow:loss = 1.989925, step = 13100 (52.817 sec) I0420 00:40:32.754449 140176207116096 basic_session_run_hooks.py:260] loss = 1.989925, step = 13100 (52.817 sec) INFO:tensorflow:global_step/sec: 1.88652 I0420 00:41:25.758733 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88652 INFO:tensorflow:loss = 2.2925627, step = 13200 (53.006 sec) I0420 00:41:25.760190 140176207116096 basic_session_run_hooks.py:260] loss = 2.2925627, step = 13200 (53.006 sec) INFO:tensorflow:global_step/sec: 1.88126 I0420 00:42:18.915191 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88126 INFO:tensorflow:loss = 1.4392086, step = 13300 (53.157 sec) I0420 00:42:18.917582 140176207116096 basic_session_run_hooks.py:260] loss = 1.4392086, step = 13300 (53.157 sec) INFO:tensorflow:global_step/sec: 1.88876 I0420 00:43:11.859222 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88876 INFO:tensorflow:loss = 2.6203196, step = 13400 (52.943 sec) I0420 00:43:11.860615 140176207116096 basic_session_run_hooks.py:260] loss = 2.6203196, step = 13400 (52.943 sec) INFO:tensorflow:global_step/sec: 1.89524 I0420 00:44:04.623090 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89524 INFO:tensorflow:loss = 1.6772763, step = 13500 (52.764 sec) I0420 00:44:04.624460 140176207116096 basic_session_run_hooks.py:260] loss = 1.6772763, step = 13500 (52.764 sec) INFO:tensorflow:global_step/sec: 1.89269 I0420 00:44:57.458077 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89269 INFO:tensorflow:loss = 1.7666938, step = 13600 (52.835 sec) I0420 00:44:57.459416 140176207116096 basic_session_run_hooks.py:260] loss = 1.7666938, step = 13600 (52.835 sec) INFO:tensorflow:global_step/sec: 1.88155 I0420 00:45:50.605686 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88155 INFO:tensorflow:loss = 1.5966332, step = 13700 (53.148 sec) I0420 00:45:50.607180 140176207116096 basic_session_run_hooks.py:260] loss = 1.5966332, step = 13700 (53.148 sec) INFO:tensorflow:global_step/sec: 1.89804 I0420 00:46:43.291746 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89804 INFO:tensorflow:loss = 1.9629525, step = 13800 (52.686 sec) I0420 00:46:43.293016 140176207116096 basic_session_run_hooks.py:260] loss = 1.9629525, step = 13800 (52.686 sec) INFO:tensorflow:global_step/sec: 1.88584 I0420 00:47:36.318599 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88584 INFO:tensorflow:loss = 1.562114, step = 13900 (53.027 sec) I0420 00:47:36.319916 140176207116096 basic_session_run_hooks.py:260] loss = 1.562114, step = 13900 (53.027 sec) INFO:tensorflow:Saving checkpoints for 13949 into ./my_model_dir/model.ckpt. I0420 00:48:01.880892 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 13949 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 00:48:03.678471 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 00:48:03.679056 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 00:48:03.679177 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 00:48:04.490198 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:48:06.794764 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:48:06.879399 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:48:06.974581 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:48:07.057901 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:48:07.144968 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 00:48:07.228760 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 00:48:08.809455 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T00:48:08Z I0420 00:48:08.826950 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T00:48:08Z INFO:tensorflow:Graph was finalized. I0420 00:48:09.357668 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 00:48:09.359022: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:48:09.359560: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 00:48:09.359715: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 00:48:09.359746: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 00:48:09.359772: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 00:48:09.359796: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 00:48:09.359819: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 00:48:09.359843: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 00:48:09.359867: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 00:48:09.359939: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:48:09.360356: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:48:09.360715: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 00:48:09.360768: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 00:48:09.360778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 00:48:09.360784: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 00:48:09.360896: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:48:09.361305: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 00:48:09.361676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-13949 I0420 00:48:09.363283 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-13949 INFO:tensorflow:Running local_init_op. I0420 00:48:10.601107 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 00:48:10.740194 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 00:48:31.309163 140169104418560 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 00:48:31.409420 140169104418560 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0420 00:48:31.432734 140169104418560 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.29s). Accumulating evaluation results... DONE (t=0.13s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.331 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.791 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.198 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.114 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.335 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.255 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.429 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.434 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.125 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.440 INFO:tensorflow:Finished evaluation at 2023-04-20-00:48:32 I0420 00:48:32.987329 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-00:48:32 INFO:tensorflow:Saving dict for global step 13949: DetectionBoxes_Precision/mAP = 0.331083, DetectionBoxes_Precision/mAP (large) = 0.3354274, DetectionBoxes_Precision/mAP (medium) = 0.114356436, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.79059976, DetectionBoxes_Precision/mAP@.75IOU = 0.1978989, DetectionBoxes_Recall/AR@1 = 0.2550547, DetectionBoxes_Recall/AR@10 = 0.42927584, DetectionBoxes_Recall/AR@100 = 0.4340264, DetectionBoxes_Recall/AR@100 (large) = 0.4396632, DetectionBoxes_Recall/AR@100 (medium) = 0.125, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 2.345384, Loss/localization_loss = 0.98354095, Loss/regularization_loss = 0.28658298, Loss/total_loss = 3.615508, global_step = 13949, learning_rate = 0.0037999998, loss = 3.615508 I0420 00:48:32.987739 140176207116096 estimator.py:2049] Saving dict for global step 13949: DetectionBoxes_Precision/mAP = 0.331083, DetectionBoxes_Precision/mAP (large) = 0.3354274, DetectionBoxes_Precision/mAP (medium) = 0.114356436, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.79059976, DetectionBoxes_Precision/mAP@.75IOU = 0.1978989, DetectionBoxes_Recall/AR@1 = 0.2550547, DetectionBoxes_Recall/AR@10 = 0.42927584, DetectionBoxes_Recall/AR@100 = 0.4340264, DetectionBoxes_Recall/AR@100 (large) = 0.4396632, DetectionBoxes_Recall/AR@100 (medium) = 0.125, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 2.345384, Loss/localization_loss = 0.98354095, Loss/regularization_loss = 0.28658298, Loss/total_loss = 3.615508, global_step = 13949, learning_rate = 0.0037999998, loss = 3.615508 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 13949: ./my_model_dir/model.ckpt-13949 I0420 00:48:32.994751 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 13949: ./my_model_dir/model.ckpt-13949 INFO:tensorflow:global_step/sec: 1.18657 I0420 00:49:00.595131 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.18657 INFO:tensorflow:loss = 1.6704224, step = 14000 (84.277 sec) I0420 00:49:00.596573 140176207116096 basic_session_run_hooks.py:260] loss = 1.6704224, step = 14000 (84.277 sec) INFO:tensorflow:global_step/sec: 1.87849 I0420 00:49:53.829268 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87849 INFO:tensorflow:loss = 1.8619276, step = 14100 (53.234 sec) I0420 00:49:53.830700 140176207116096 basic_session_run_hooks.py:260] loss = 1.8619276, step = 14100 (53.234 sec) INFO:tensorflow:global_step/sec: 1.88572 I0420 00:50:46.859394 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88572 INFO:tensorflow:loss = 2.0432858, step = 14200 (53.031 sec) I0420 00:50:46.861331 140176207116096 basic_session_run_hooks.py:260] loss = 2.0432858, step = 14200 (53.031 sec) INFO:tensorflow:global_step/sec: 1.89449 I0420 00:51:39.644814 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89449 INFO:tensorflow:loss = 1.5535696, step = 14300 (52.785 sec) I0420 00:51:39.646764 140176207116096 basic_session_run_hooks.py:260] loss = 1.5535696, step = 14300 (52.785 sec) INFO:tensorflow:global_step/sec: 1.85848 I0420 00:52:33.451565 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85848 INFO:tensorflow:loss = 1.9297872, step = 14400 (53.807 sec) I0420 00:52:33.453459 140176207116096 basic_session_run_hooks.py:260] loss = 1.9297872, step = 14400 (53.807 sec) INFO:tensorflow:global_step/sec: 1.87455 I0420 00:53:26.797628 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87455 INFO:tensorflow:loss = 2.224146, step = 14500 (53.346 sec) I0420 00:53:26.799054 140176207116096 basic_session_run_hooks.py:260] loss = 2.224146, step = 14500 (53.346 sec) INFO:tensorflow:global_step/sec: 1.89981 I0420 00:54:19.434628 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89981 INFO:tensorflow:loss = 1.2253306, step = 14600 (52.637 sec) I0420 00:54:19.436556 140176207116096 basic_session_run_hooks.py:260] loss = 1.2253306, step = 14600 (52.637 sec) INFO:tensorflow:global_step/sec: 1.86947 I0420 00:55:12.925597 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86947 INFO:tensorflow:loss = 1.5012391, step = 14700 (53.490 sec) I0420 00:55:12.926927 140176207116096 basic_session_run_hooks.py:260] loss = 1.5012391, step = 14700 (53.490 sec) INFO:tensorflow:global_step/sec: 1.88537 I0420 00:56:05.965618 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88537 INFO:tensorflow:loss = 1.910661, step = 14800 (53.040 sec) I0420 00:56:05.967105 140176207116096 basic_session_run_hooks.py:260] loss = 1.910661, step = 14800 (53.040 sec) INFO:tensorflow:global_step/sec: 1.8866 I0420 00:56:58.970979 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8866 INFO:tensorflow:loss = 1.9106957, step = 14900 (53.005 sec) I0420 00:56:58.972391 140176207116096 basic_session_run_hooks.py:260] loss = 1.9106957, step = 14900 (53.005 sec) INFO:tensorflow:global_step/sec: 1.8786 I0420 00:57:52.202281 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8786 INFO:tensorflow:loss = 1.9149847, step = 15000 (53.231 sec) I0420 00:57:52.203833 140176207116096 basic_session_run_hooks.py:260] loss = 1.9149847, step = 15000 (53.231 sec) INFO:tensorflow:Saving checkpoints for 15019 into ./my_model_dir/model.ckpt. I0420 00:58:01.884211 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 15019 into ./my_model_dir/model.ckpt. INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs). I0420 00:58:03.584474 140176207116096 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs). INFO:tensorflow:global_step/sec: 1.82579 I0420 00:58:46.972972 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.82579 INFO:tensorflow:loss = 1.9081503, step = 15100 (54.771 sec) I0420 00:58:46.974430 140176207116096 basic_session_run_hooks.py:260] loss = 1.9081503, step = 15100 (54.771 sec) INFO:tensorflow:global_step/sec: 1.89275 I0420 00:59:39.806073 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89275 INFO:tensorflow:loss = 2.0652926, step = 15200 (52.833 sec) I0420 00:59:39.807394 140176207116096 basic_session_run_hooks.py:260] loss = 2.0652926, step = 15200 (52.833 sec) INFO:tensorflow:global_step/sec: 1.87816 I0420 01:00:33.049786 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87816 INFO:tensorflow:loss = 1.8521225, step = 15300 (53.244 sec) I0420 01:00:33.051060 140176207116096 basic_session_run_hooks.py:260] loss = 1.8521225, step = 15300 (53.244 sec) INFO:tensorflow:global_step/sec: 1.8762 I0420 01:01:26.349091 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8762 INFO:tensorflow:loss = 1.655903, step = 15400 (53.299 sec) I0420 01:01:26.350504 140176207116096 basic_session_run_hooks.py:260] loss = 1.655903, step = 15400 (53.299 sec) INFO:tensorflow:global_step/sec: 1.87177 I0420 01:02:19.774556 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87177 INFO:tensorflow:loss = 1.7240535, step = 15500 (53.425 sec) I0420 01:02:19.775929 140176207116096 basic_session_run_hooks.py:260] loss = 1.7240535, step = 15500 (53.425 sec) INFO:tensorflow:global_step/sec: 1.88122 I0420 01:03:12.931653 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88122 INFO:tensorflow:loss = 1.6639318, step = 15600 (53.157 sec) I0420 01:03:12.932986 140176207116096 basic_session_run_hooks.py:260] loss = 1.6639318, step = 15600 (53.157 sec) INFO:tensorflow:global_step/sec: 1.88477 I0420 01:04:05.988646 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88477 INFO:tensorflow:loss = 1.9312787, step = 15700 (53.057 sec) I0420 01:04:05.990060 140176207116096 basic_session_run_hooks.py:260] loss = 1.9312787, step = 15700 (53.057 sec) INFO:tensorflow:global_step/sec: 1.89583 I0420 01:04:58.736001 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89583 INFO:tensorflow:loss = 1.7636299, step = 15800 (52.747 sec) I0420 01:04:58.737348 140176207116096 basic_session_run_hooks.py:260] loss = 1.7636299, step = 15800 (52.747 sec) INFO:tensorflow:global_step/sec: 1.87419 I0420 01:05:52.092236 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87419 INFO:tensorflow:loss = 1.285939, step = 15900 (53.356 sec) I0420 01:05:52.093464 140176207116096 basic_session_run_hooks.py:260] loss = 1.285939, step = 15900 (53.356 sec) INFO:tensorflow:global_step/sec: 1.88512 I0420 01:06:45.139207 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88512 INFO:tensorflow:loss = 1.7065532, step = 16000 (53.047 sec) I0420 01:06:45.140529 140176207116096 basic_session_run_hooks.py:260] loss = 1.7065532, step = 16000 (53.047 sec) INFO:tensorflow:global_step/sec: 1.88608 I0420 01:07:38.159247 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88608 INFO:tensorflow:loss = 2.036896, step = 16100 (53.020 sec) I0420 01:07:38.160543 140176207116096 basic_session_run_hooks.py:260] loss = 2.036896, step = 16100 (53.020 sec) INFO:tensorflow:Saving checkpoints for 16146 into ./my_model_dir/model.ckpt. I0420 01:08:02.045411 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 16146 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 01:08:03.785171 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 01:08:03.785863 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 01:08:03.786051 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 01:08:04.589421 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:08:06.895491 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:08:06.991570 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:08:07.676169 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:08:07.772900 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:08:07.858700 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:08:07.944261 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 01:08:09.504468 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T01:08:09Z I0420 01:08:09.521222 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T01:08:09Z INFO:tensorflow:Graph was finalized. I0420 01:08:10.040831 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 01:08:10.041673: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:08:10.042176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:08:10.042339: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:08:10.042383: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:08:10.042413: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:08:10.042437: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:08:10.042460: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:08:10.042483: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:08:10.042507: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:08:10.042572: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:08:10.042971: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:08:10.043317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:08:10.043360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 01:08:10.043384: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 01:08:10.043392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 01:08:10.043492: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:08:10.043892: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:08:10.044253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-16146 I0420 01:08:10.045751 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-16146 INFO:tensorflow:Running local_init_op. I0420 01:08:11.302320 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 01:08:11.443921 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 01:08:33.468104 140169087633152 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 01:08:33.478599 140169087633152 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.02s) I0420 01:08:33.500880 140169087633152 coco_tools.py:138] DONE (t=0.02s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.18s). Accumulating evaluation results... DONE (t=0.13s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.535 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.962 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.514 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.342 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.537 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.362 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.599 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.599 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.350 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.603 INFO:tensorflow:Finished evaluation at 2023-04-20-01:08:34 I0420 01:08:34.940027 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-01:08:34 INFO:tensorflow:Saving dict for global step 16146: DetectionBoxes_Precision/mAP = 0.5348967, DetectionBoxes_Precision/mAP (large) = 0.53738725, DetectionBoxes_Precision/mAP (medium) = 0.34224424, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.96171063, DetectionBoxes_Precision/mAP@.75IOU = 0.51414376, DetectionBoxes_Recall/AR@1 = 0.36241713, DetectionBoxes_Recall/AR@10 = 0.5988542, DetectionBoxes_Recall/AR@100 = 0.5990634, DetectionBoxes_Recall/AR@100 (large) = 0.60327524, DetectionBoxes_Recall/AR@100 (medium) = 0.35, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.017429, Loss/localization_loss = 0.45888254, Loss/regularization_loss = 0.28628948, Loss/total_loss = 1.7626022, global_step = 16146, learning_rate = 0.0037999998, loss = 1.7626022 I0420 01:08:34.940437 140176207116096 estimator.py:2049] Saving dict for global step 16146: DetectionBoxes_Precision/mAP = 0.5348967, DetectionBoxes_Precision/mAP (large) = 0.53738725, DetectionBoxes_Precision/mAP (medium) = 0.34224424, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.96171063, DetectionBoxes_Precision/mAP@.75IOU = 0.51414376, DetectionBoxes_Recall/AR@1 = 0.36241713, DetectionBoxes_Recall/AR@10 = 0.5988542, DetectionBoxes_Recall/AR@100 = 0.5990634, DetectionBoxes_Recall/AR@100 (large) = 0.60327524, DetectionBoxes_Recall/AR@100 (medium) = 0.35, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.017429, Loss/localization_loss = 0.45888254, Loss/regularization_loss = 0.28628948, Loss/total_loss = 1.7626022, global_step = 16146, learning_rate = 0.0037999998, loss = 1.7626022 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 16146: ./my_model_dir/model.ckpt-16146 I0420 01:08:34.946830 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 16146: ./my_model_dir/model.ckpt-16146 INFO:tensorflow:global_step/sec: 1.15774 I0420 01:09:04.534464 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.15774 INFO:tensorflow:loss = 2.6300843, step = 16200 (86.375 sec) I0420 01:09:04.535858 140176207116096 basic_session_run_hooks.py:260] loss = 2.6300843, step = 16200 (86.375 sec) INFO:tensorflow:global_step/sec: 1.85252 I0420 01:09:58.514929 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85252 INFO:tensorflow:loss = 1.8682594, step = 16300 (53.980 sec) I0420 01:09:58.516319 140176207116096 basic_session_run_hooks.py:260] loss = 1.8682594, step = 16300 (53.980 sec) INFO:tensorflow:global_step/sec: 1.86339 I0420 01:10:52.180723 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86339 INFO:tensorflow:loss = 2.6552408, step = 16400 (53.666 sec) I0420 01:10:52.182066 140176207116096 basic_session_run_hooks.py:260] loss = 2.6552408, step = 16400 (53.666 sec) INFO:tensorflow:global_step/sec: 1.85719 I0420 01:11:46.025622 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85719 INFO:tensorflow:loss = 1.4233844, step = 16500 (53.845 sec) I0420 01:11:46.026932 140176207116096 basic_session_run_hooks.py:260] loss = 1.4233844, step = 16500 (53.845 sec) INFO:tensorflow:global_step/sec: 1.89747 I0420 01:12:38.727601 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89747 INFO:tensorflow:loss = 1.6952581, step = 16600 (52.702 sec) I0420 01:12:38.728899 140176207116096 basic_session_run_hooks.py:260] loss = 1.6952581, step = 16600 (52.702 sec) INFO:tensorflow:global_step/sec: 1.86192 I0420 01:13:32.436150 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86192 INFO:tensorflow:loss = 1.377973, step = 16700 (53.709 sec) I0420 01:13:32.438290 140176207116096 basic_session_run_hooks.py:260] loss = 1.377973, step = 16700 (53.709 sec) INFO:tensorflow:global_step/sec: 1.87559 I0420 01:14:25.752093 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87559 INFO:tensorflow:loss = 1.7626233, step = 16800 (53.315 sec) I0420 01:14:25.753618 140176207116096 basic_session_run_hooks.py:260] loss = 1.7626233, step = 16800 (53.315 sec) INFO:tensorflow:global_step/sec: 1.88151 I0420 01:15:18.901688 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88151 INFO:tensorflow:loss = 2.2404213, step = 16900 (53.150 sec) I0420 01:15:18.903804 140176207116096 basic_session_run_hooks.py:260] loss = 2.2404213, step = 16900 (53.150 sec) INFO:tensorflow:global_step/sec: 1.86626 I0420 01:16:12.484007 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86626 INFO:tensorflow:loss = 1.6422467, step = 17000 (53.581 sec) I0420 01:16:12.485231 140176207116096 basic_session_run_hooks.py:260] loss = 1.6422467, step = 17000 (53.581 sec) INFO:tensorflow:global_step/sec: 1.85771 I0420 01:17:06.313599 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85771 INFO:tensorflow:loss = 1.5077852, step = 17100 (53.830 sec) I0420 01:17:06.314903 140176207116096 basic_session_run_hooks.py:260] loss = 1.5077852, step = 17100 (53.830 sec) INFO:tensorflow:global_step/sec: 1.87099 I0420 01:17:59.761353 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87099 INFO:tensorflow:loss = 1.3955805, step = 17200 (53.448 sec) I0420 01:17:59.762716 140176207116096 basic_session_run_hooks.py:260] loss = 1.3955805, step = 17200 (53.448 sec) INFO:tensorflow:Saving checkpoints for 17206 into ./my_model_dir/model.ckpt. I0420 01:18:02.453213 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 17206 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 01:18:04.341036 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 01:18:04.341787 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 01:18:04.341992 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 01:18:05.145855 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:18:07.445394 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:18:07.529596 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:18:07.624628 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:18:07.711485 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:18:07.801305 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:18:07.885867 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 01:18:09.460984 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T01:18:09Z I0420 01:18:09.480965 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T01:18:09Z INFO:tensorflow:Graph was finalized. I0420 01:18:10.566163 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 01:18:10.567387: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:18:10.567938: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:18:10.568120: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:18:10.568156: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:18:10.568189: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:18:10.568224: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:18:10.568251: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:18:10.568276: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:18:10.568304: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:18:10.568396: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:18:10.568835: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:18:10.569224: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:18:10.569279: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 01:18:10.569289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 01:18:10.569296: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 01:18:10.569440: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:18:10.569880: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:18:10.570284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-17206 I0420 01:18:10.572092 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-17206 INFO:tensorflow:Running local_init_op. I0420 01:18:11.861316 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 01:18:12.001485 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 01:18:33.710103 140168203859712 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 01:18:33.711994 140168203859712 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.05s) I0420 01:18:33.763521 140168203859712 coco_tools.py:138] DONE (t=0.05s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.13s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.514 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.953 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.486 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.247 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.520 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.360 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.588 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.590 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.275 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.596 INFO:tensorflow:Finished evaluation at 2023-04-20-01:18:35 I0420 01:18:35.179098 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-01:18:35 INFO:tensorflow:Saving dict for global step 17206: DetectionBoxes_Precision/mAP = 0.5144083, DetectionBoxes_Precision/mAP (large) = 0.5202623, DetectionBoxes_Precision/mAP (medium) = 0.24653466, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.95294833, DetectionBoxes_Precision/mAP@.75IOU = 0.4855384, DetectionBoxes_Recall/AR@1 = 0.35992438, DetectionBoxes_Recall/AR@10 = 0.5880544, DetectionBoxes_Recall/AR@100 = 0.5897651, DetectionBoxes_Recall/AR@100 (large) = 0.59588104, DetectionBoxes_Recall/AR@100 (medium) = 0.275, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.5180346, Loss/localization_loss = 0.44197774, Loss/regularization_loss = 0.2861443, Loss/total_loss = 2.2461574, global_step = 17206, learning_rate = 0.0037999998, loss = 2.2461574 I0420 01:18:35.179554 140176207116096 estimator.py:2049] Saving dict for global step 17206: DetectionBoxes_Precision/mAP = 0.5144083, DetectionBoxes_Precision/mAP (large) = 0.5202623, DetectionBoxes_Precision/mAP (medium) = 0.24653466, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.95294833, DetectionBoxes_Precision/mAP@.75IOU = 0.4855384, DetectionBoxes_Recall/AR@1 = 0.35992438, DetectionBoxes_Recall/AR@10 = 0.5880544, DetectionBoxes_Recall/AR@100 = 0.5897651, DetectionBoxes_Recall/AR@100 (large) = 0.59588104, DetectionBoxes_Recall/AR@100 (medium) = 0.275, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.5180346, Loss/localization_loss = 0.44197774, Loss/regularization_loss = 0.2861443, Loss/total_loss = 2.2461574, global_step = 17206, learning_rate = 0.0037999998, loss = 2.2461574 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 17206: ./my_model_dir/model.ckpt-17206 I0420 01:18:35.188548 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 17206: ./my_model_dir/model.ckpt-17206 INFO:tensorflow:global_step/sec: 1.16147 I0420 01:19:25.859363 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.16147 INFO:tensorflow:loss = 1.5912938, step = 17300 (86.098 sec) I0420 01:19:25.860961 140176207116096 basic_session_run_hooks.py:260] loss = 1.5912938, step = 17300 (86.098 sec) INFO:tensorflow:global_step/sec: 1.89313 I0420 01:20:18.681984 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89313 INFO:tensorflow:loss = 1.404489, step = 17400 (52.823 sec) I0420 01:20:18.683513 140176207116096 basic_session_run_hooks.py:260] loss = 1.404489, step = 17400 (52.823 sec) INFO:tensorflow:global_step/sec: 1.87612 I0420 01:21:11.983325 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.87612 INFO:tensorflow:loss = 1.9755485, step = 17500 (53.301 sec) I0420 01:21:11.984783 140176207116096 basic_session_run_hooks.py:260] loss = 1.9755485, step = 17500 (53.301 sec) INFO:tensorflow:global_step/sec: 1.88456 I0420 01:22:05.046071 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88456 INFO:tensorflow:loss = 1.8240697, step = 17600 (53.063 sec) I0420 01:22:05.047609 140176207116096 basic_session_run_hooks.py:260] loss = 1.8240697, step = 17600 (53.063 sec) INFO:tensorflow:global_step/sec: 1.89003 I0420 01:22:57.955338 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89003 INFO:tensorflow:loss = 1.6217228, step = 17700 (52.909 sec) I0420 01:22:57.956995 140176207116096 basic_session_run_hooks.py:260] loss = 1.6217228, step = 17700 (52.909 sec) INFO:tensorflow:global_step/sec: 1.89577 I0420 01:23:50.704308 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89577 INFO:tensorflow:loss = 1.7322894, step = 17800 (52.749 sec) I0420 01:23:50.705575 140176207116096 basic_session_run_hooks.py:260] loss = 1.7322894, step = 17800 (52.749 sec) INFO:tensorflow:global_step/sec: 1.88662 I0420 01:24:43.709187 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88662 INFO:tensorflow:loss = 1.7711031, step = 17900 (53.005 sec) I0420 01:24:43.710576 140176207116096 basic_session_run_hooks.py:260] loss = 1.7711031, step = 17900 (53.005 sec) INFO:tensorflow:global_step/sec: 1.88606 I0420 01:25:36.729800 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.88606 INFO:tensorflow:loss = 1.6898121, step = 18000 (53.021 sec) I0420 01:25:36.731122 140176207116096 basic_session_run_hooks.py:260] loss = 1.6898121, step = 18000 (53.021 sec) INFO:tensorflow:global_step/sec: 1.89909 I0420 01:26:29.387450 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.89909 INFO:tensorflow:loss = 1.994015, step = 18100 (52.659 sec) I0420 01:26:29.390101 140176207116096 basic_session_run_hooks.py:260] loss = 1.994015, step = 18100 (52.659 sec) INFO:tensorflow:global_step/sec: 1.8748 I0420 01:27:22.725716 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8748 INFO:tensorflow:loss = 1.6200659, step = 18200 (53.338 sec) I0420 01:27:22.727610 140176207116096 basic_session_run_hooks.py:260] loss = 1.6200659, step = 18200 (53.338 sec) INFO:tensorflow:Saving checkpoints for 18277 into ./my_model_dir/model.ckpt. I0420 01:28:02.545282 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 18277 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 01:28:04.360965 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 01:28:04.361851 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 01:28:04.362052 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 01:28:05.166756 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:28:07.442851 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:28:07.530510 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:28:07.622928 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:28:07.709677 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:28:07.793828 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:28:07.878098 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 01:28:09.431158 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T01:28:09Z I0420 01:28:09.447979 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T01:28:09Z INFO:tensorflow:Graph was finalized. I0420 01:28:09.963931 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 01:28:09.965021: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:28:09.965547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:28:09.965700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:28:09.965731: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:28:09.965758: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:28:09.965781: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:28:09.965804: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:28:09.965826: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:28:09.965850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:28:09.965914: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:28:09.966317: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:28:09.966680: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:28:09.966726: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 01:28:09.966734: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 01:28:09.966740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 01:28:09.966840: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:28:09.967235: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:28:09.967614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-18277 I0420 01:28:09.968857 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-18277 INFO:tensorflow:Running local_init_op. I0420 01:28:11.149863 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 01:28:11.284569 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 01:28:33.366546 140169121203968 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 01:28:33.416968 140169121203968 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.03s) I0420 01:28:33.442333 140169121203968 coco_tools.py:138] DONE (t=0.03s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.16s). Accumulating evaluation results... DONE (t=0.14s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.576 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.966 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.598 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.235 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.584 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.404 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.635 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.637 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.275 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.644 INFO:tensorflow:Finished evaluation at 2023-04-20-01:28:34 I0420 01:28:34.883695 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-01:28:34 INFO:tensorflow:Saving dict for global step 18277: DetectionBoxes_Precision/mAP = 0.5762955, DetectionBoxes_Precision/mAP (large) = 0.5836706, DetectionBoxes_Precision/mAP (medium) = 0.23465346, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.96635884, DetectionBoxes_Precision/mAP@.75IOU = 0.5982647, DetectionBoxes_Recall/AR@1 = 0.404065, DetectionBoxes_Recall/AR@10 = 0.63516575, DetectionBoxes_Recall/AR@100 = 0.6365626, DetectionBoxes_Recall/AR@100 (large) = 0.64444405, DetectionBoxes_Recall/AR@100 (medium) = 0.275, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.2222483, Loss/localization_loss = 0.3039433, Loss/regularization_loss = 0.28596392, Loss/total_loss = 1.8121543, global_step = 18277, learning_rate = 0.0037999998, loss = 1.8121543 I0420 01:28:34.884294 140176207116096 estimator.py:2049] Saving dict for global step 18277: DetectionBoxes_Precision/mAP = 0.5762955, DetectionBoxes_Precision/mAP (large) = 0.5836706, DetectionBoxes_Precision/mAP (medium) = 0.23465346, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.96635884, DetectionBoxes_Precision/mAP@.75IOU = 0.5982647, DetectionBoxes_Recall/AR@1 = 0.404065, DetectionBoxes_Recall/AR@10 = 0.63516575, DetectionBoxes_Recall/AR@100 = 0.6365626, DetectionBoxes_Recall/AR@100 (large) = 0.64444405, DetectionBoxes_Recall/AR@100 (medium) = 0.275, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.2222483, Loss/localization_loss = 0.3039433, Loss/regularization_loss = 0.28596392, Loss/total_loss = 1.8121543, global_step = 18277, learning_rate = 0.0037999998, loss = 1.8121543 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 18277: ./my_model_dir/model.ckpt-18277 I0420 01:28:34.891807 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 18277: ./my_model_dir/model.ckpt-18277 INFO:tensorflow:global_step/sec: 1.17349 I0420 01:28:47.941772 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.17349 INFO:tensorflow:loss = 1.544658, step = 18300 (85.216 sec) I0420 01:28:47.943663 140176207116096 basic_session_run_hooks.py:260] loss = 1.544658, step = 18300 (85.216 sec) INFO:tensorflow:global_step/sec: 1.859 I0420 01:29:41.734353 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.859 INFO:tensorflow:loss = 1.6872858, step = 18400 (53.792 sec) I0420 01:29:41.736022 140176207116096 basic_session_run_hooks.py:260] loss = 1.6872858, step = 18400 (53.792 sec) INFO:tensorflow:global_step/sec: 1.85217 I0420 01:30:35.725031 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85217 INFO:tensorflow:loss = 1.6124195, step = 18500 (53.990 sec) I0420 01:30:35.726509 140176207116096 basic_session_run_hooks.py:260] loss = 1.6124195, step = 18500 (53.990 sec) INFO:tensorflow:global_step/sec: 1.8663 I0420 01:31:29.306959 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8663 INFO:tensorflow:loss = 1.7056823, step = 18600 (53.582 sec) I0420 01:31:29.308319 140176207116096 basic_session_run_hooks.py:260] loss = 1.7056823, step = 18600 (53.582 sec) INFO:tensorflow:global_step/sec: 1.86533 I0420 01:32:22.916792 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86533 INFO:tensorflow:loss = 1.570295, step = 18700 (53.610 sec) I0420 01:32:22.918483 140176207116096 basic_session_run_hooks.py:260] loss = 1.570295, step = 18700 (53.610 sec) INFO:tensorflow:global_step/sec: 1.86493 I0420 01:33:16.537940 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86493 INFO:tensorflow:loss = 2.0630293, step = 18800 (53.621 sec) I0420 01:33:16.539102 140176207116096 basic_session_run_hooks.py:260] loss = 2.0630293, step = 18800 (53.621 sec) INFO:tensorflow:global_step/sec: 1.85412 I0420 01:34:10.471765 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85412 INFO:tensorflow:loss = 2.0213647, step = 18900 (53.935 sec) I0420 01:34:10.473843 140176207116096 basic_session_run_hooks.py:260] loss = 2.0213647, step = 18900 (53.935 sec) INFO:tensorflow:global_step/sec: 1.8883 I0420 01:35:03.429342 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.8883 INFO:tensorflow:loss = 1.7716267, step = 19000 (52.957 sec) I0420 01:35:03.430895 140176207116096 basic_session_run_hooks.py:260] loss = 1.7716267, step = 19000 (52.957 sec) INFO:tensorflow:global_step/sec: 1.86248 I0420 01:35:57.121264 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86248 INFO:tensorflow:loss = 2.0549922, step = 19100 (53.692 sec) I0420 01:35:57.122716 140176207116096 basic_session_run_hooks.py:260] loss = 2.0549922, step = 19100 (53.692 sec) INFO:tensorflow:global_step/sec: 1.85665 I0420 01:36:50.981663 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85665 INFO:tensorflow:loss = 2.1041684, step = 19200 (53.860 sec) I0420 01:36:50.982981 140176207116096 basic_session_run_hooks.py:260] loss = 2.1041684, step = 19200 (53.860 sec) INFO:tensorflow:global_step/sec: 1.86542 I0420 01:37:44.589024 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.86542 INFO:tensorflow:loss = 1.8450484, step = 19300 (53.608 sec) I0420 01:37:44.590559 140176207116096 basic_session_run_hooks.py:260] loss = 1.8450484, step = 19300 (53.608 sec) INFO:tensorflow:Saving checkpoints for 19335 into ./my_model_dir/model.ckpt. I0420 01:38:02.773486 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 19335 into ./my_model_dir/model.ckpt. INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 01:38:04.700869 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 01:38:04.701535 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 01:38:04.701742 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 01:38:05.513135 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:38:08.407592 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:38:08.492791 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:38:08.586458 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:38:08.669818 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:38:08.755483 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:38:08.843415 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 01:38:10.441807 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T01:38:10Z I0420 01:38:10.459113 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T01:38:10Z INFO:tensorflow:Graph was finalized. I0420 01:38:10.989116 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 01:38:10.990467: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:38:10.990979: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:38:10.991126: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:38:10.991158: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:38:10.991184: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:38:10.991208: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:38:10.991232: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:38:10.991256: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:38:10.991280: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:38:10.991357: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:38:10.991784: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:38:10.992126: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:38:10.992174: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 01:38:10.992183: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 01:38:10.992188: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 01:38:10.992296: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:38:10.992714: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:38:10.993067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-19335 I0420 01:38:10.995016 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-19335 INFO:tensorflow:Running local_init_op. I0420 01:38:12.282600 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 01:38:12.434214 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 01:38:34.407401 140169096025856 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 01:38:34.411494 140169096025856 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.03s) I0420 01:38:34.441349 140169096025856 coco_tools.py:138] DONE (t=0.03s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.27s). Accumulating evaluation results... DONE (t=0.13s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.564 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.954 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.580 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.313 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.568 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.383 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.630 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.632 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.325 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.638 INFO:tensorflow:Finished evaluation at 2023-04-20-01:38:35 I0420 01:38:35.989565 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-01:38:35 INFO:tensorflow:Saving dict for global step 19335: DetectionBoxes_Precision/mAP = 0.5644086, DetectionBoxes_Precision/mAP (large) = 0.56844556, DetectionBoxes_Precision/mAP (medium) = 0.31287128, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.95398945, DetectionBoxes_Precision/mAP@.75IOU = 0.58043736, DetectionBoxes_Recall/AR@1 = 0.38346797, DetectionBoxes_Recall/AR@10 = 0.6303782, DetectionBoxes_Recall/AR@100 = 0.63215643, DetectionBoxes_Recall/AR@100 (large) = 0.6380959, DetectionBoxes_Recall/AR@100 (medium) = 0.325, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.6647975, Loss/localization_loss = 0.3156528, Loss/regularization_loss = 0.2857897, Loss/total_loss = 2.2662401, global_step = 19335, learning_rate = 0.0037999998, loss = 2.2662401 I0420 01:38:35.990257 140176207116096 estimator.py:2049] Saving dict for global step 19335: DetectionBoxes_Precision/mAP = 0.5644086, DetectionBoxes_Precision/mAP (large) = 0.56844556, DetectionBoxes_Precision/mAP (medium) = 0.31287128, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.95398945, DetectionBoxes_Precision/mAP@.75IOU = 0.58043736, DetectionBoxes_Recall/AR@1 = 0.38346797, DetectionBoxes_Recall/AR@10 = 0.6303782, DetectionBoxes_Recall/AR@100 = 0.63215643, DetectionBoxes_Recall/AR@100 (large) = 0.6380959, DetectionBoxes_Recall/AR@100 (medium) = 0.325, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.6647975, Loss/localization_loss = 0.3156528, Loss/regularization_loss = 0.2857897, Loss/total_loss = 2.2662401, global_step = 19335, learning_rate = 0.0037999998, loss = 2.2662401 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 19335: ./my_model_dir/model.ckpt-19335 I0420 01:38:35.997551 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 19335: ./my_model_dir/model.ckpt-19335 INFO:tensorflow:global_step/sec: 1.14965 I0420 01:39:11.572386 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.14965 INFO:tensorflow:loss = 1.8993635, step = 19400 (86.984 sec) I0420 01:39:11.574741 140176207116096 basic_session_run_hooks.py:260] loss = 1.8993635, step = 19400 (86.984 sec) INFO:tensorflow:global_step/sec: 1.83479 I0420 01:40:06.074068 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.83479 INFO:tensorflow:loss = 1.9952282, step = 19500 (54.501 sec) I0420 01:40:06.076090 140176207116096 basic_session_run_hooks.py:260] loss = 1.9952282, step = 19500 (54.501 sec) INFO:tensorflow:global_step/sec: 1.84953 I0420 01:41:00.141682 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.84953 INFO:tensorflow:loss = 1.647923, step = 19600 (54.067 sec) I0420 01:41:00.143114 140176207116096 basic_session_run_hooks.py:260] loss = 1.647923, step = 19600 (54.067 sec) INFO:tensorflow:global_step/sec: 1.84562 I0420 01:41:54.323801 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.84562 INFO:tensorflow:loss = 1.8752277, step = 19700 (54.182 sec) I0420 01:41:54.325176 140176207116096 basic_session_run_hooks.py:260] loss = 1.8752277, step = 19700 (54.182 sec) INFO:tensorflow:global_step/sec: 1.85852 I0420 01:42:48.129944 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.85852 INFO:tensorflow:loss = 2.047365, step = 19800 (53.806 sec) I0420 01:42:48.131517 140176207116096 basic_session_run_hooks.py:260] loss = 2.047365, step = 19800 (53.806 sec) INFO:tensorflow:global_step/sec: 1.84845 I0420 01:43:42.229352 140176207116096 basic_session_run_hooks.py:692] global_step/sec: 1.84845 INFO:tensorflow:loss = 1.9515204, step = 19900 (54.099 sec) I0420 01:43:42.230626 140176207116096 basic_session_run_hooks.py:260] loss = 1.9515204, step = 19900 (54.099 sec) INFO:tensorflow:Saving checkpoints for 20000 into ./my_model_dir/model.ckpt. I0420 01:44:35.520356 140176207116096 basic_session_run_hooks.py:606] Saving checkpoints for 20000 into ./my_model_dir/model.ckpt. INFO:tensorflow:Skip the current checkpoint eval due to throttle secs (600 secs). I0420 01:44:37.283586 140176207116096 training.py:527] Skip the current checkpoint eval due to throttle secs (600 secs). INFO:tensorflow:Reading unweighted datasets: ['./data/annotations/test.record'] I0420 01:44:37.311938 140176207116096 dataset_builder.py:162] Reading unweighted datasets: ['./data/annotations/test.record'] INFO:tensorflow:Reading record datasets for input file: ['./data/annotations/test.record'] I0420 01:44:37.312528 140176207116096 dataset_builder.py:79] Reading record datasets for input file: ['./data/annotations/test.record'] INFO:tensorflow:Number of filenames to read: 1 I0420 01:44:37.312724 140176207116096 dataset_builder.py:80] Number of filenames to read: 1 INFO:tensorflow:Calling model_fn. I0420 01:44:38.109782 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:44:40.814146 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:44:40.902763 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:44:40.994651 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:44:41.080452 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:44:41.163629 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:44:41.250771 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 01:44:42.841770 140176207116096 estimator.py:1150] Done calling model_fn. INFO:tensorflow:Starting evaluation at 2023-04-20T01:44:42Z I0420 01:44:42.859129 140176207116096 evaluation.py:255] Starting evaluation at 2023-04-20T01:44:42Z INFO:tensorflow:Graph was finalized. I0420 01:44:43.383125 140176207116096 monitored_session.py:240] Graph was finalized. 2023-04-20 01:44:43.384462: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:44:43.384970: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:44:43.385121: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:44:43.385151: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:44:43.385177: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:44:43.385201: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:44:43.385224: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:44:43.385248: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:44:43.385271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:44:43.385336: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:44:43.385769: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:44:43.386118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:44:43.386168: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 01:44:43.386176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 01:44:43.386182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 01:44:43.386289: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:44:43.386728: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:44:43.387080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-20000 I0420 01:44:43.388306 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-20000 INFO:tensorflow:Running local_init_op. I0420 01:44:44.653465 140176207116096 session_manager.py:500] Running local_init_op. INFO:tensorflow:Done running local_init_op. I0420 01:44:44.789717 140176207116096 session_manager.py:502] Done running local_init_op. INFO:tensorflow:Performing evaluation on 201 images. I0420 01:45:06.357150 140169129596672 coco_evaluation.py:293] Performing evaluation on 201 images. creating index... index created! INFO:tensorflow:Loading and preparing annotation results... I0420 01:45:06.361973 140169129596672 coco_tools.py:116] Loading and preparing annotation results... INFO:tensorflow:DONE (t=0.03s) I0420 01:45:06.388555 140169129596672 coco_tools.py:138] DONE (t=0.03s) creating index... index created! Running per image evaluation... Evaluate annotation type *bbox* DONE (t=1.30s). Accumulating evaluation results... DONE (t=0.13s). Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.600 Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.976 Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.644 Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.337 Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.604 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.427 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.658 Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.659 Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000 Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.425 Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.663 INFO:tensorflow:Finished evaluation at 2023-04-20-01:45:07 I0420 01:45:07.951333 140176207116096 evaluation.py:275] Finished evaluation at 2023-04-20-01:45:07 INFO:tensorflow:Saving dict for global step 20000: DetectionBoxes_Precision/mAP = 0.59989136, DetectionBoxes_Precision/mAP (large) = 0.60423017, DetectionBoxes_Precision/mAP (medium) = 0.33663365, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.97592956, DetectionBoxes_Precision/mAP@.75IOU = 0.64375913, DetectionBoxes_Recall/AR@1 = 0.4273753, DetectionBoxes_Recall/AR@10 = 0.6576875, DetectionBoxes_Recall/AR@100 = 0.6588381, DetectionBoxes_Recall/AR@100 (large) = 0.6625077, DetectionBoxes_Recall/AR@100 (medium) = 0.425, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.3494545, Loss/localization_loss = 0.2655663, Loss/regularization_loss = 0.28567785, Loss/total_loss = 1.9006976, global_step = 20000, learning_rate = 0.0036099998, loss = 1.9006976 I0420 01:45:07.951960 140176207116096 estimator.py:2049] Saving dict for global step 20000: DetectionBoxes_Precision/mAP = 0.59989136, DetectionBoxes_Precision/mAP (large) = 0.60423017, DetectionBoxes_Precision/mAP (medium) = 0.33663365, DetectionBoxes_Precision/mAP (small) = -1.0, DetectionBoxes_Precision/mAP@.50IOU = 0.97592956, DetectionBoxes_Precision/mAP@.75IOU = 0.64375913, DetectionBoxes_Recall/AR@1 = 0.4273753, DetectionBoxes_Recall/AR@10 = 0.6576875, DetectionBoxes_Recall/AR@100 = 0.6588381, DetectionBoxes_Recall/AR@100 (large) = 0.6625077, DetectionBoxes_Recall/AR@100 (medium) = 0.425, DetectionBoxes_Recall/AR@100 (small) = -1.0, Loss/classification_loss = 1.3494545, Loss/localization_loss = 0.2655663, Loss/regularization_loss = 0.28567785, Loss/total_loss = 1.9006976, global_step = 20000, learning_rate = 0.0036099998, loss = 1.9006976 INFO:tensorflow:Saving 'checkpoint_path' summary for global step 20000: ./my_model_dir/model.ckpt-20000 I0420 01:45:07.959116 140176207116096 estimator.py:2109] Saving 'checkpoint_path' summary for global step 20000: ./my_model_dir/model.ckpt-20000 INFO:tensorflow:Performing the final export in the end of training. I0420 01:45:07.960670 140176207116096 exporter.py:410] Performing the final export in the end of training. INFO:tensorflow:Calling model_fn. I0420 01:45:08.241314 140176207116096 estimator.py:1148] Calling model_fn. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:45:10.524664 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:45:10.614420 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:45:10.733267 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:45:10.817312 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:45:10.901561 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:45:10.985669 140176207116096 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:Done calling model_fn. I0420 01:45:11.811144 140176207116096 estimator.py:1150] Done calling model_fn. WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. W0420 01:45:11.811500 140176207116096 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. INFO:tensorflow:Signatures INCLUDED in export for Classify: None I0420 01:45:11.812207 140176207116096 export_utils.py:170] Signatures INCLUDED in export for Classify: None INFO:tensorflow:Signatures INCLUDED in export for Regress: None I0420 01:45:11.812292 140176207116096 export_utils.py:170] Signatures INCLUDED in export for Regress: None INFO:tensorflow:Signatures INCLUDED in export for Predict: ['tensorflow/serving/predict', 'serving_default'] I0420 01:45:11.812362 140176207116096 export_utils.py:170] Signatures INCLUDED in export for Predict: ['tensorflow/serving/predict', 'serving_default'] INFO:tensorflow:Signatures INCLUDED in export for Train: None I0420 01:45:11.812439 140176207116096 export_utils.py:170] Signatures INCLUDED in export for Train: None INFO:tensorflow:Signatures INCLUDED in export for Eval: None I0420 01:45:11.812491 140176207116096 export_utils.py:170] Signatures INCLUDED in export for Eval: None 2023-04-20 01:45:11.813550: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:45:11.814079: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:45:11.814225: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:45:11.814265: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:45:11.814302: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:45:11.814328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:45:11.814352: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:45:11.814391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:45:11.814417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:45:11.814484: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:45:11.814911: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:45:11.815249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:45:11.815322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 01:45:11.815334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 01:45:11.815340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 01:45:11.815468: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:45:11.815879: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:45:11.816232: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-20000 I0420 01:45:11.819316 140176207116096 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-20000 INFO:tensorflow:Assets added to graph. I0420 01:45:12.402481 140176207116096 builder_impl.py:665] Assets added to graph. INFO:tensorflow:No assets to write. I0420 01:45:12.402714 140176207116096 builder_impl.py:460] No assets to write. INFO:tensorflow:SavedModel written to: ./my_model_dir/export/Servo/temp-b'1681926307'/saved_model.pb I0420 01:45:13.214741 140176207116096 builder_impl.py:425] SavedModel written to: ./my_model_dir/export/Servo/temp-b'1681926307'/saved_model.pb INFO:tensorflow:Loss for final step: 1.7440276. I0420 01:45:13.901421 140176207116096 estimator.py:371] Loss for final step: 1.7440276.
!python ./export_inference_graph.py --input_type=image_tensor --pipeline_config_path=./data/ssdlite_mobilenet_v2_coco.config --trained_checkpoint_prefix=./my_model_dir/model.ckpt-20000 --output_directory=./inference_model
WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. W0420 01:46:11.083471 139993840469824 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tf_slim/layers/layers.py:1089: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use `layer.__call__` method instead. INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:46:13.379525 139993840469824 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:46:13.469231 139993840469824 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:46:13.564844 139993840469824 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:46:13.661527 139993840469824 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:46:13.752012 139993840469824 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 INFO:tensorflow:depth of additional conv before box predictor: 0 I0420 01:46:13.840643 139993840469824 convolutional_box_predictor.py:155] depth of additional conv before box predictor: 0 WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/core/post_processing.py:623: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where W0420 01:46:14.144064 139993840469824 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/core/post_processing.py:623: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.where in 2.0, which has the same broadcast rule as np.where WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/exporter.py:481: get_or_create_global_step (from tf_slim.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.get_or_create_global_step W0420 01:46:14.492765 139993840469824 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/exporter.py:481: get_or_create_global_step (from tf_slim.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.get_or_create_global_step WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/exporter.py:660: print_model_analysis (from tensorflow.contrib.tfprof.model_analyzer) is deprecated and will be removed after 2018-01-01. Instructions for updating: Use `tf.profiler.profile(graph, run_meta, op_log, cmd, options)`. Build `options` with `tf.profiler.ProfileOptionBuilder`. See README.md for details W0420 01:46:14.495997 139993840469824 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/exporter.py:660: print_model_analysis (from tensorflow.contrib.tfprof.model_analyzer) is deprecated and will be removed after 2018-01-01. Instructions for updating: Use `tf.profiler.profile(graph, run_meta, op_log, cmd, options)`. Build `options` with `tf.profiler.ProfileOptionBuilder`. See README.md for details WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.compat.v1.graph_util.tensor_shape_from_node_def_name` W0420 01:46:14.496693 139993840469824 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/profiler/internal/flops_registry.py:142: tensor_shape_from_node_def_name (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.compat.v1.graph_util.tensor_shape_from_node_def_name` 151 ops no flops stats due to incomplete shapes. Parsing Inputs... Incomplete shape. =========================Options============================= -max_depth 10000 -min_bytes 0 -min_peak_bytes 0 -min_residual_bytes 0 -min_output_bytes 0 -min_micros 0 -min_accelerator_micros 0 -min_cpu_micros 0 -min_params 0 -min_float_ops 0 -min_occurrence 0 -step -1 -order_by name -account_type_regexes _trainable_variables -start_name_regexes .* -trim_name_regexes .*BatchNorm.* -show_name_regexes .* -hide_name_regexes -account_displayed_op_only true -select params -output stdout: ==================Model Analysis Report====================== Incomplete shape. Doc: scope: The nodes in the model graph are organized by their names, which is hierarchical like filesystem. param: Number of parameters (in the Variable). Profile: node name | # parameters _TFProfRoot (--/3.01m params) BoxPredictor_0 (--/22.48k params) BoxPredictor_0/BoxEncodingPredictor (--/6.92k params) BoxPredictor_0/BoxEncodingPredictor/biases (12, 12/12 params) BoxPredictor_0/BoxEncodingPredictor/weights (1x1x576x12, 6.91k/6.91k params) BoxPredictor_0/BoxEncodingPredictor_depthwise (--/5.18k params) BoxPredictor_0/BoxEncodingPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_0/BoxEncodingPredictor_depthwise/depthwise_weights (3x3x576x1, 5.18k/5.18k params) BoxPredictor_0/ClassPredictor (--/5.19k params) BoxPredictor_0/ClassPredictor/biases (9, 9/9 params) BoxPredictor_0/ClassPredictor/weights (1x1x576x9, 5.18k/5.18k params) BoxPredictor_0/ClassPredictor_depthwise (--/5.18k params) BoxPredictor_0/ClassPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_0/ClassPredictor_depthwise/depthwise_weights (3x3x576x1, 5.18k/5.18k params) BoxPredictor_1 (--/76.84k params) BoxPredictor_1/BoxEncodingPredictor (--/30.74k params) BoxPredictor_1/BoxEncodingPredictor/biases (24, 24/24 params) BoxPredictor_1/BoxEncodingPredictor/weights (1x1x1280x24, 30.72k/30.72k params) BoxPredictor_1/BoxEncodingPredictor_depthwise (--/11.52k params) BoxPredictor_1/BoxEncodingPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_1/BoxEncodingPredictor_depthwise/depthwise_weights (3x3x1280x1, 11.52k/11.52k params) BoxPredictor_1/ClassPredictor (--/23.06k params) BoxPredictor_1/ClassPredictor/biases (18, 18/18 params) BoxPredictor_1/ClassPredictor/weights (1x1x1280x18, 23.04k/23.04k params) BoxPredictor_1/ClassPredictor_depthwise (--/11.52k params) BoxPredictor_1/ClassPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_1/ClassPredictor_depthwise/depthwise_weights (3x3x1280x1, 11.52k/11.52k params) BoxPredictor_2 (--/30.76k params) BoxPredictor_2/BoxEncodingPredictor (--/12.31k params) BoxPredictor_2/BoxEncodingPredictor/biases (24, 24/24 params) BoxPredictor_2/BoxEncodingPredictor/weights (1x1x512x24, 12.29k/12.29k params) BoxPredictor_2/BoxEncodingPredictor_depthwise (--/4.61k params) BoxPredictor_2/BoxEncodingPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_2/BoxEncodingPredictor_depthwise/depthwise_weights (3x3x512x1, 4.61k/4.61k params) BoxPredictor_2/ClassPredictor (--/9.23k params) BoxPredictor_2/ClassPredictor/biases (18, 18/18 params) BoxPredictor_2/ClassPredictor/weights (1x1x512x18, 9.22k/9.22k params) BoxPredictor_2/ClassPredictor_depthwise (--/4.61k params) BoxPredictor_2/ClassPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_2/ClassPredictor_depthwise/depthwise_weights (3x3x512x1, 4.61k/4.61k params) BoxPredictor_3 (--/15.40k params) BoxPredictor_3/BoxEncodingPredictor (--/6.17k params) BoxPredictor_3/BoxEncodingPredictor/biases (24, 24/24 params) BoxPredictor_3/BoxEncodingPredictor/weights (1x1x256x24, 6.14k/6.14k params) BoxPredictor_3/BoxEncodingPredictor_depthwise (--/2.30k params) BoxPredictor_3/BoxEncodingPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_3/BoxEncodingPredictor_depthwise/depthwise_weights (3x3x256x1, 2.30k/2.30k params) BoxPredictor_3/ClassPredictor (--/4.63k params) BoxPredictor_3/ClassPredictor/biases (18, 18/18 params) BoxPredictor_3/ClassPredictor/weights (1x1x256x18, 4.61k/4.61k params) BoxPredictor_3/ClassPredictor_depthwise (--/2.30k params) BoxPredictor_3/ClassPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_3/ClassPredictor_depthwise/depthwise_weights (3x3x256x1, 2.30k/2.30k params) BoxPredictor_4 (--/15.40k params) BoxPredictor_4/BoxEncodingPredictor (--/6.17k params) BoxPredictor_4/BoxEncodingPredictor/biases (24, 24/24 params) BoxPredictor_4/BoxEncodingPredictor/weights (1x1x256x24, 6.14k/6.14k params) BoxPredictor_4/BoxEncodingPredictor_depthwise (--/2.30k params) BoxPredictor_4/BoxEncodingPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_4/BoxEncodingPredictor_depthwise/depthwise_weights (3x3x256x1, 2.30k/2.30k params) BoxPredictor_4/ClassPredictor (--/4.63k params) BoxPredictor_4/ClassPredictor/biases (18, 18/18 params) BoxPredictor_4/ClassPredictor/weights (1x1x256x18, 4.61k/4.61k params) BoxPredictor_4/ClassPredictor_depthwise (--/2.30k params) BoxPredictor_4/ClassPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_4/ClassPredictor_depthwise/depthwise_weights (3x3x256x1, 2.30k/2.30k params) BoxPredictor_5 (--/7.72k params) BoxPredictor_5/BoxEncodingPredictor (--/3.10k params) BoxPredictor_5/BoxEncodingPredictor/biases (24, 24/24 params) BoxPredictor_5/BoxEncodingPredictor/weights (1x1x128x24, 3.07k/3.07k params) BoxPredictor_5/BoxEncodingPredictor_depthwise (--/1.15k params) BoxPredictor_5/BoxEncodingPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_5/BoxEncodingPredictor_depthwise/depthwise_weights (3x3x128x1, 1.15k/1.15k params) BoxPredictor_5/ClassPredictor (--/2.32k params) BoxPredictor_5/ClassPredictor/biases (18, 18/18 params) BoxPredictor_5/ClassPredictor/weights (1x1x128x18, 2.30k/2.30k params) BoxPredictor_5/ClassPredictor_depthwise (--/1.15k params) BoxPredictor_5/ClassPredictor_depthwise/BatchNorm (--/0 params) BoxPredictor_5/ClassPredictor_depthwise/depthwise_weights (3x3x128x1, 1.15k/1.15k params) FeatureExtractor (--/2.84m params) FeatureExtractor/MobilenetV2 (--/2.84m params) FeatureExtractor/MobilenetV2/Conv (--/864 params) FeatureExtractor/MobilenetV2/Conv/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/Conv/weights (3x3x3x32, 864/864 params) FeatureExtractor/MobilenetV2/Conv_1 (--/409.60k params) FeatureExtractor/MobilenetV2/Conv_1/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/Conv_1/weights (1x1x320x1280, 409.60k/409.60k params) FeatureExtractor/MobilenetV2/expanded_conv (--/800 params) FeatureExtractor/MobilenetV2/expanded_conv/depthwise (--/288 params) FeatureExtractor/MobilenetV2/expanded_conv/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv/depthwise/depthwise_weights (3x3x32x1, 288/288 params) FeatureExtractor/MobilenetV2/expanded_conv/project (--/512 params) FeatureExtractor/MobilenetV2/expanded_conv/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv/project/weights (1x1x32x16, 512/512 params) FeatureExtractor/MobilenetV2/expanded_conv_1 (--/4.70k params) FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise (--/864 params) FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_1/depthwise/depthwise_weights (3x3x96x1, 864/864 params) FeatureExtractor/MobilenetV2/expanded_conv_1/expand (--/1.54k params) FeatureExtractor/MobilenetV2/expanded_conv_1/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_1/expand/weights (1x1x16x96, 1.54k/1.54k params) FeatureExtractor/MobilenetV2/expanded_conv_1/project (--/2.30k params) FeatureExtractor/MobilenetV2/expanded_conv_1/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_1/project/weights (1x1x96x24, 2.30k/2.30k params) FeatureExtractor/MobilenetV2/expanded_conv_10 (--/64.90k params) FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise (--/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_10/depthwise/depthwise_weights (3x3x384x1, 3.46k/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_10/expand (--/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_10/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_10/expand/weights (1x1x64x384, 24.58k/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_10/project (--/36.86k params) FeatureExtractor/MobilenetV2/expanded_conv_10/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_10/project/weights (1x1x384x96, 36.86k/36.86k params) FeatureExtractor/MobilenetV2/expanded_conv_11 (--/115.78k params) FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise (--/5.18k params) FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_11/depthwise/depthwise_weights (3x3x576x1, 5.18k/5.18k params) FeatureExtractor/MobilenetV2/expanded_conv_11/expand (--/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_11/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_11/expand/weights (1x1x96x576, 55.30k/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_11/project (--/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_11/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_11/project/weights (1x1x576x96, 55.30k/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_12 (--/115.78k params) FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise (--/5.18k params) FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_12/depthwise/depthwise_weights (3x3x576x1, 5.18k/5.18k params) FeatureExtractor/MobilenetV2/expanded_conv_12/expand (--/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_12/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_12/expand/weights (1x1x96x576, 55.30k/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_12/project (--/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_12/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_12/project/weights (1x1x576x96, 55.30k/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_13 (--/152.64k params) FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise (--/5.18k params) FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_13/depthwise/depthwise_weights (3x3x576x1, 5.18k/5.18k params) FeatureExtractor/MobilenetV2/expanded_conv_13/expand (--/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_13/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_13/expand/weights (1x1x96x576, 55.30k/55.30k params) FeatureExtractor/MobilenetV2/expanded_conv_13/project (--/92.16k params) FeatureExtractor/MobilenetV2/expanded_conv_13/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_13/project/weights (1x1x576x160, 92.16k/92.16k params) FeatureExtractor/MobilenetV2/expanded_conv_14 (--/315.84k params) FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise (--/8.64k params) FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_14/depthwise/depthwise_weights (3x3x960x1, 8.64k/8.64k params) FeatureExtractor/MobilenetV2/expanded_conv_14/expand (--/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_14/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_14/expand/weights (1x1x160x960, 153.60k/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_14/project (--/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_14/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_14/project/weights (1x1x960x160, 153.60k/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_15 (--/315.84k params) FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise (--/8.64k params) FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_15/depthwise/depthwise_weights (3x3x960x1, 8.64k/8.64k params) FeatureExtractor/MobilenetV2/expanded_conv_15/expand (--/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_15/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_15/expand/weights (1x1x160x960, 153.60k/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_15/project (--/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_15/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_15/project/weights (1x1x960x160, 153.60k/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_16 (--/469.44k params) FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise (--/8.64k params) FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_16/depthwise/depthwise_weights (3x3x960x1, 8.64k/8.64k params) FeatureExtractor/MobilenetV2/expanded_conv_16/expand (--/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_16/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_16/expand/weights (1x1x160x960, 153.60k/153.60k params) FeatureExtractor/MobilenetV2/expanded_conv_16/project (--/307.20k params) FeatureExtractor/MobilenetV2/expanded_conv_16/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_16/project/weights (1x1x960x320, 307.20k/307.20k params) FeatureExtractor/MobilenetV2/expanded_conv_2 (--/8.21k params) FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise (--/1.30k params) FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_2/depthwise/depthwise_weights (3x3x144x1, 1.30k/1.30k params) FeatureExtractor/MobilenetV2/expanded_conv_2/expand (--/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_2/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_2/expand/weights (1x1x24x144, 3.46k/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_2/project (--/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_2/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_2/project/weights (1x1x144x24, 3.46k/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_3 (--/9.36k params) FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise (--/1.30k params) FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_3/depthwise/depthwise_weights (3x3x144x1, 1.30k/1.30k params) FeatureExtractor/MobilenetV2/expanded_conv_3/expand (--/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_3/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_3/expand/weights (1x1x24x144, 3.46k/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_3/project (--/4.61k params) FeatureExtractor/MobilenetV2/expanded_conv_3/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_3/project/weights (1x1x144x32, 4.61k/4.61k params) FeatureExtractor/MobilenetV2/expanded_conv_4 (--/14.02k params) FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise (--/1.73k params) FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_4/depthwise/depthwise_weights (3x3x192x1, 1.73k/1.73k params) FeatureExtractor/MobilenetV2/expanded_conv_4/expand (--/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_4/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_4/expand/weights (1x1x32x192, 6.14k/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_4/project (--/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_4/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_4/project/weights (1x1x192x32, 6.14k/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_5 (--/14.02k params) FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise (--/1.73k params) FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_5/depthwise/depthwise_weights (3x3x192x1, 1.73k/1.73k params) FeatureExtractor/MobilenetV2/expanded_conv_5/expand (--/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_5/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_5/expand/weights (1x1x32x192, 6.14k/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_5/project (--/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_5/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_5/project/weights (1x1x192x32, 6.14k/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_6 (--/20.16k params) FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise (--/1.73k params) FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_6/depthwise/depthwise_weights (3x3x192x1, 1.73k/1.73k params) FeatureExtractor/MobilenetV2/expanded_conv_6/expand (--/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_6/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_6/expand/weights (1x1x32x192, 6.14k/6.14k params) FeatureExtractor/MobilenetV2/expanded_conv_6/project (--/12.29k params) FeatureExtractor/MobilenetV2/expanded_conv_6/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_6/project/weights (1x1x192x64, 12.29k/12.29k params) FeatureExtractor/MobilenetV2/expanded_conv_7 (--/52.61k params) FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise (--/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_7/depthwise/depthwise_weights (3x3x384x1, 3.46k/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_7/expand (--/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_7/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_7/expand/weights (1x1x64x384, 24.58k/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_7/project (--/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_7/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_7/project/weights (1x1x384x64, 24.58k/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_8 (--/52.61k params) FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise (--/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_8/depthwise/depthwise_weights (3x3x384x1, 3.46k/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_8/expand (--/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_8/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_8/expand/weights (1x1x64x384, 24.58k/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_8/project (--/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_8/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_8/project/weights (1x1x384x64, 24.58k/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_9 (--/52.61k params) FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise (--/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_9/depthwise/depthwise_weights (3x3x384x1, 3.46k/3.46k params) FeatureExtractor/MobilenetV2/expanded_conv_9/expand (--/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_9/expand/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_9/expand/weights (1x1x64x384, 24.58k/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_9/project (--/24.58k params) FeatureExtractor/MobilenetV2/expanded_conv_9/project/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/expanded_conv_9/project/weights (1x1x384x64, 24.58k/24.58k params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256 (--/327.68k params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_2_1x1_256/weights (1x1x1280x256, 327.68k/327.68k params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128 (--/65.54k params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_3_1x1_128/weights (1x1x512x128, 65.54k/65.54k params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128 (--/32.77k params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_4_1x1_128/weights (1x1x256x128, 32.77k/32.77k params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64 (--/16.38k params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_1_Conv2d_5_1x1_64/weights (1x1x256x64, 16.38k/16.38k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512 (--/131.07k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512/weights (1x1x256x512, 131.07k/131.07k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise (--/2.30k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_2_3x3_s2_512_depthwise/depthwise_weights (3x3x256x1, 2.30k/2.30k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256 (--/32.77k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256/weights (1x1x128x256, 32.77k/32.77k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256_depthwise (--/1.15k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256_depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_3_3x3_s2_256_depthwise/depthwise_weights (3x3x128x1, 1.15k/1.15k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256 (--/32.77k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256/weights (1x1x128x256, 32.77k/32.77k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256_depthwise (--/1.15k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256_depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_4_3x3_s2_256_depthwise/depthwise_weights (3x3x128x1, 1.15k/1.15k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128 (--/8.19k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128/weights (1x1x64x128, 8.19k/8.19k params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128_depthwise (--/576 params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128_depthwise/BatchNorm (--/0 params) FeatureExtractor/MobilenetV2/layer_19_2_Conv2d_5_3x3_s2_128_depthwise/depthwise_weights (3x3x64x1, 576/576 params) ======================End of Report========================== 151 ops no flops stats due to incomplete shapes. Parsing Inputs... Incomplete shape. =========================Options============================= -max_depth 10000 -min_bytes 0 -min_peak_bytes 0 -min_residual_bytes 0 -min_output_bytes 0 -min_micros 0 -min_accelerator_micros 0 -min_cpu_micros 0 -min_params 0 -min_float_ops 1 -min_occurrence 0 -step -1 -order_by float_ops -account_type_regexes .* -start_name_regexes .* -trim_name_regexes .*BatchNorm.*,.*Initializer.*,.*Regularizer.*,.*BiasAdd.* -show_name_regexes .* -hide_name_regexes -account_displayed_op_only true -select float_ops -output stdout: ==================Model Analysis Report====================== Incomplete shape. Doc: scope: The nodes in the model graph are organized by their names, which is hierarchical like filesystem. flops: Number of float operations. Note: Please read the implementation for the math behind it. Profile: node name | # float_ops _TFProfRoot (--/13.71k flops) MultipleGridAnchorGenerator/sub (2.17k/2.17k flops) MultipleGridAnchorGenerator/mul_20 (2.17k/2.17k flops) MultipleGridAnchorGenerator/mul_19 (2.17k/2.17k flops) MultipleGridAnchorGenerator/sub_1 (1.20k/1.20k flops) MultipleGridAnchorGenerator/mul_28 (1.20k/1.20k flops) MultipleGridAnchorGenerator/mul_27 (1.20k/1.20k flops) MultipleGridAnchorGenerator/mul_21 (1.08k/1.08k flops) MultipleGridAnchorGenerator/mul_29 (600/600 flops) MultipleGridAnchorGenerator/mul_35 (300/300 flops) MultipleGridAnchorGenerator/sub_2 (300/300 flops) MultipleGridAnchorGenerator/mul_36 (300/300 flops) MultipleGridAnchorGenerator/mul_37 (150/150 flops) MultipleGridAnchorGenerator/mul_43 (108/108 flops) MultipleGridAnchorGenerator/sub_3 (108/108 flops) MultipleGridAnchorGenerator/mul_44 (108/108 flops) MultipleGridAnchorGenerator/mul_45 (54/54 flops) MultipleGridAnchorGenerator/sub_4 (48/48 flops) MultipleGridAnchorGenerator/mul_52 (48/48 flops) MultipleGridAnchorGenerator/mul_51 (48/48 flops) MultipleGridAnchorGenerator/mul_53 (24/24 flops) MultipleGridAnchorGenerator/mul_18 (19/19 flops) MultipleGridAnchorGenerator/mul_17 (19/19 flops) MultipleGridAnchorGenerator/mul_59 (12/12 flops) MultipleGridAnchorGenerator/sub_5 (12/12 flops) MultipleGridAnchorGenerator/mul_60 (12/12 flops) MultipleGridAnchorGenerator/mul_25 (10/10 flops) MultipleGridAnchorGenerator/mul_26 (10/10 flops) MultipleGridAnchorGenerator/mul_47 (6/6 flops) MultipleGridAnchorGenerator/mul_40 (6/6 flops) MultipleGridAnchorGenerator/mul_61 (6/6 flops) MultipleGridAnchorGenerator/mul_46 (6/6 flops) MultipleGridAnchorGenerator/mul_31 (6/6 flops) MultipleGridAnchorGenerator/mul_48 (6/6 flops) MultipleGridAnchorGenerator/mul_56 (6/6 flops) MultipleGridAnchorGenerator/mul_55 (6/6 flops) MultipleGridAnchorGenerator/mul_54 (6/6 flops) MultipleGridAnchorGenerator/mul_39 (6/6 flops) MultipleGridAnchorGenerator/mul_38 (6/6 flops) MultipleGridAnchorGenerator/mul_32 (6/6 flops) MultipleGridAnchorGenerator/mul_30 (6/6 flops) MultipleGridAnchorGenerator/mul_24 (6/6 flops) MultipleGridAnchorGenerator/mul_23 (6/6 flops) MultipleGridAnchorGenerator/mul_22 (6/6 flops) MultipleGridAnchorGenerator/truediv_19 (6/6 flops) MultipleGridAnchorGenerator/truediv_18 (6/6 flops) MultipleGridAnchorGenerator/truediv_17 (6/6 flops) MultipleGridAnchorGenerator/truediv_16 (6/6 flops) MultipleGridAnchorGenerator/truediv_15 (6/6 flops) MultipleGridAnchorGenerator/mul_34 (5/5 flops) MultipleGridAnchorGenerator/mul_33 (5/5 flops) MultipleGridAnchorGenerator/mul_42 (3/3 flops) MultipleGridAnchorGenerator/truediv_14 (3/3 flops) MultipleGridAnchorGenerator/mul_41 (3/3 flops) MultipleGridAnchorGenerator/mul_14 (3/3 flops) MultipleGridAnchorGenerator/mul_15 (3/3 flops) MultipleGridAnchorGenerator/mul_16 (3/3 flops) MultipleGridAnchorGenerator/mul_49 (2/2 flops) MultipleGridAnchorGenerator/mul_50 (2/2 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater_1 (1/1 flops) MultipleGridAnchorGenerator/truediv (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/SortByField_1/Equal (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/sub (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/sub_1 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/sub_2 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/sub_3 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater_2 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater_9 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater_8 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater_7 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/SortByField/Equal (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater_6 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater_5 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater_4 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/Greater_3 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_2 (1/1 flops) Preprocessor/map/while/Less_1 (1/1 flops) Preprocessor/map/while/Less (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/ones/Less (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_9 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_8 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_7 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_6 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_5 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_4 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_3 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_1 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_19 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_18 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_17 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_16 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_15 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_14 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_13 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_12 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_11 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/PadOrClipBoxList/sub_10 (1/1 flops) MultipleGridAnchorGenerator/mul_5 (1/1 flops) MultipleGridAnchorGenerator/truediv_10 (1/1 flops) MultipleGridAnchorGenerator/truediv_1 (1/1 flops) MultipleGridAnchorGenerator/Minimum (1/1 flops) MultipleGridAnchorGenerator/mul_9 (1/1 flops) MultipleGridAnchorGenerator/mul_8 (1/1 flops) MultipleGridAnchorGenerator/mul_7 (1/1 flops) MultipleGridAnchorGenerator/mul_6 (1/1 flops) MultipleGridAnchorGenerator/mul_58 (1/1 flops) MultipleGridAnchorGenerator/mul_57 (1/1 flops) MultipleGridAnchorGenerator/truediv_11 (1/1 flops) MultipleGridAnchorGenerator/mul_4 (1/1 flops) MultipleGridAnchorGenerator/mul_3 (1/1 flops) MultipleGridAnchorGenerator/mul_2 (1/1 flops) MultipleGridAnchorGenerator/mul_13 (1/1 flops) MultipleGridAnchorGenerator/mul_12 (1/1 flops) MultipleGridAnchorGenerator/mul_11 (1/1 flops) MultipleGridAnchorGenerator/mul_10 (1/1 flops) MultipleGridAnchorGenerator/mul_1 (1/1 flops) MultipleGridAnchorGenerator/mul (1/1 flops) MultipleGridAnchorGenerator/truediv_9 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Minimum_1 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Minimum (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Greater (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/ChangeCoordinateFrame/truediv_1 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/ChangeCoordinateFrame/truediv (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/ChangeCoordinateFrame/sub_1 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/ChangeCoordinateFrame/sub (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/Less_1 (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/Less (1/1 flops) Postprocessor/BatchMultiClassNonMaxSuppression/map/while/MultiClassNonMaxSuppression/Minimum_2 (1/1 flops) MultipleGridAnchorGenerator/truediv_8 (1/1 flops) MultipleGridAnchorGenerator/truediv_7 (1/1 flops) MultipleGridAnchorGenerator/truediv_6 (1/1 flops) MultipleGridAnchorGenerator/truediv_5 (1/1 flops) MultipleGridAnchorGenerator/truediv_4 (1/1 flops) MultipleGridAnchorGenerator/truediv_3 (1/1 flops) MultipleGridAnchorGenerator/truediv_2 (1/1 flops) MultipleGridAnchorGenerator/truediv_13 (1/1 flops) MultipleGridAnchorGenerator/truediv_12 (1/1 flops) ======================End of Report========================== 2023-04-20 01:46:16.676536: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2023-04-20 01:46:16.702787: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:16.703729: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:46:16.703997: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:46:16.705580: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:46:16.707081: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:46:16.707360: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:46:16.709260: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:46:16.710674: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:46:16.714912: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:46:16.715173: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:16.716207: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:16.717122: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:46:16.717615: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2023-04-20 01:46:16.729465: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz 2023-04-20 01:46:16.730566: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556e84358250 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2023-04-20 01:46:16.730601: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2023-04-20 01:46:16.843084: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:16.844114: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x556e82ffdfd0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2023-04-20 01:46:16.844140: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0 2023-04-20 01:46:16.844364: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:16.845271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:46:16.845349: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:46:16.845390: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:46:16.845424: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:46:16.845444: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:46:16.845460: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:46:16.845475: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:46:16.845491: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:46:16.845549: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:16.846475: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:16.847334: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:46:16.847409: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:46:16.849069: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 01:46:16.849086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 01:46:16.849097: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 01:46:16.849212: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:16.850163: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:16.851077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-20000 I0420 01:46:16.854092 139993840469824 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-20000 WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. W0420 01:46:19.040910 139993840469824 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version. Instructions for updating: Use standard file APIs to check for files with this prefix. 2023-04-20 01:46:19.863210: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:19.863754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:46:19.863827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:46:19.863847: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:46:19.863863: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:46:19.863881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:46:19.863905: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:46:19.863927: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:46:19.863943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:46:19.864001: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:19.864414: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:19.864760: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:46:19.864795: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 01:46:19.864803: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 01:46:19.864809: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 01:46:19.864892: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:19.865300: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:19.865674: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) INFO:tensorflow:Restoring parameters from ./my_model_dir/model.ckpt-20000 I0420 01:46:19.866678 139993840469824 saver.py:1284] Restoring parameters from ./my_model_dir/model.ckpt-20000 WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.compat.v1.graph_util.convert_variables_to_constants` W0420 01:46:20.739303 139993840469824 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/tools/freeze_graph.py:233: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.compat.v1.graph_util.convert_variables_to_constants` WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.compat.v1.graph_util.extract_sub_graph` W0420 01:46:20.739585 139993840469824 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/tensorflow_core/python/framework/graph_util_impl.py:277: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version. Instructions for updating: Use `tf.compat.v1.graph_util.extract_sub_graph` INFO:tensorflow:Froze 404 variables. I0420 01:46:21.221033 139993840469824 graph_util_impl.py:334] Froze 404 variables. INFO:tensorflow:Converted 404 variables to const ops. I0420 01:46:21.289041 139993840469824 graph_util_impl.py:394] Converted 404 variables to const ops. 2023-04-20 01:46:21.397261: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:21.397828: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285 pciBusID: 0000:00:0d.0 2023-04-20 01:46:21.397919: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2023-04-20 01:46:21.397941: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2023-04-20 01:46:21.397958: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2023-04-20 01:46:21.397973: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2023-04-20 01:46:21.397988: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2023-04-20 01:46:21.398003: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2023-04-20 01:46:21.398018: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2023-04-20 01:46:21.398075: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:21.398499: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:21.398856: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2023-04-20 01:46:21.398894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-04-20 01:46:21.398913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2023-04-20 01:46:21.398920: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2023-04-20 01:46:21.399086: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:21.399519: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-04-20 01:46:21.399910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 15216 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:0d.0, compute capability: 6.0) WARNING:tensorflow:From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/exporter.py:391: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. W0420 01:46:21.836742 139993840469824 deprecation.py:323] From /home/ma-user/anaconda3/envs/py36/lib/python3.6/site-packages/object_detection/exporter.py:391: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version. Instructions for updating: This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info. INFO:tensorflow:No assets to save. I0420 01:46:21.837533 139993840469824 builder_impl.py:640] No assets to save. INFO:tensorflow:No assets to write. I0420 01:46:21.837623 139993840469824 builder_impl.py:460] No assets to write. INFO:tensorflow:SavedModel written to: ./inference_model/saved_model/saved_model.pb I0420 01:46:22.210551 139993840469824 builder_impl.py:425] SavedModel written to: ./inference_model/saved_model/saved_model.pb INFO:tensorflow:Writing pipeline config file to ./inference_model/pipeline.config I0420 01:46:22.226193 139993840469824 config_util.py:254] Writing pipeline config file to ./inference_model/pipeline.config
import os
import moxing as mox
INFO:root:Using MoXing-v2.1.0.5d9c87c8-5d9c87c8 INFO:root:Using OBS-Python-SDK-3.20.9.1
mox.file.copy_parallel('object_detection/inference_model', 'obs://houyansong/inference_model')
版本号 | 版本ID | 发布时间 | 发布状态 | 版本说明 |
---|
1.0.0 | 1.0.0 | 2023-04-19 18:02 | 已完成 | Initial release. |
请输入预览内容
预测结果显示
若您怀疑合法知识产权遭受侵犯,可以通过此链接进行投诉与建议。