建议使用以下浏览器，以获得最佳体验。 IE 9.0+以上版本 Chrome 31+ 谷歌浏览器 Firefox 30+ 火狐浏览器
温馨提示

抱歉，您需设置社区昵称后才能参与社区互动！

前往修改
我再想想
✖

深圳开放数据应用创新大赛

话题 : 83 成员 : 215

加入HCSD

训练40轮次，第34轮 GPU报错 CUDA_ERROR_ECC_UNCORRECTABLE

哈比 2020/5/13 896

Epoch 34/40


  1/420 [..............................] - ETA: 5:25 - loss: 18.5104

  2/420 [..............................] - ETA: 5:15 - loss: 18.3590

  3/420 [..............................] - ETA: 5:11 - loss: 17.5936

  4/420 [..............................] - ETA: 5:08 - loss: 16.8200

  5/420 [..............................] - ETA: 5:07 - loss: 17.1277

  6/420 [..............................] - ETA: 5:05 - loss: 17.3498

  7/420 [..............................] - ETA: 5:04 - loss: 17.8328

  8/420 [..............................] - ETA: 5:03 - loss: 17.6816

  9/420 [..............................] - ETA: 5:02 - loss: 18.1138

 10/420 [..............................] - ETA: 5:01 - loss: 18.3598

 11/420 [..............................] - ETA: 5:00 - loss: 18.1559

 12/420 [..............................] - ETA: 4:59 - loss: 17.9971

 13/420 [..............................] - ETA: 4:58 - loss: 18.5058

 14/420 [>.............................] - ETA: 4:58 - loss: 18.3141

 15/420 [>.............................] - ETA: 4:57 - loss: 18.0721

 16/420 [>.............................] - ETA: 5:05 - loss: 18.3228

 17/420 [>.............................] - ETA: 5:36 - loss: 18.3057

 18/420 [>.............................] - ETA: 6:02 - loss: 18.1159

 19/420 [>.............................] - ETA: 6:26 - loss: 18.1399

 20/420 [>.............................] - ETA: 6:52 - loss: 18.0465

 21/420 [>.............................] - ETA: 7:11 - loss: 17.9300

 22/420 [>.............................] - ETA: 7:31 - loss: 17.8470

 23/420 [>.............................] - ETA: 7:45 - loss: 18.1962

 24/420 [>.............................] - ETA: 7:59 - loss: 18.3780

 25/420 [>.............................] - ETA: 8:10 - loss: 18.3227

 26/420 [>.............................] - ETA: 8:19 - loss: 18.1642

 27/420 [>.............................] - ETA: 8:27 - loss: 18.1588

 28/420 [=>............................] - ETA: 8:36 - loss: 18.1718

 29/420 [=>............................] - ETA: 8:45 - loss: 18.2434

 30/420 [=>............................] - ETA: 8:52 - loss: 18.2517

 31/420 [=>............................] - ETA: 8:59 - loss: 18.1228

 32/420 [=>............................] - ETA: 9:04 - loss: 18.1457

 33/420 [=>............................] - ETA: 9:12 - loss: 18.0834

 34/420 [=>............................] - ETA: 9:17 - loss: 18.0259

 35/420 [=>............................] - ETA: 9:21 - loss: 18.0105

 36/420 [=>............................] - ETA: 9:24 - loss: 17.9040

 37/420 [=>............................] - ETA: 9:30 - loss: 17.9191

 38/420 [=>............................] - ETA: 9:32 - loss: 17.9225

 39/420 [=>............................] - ETA: 9:35 - loss: 17.8731

 40/420 [=>............................] - ETA: 9:37 - loss: 17.8323

 41/420 [=>............................] - ETA: 9:40 - loss: 17.7485

 42/420 [==>...........................] - ETA: 9:42 - loss: 17.6677

 43/420 [==>...........................] - ETA: 9:44 - loss: 17.6094

 44/420 [==>...........................] - ETA: 9:47 - loss: 17.5825

 45/420 [==>...........................] - ETA: 9:50 - loss: 17.5997

 46/420 [==>...........................] - ETA: 9:53 - loss: 17.6347

 47/420 [==>...........................] - ETA: 9:55 - loss: 17.6082

 48/420 [==>...........................] - ETA: 9:57 - loss: 17.5481

 49/420 [==>...........................] - ETA: 9:58 - loss: 17.5517

 50/420 [==>...........................] - ETA: 10:01 - loss: 17.5625

 51/420 [==>...........................] - ETA: 10:02 - loss: 17.5972

 52/420 [==>...........................] - ETA: 10:03 - loss: 17.5366

 53/420 [==>...........................] - ETA: 10:03 - loss: 17.5583

 54/420 [==>...........................] - ETA: 10:04 - loss: 17.5761

 55/420 [==>...........................] - ETA: 10:04 - loss: 17.5531

 56/420 [===>..........................] - ETA: 10:05 - loss: 17.6171

 57/420 [===>..........................] - ETA: 10:07 - loss: 17.5667

 58/420 [===>..........................] - ETA: 10:08 - loss: 17.5386

 59/420 [===>..........................] - ETA: 10:08 - loss: 17.6176

 60/420 [===>..........................] - ETA: 10:08 - loss: 17.6191

 61/420 [===>..........................] - ETA: 10:08 - loss: 17.5991

 62/420 [===>..........................] - ETA: 10:08 - loss: 17.6671

 63/420 [===>..........................] - ETA: 10:08 - loss: 17.6406

 64/420 [===>..........................] - ETA: 10:07 - loss: 17.6188

 65/420 [===>..........................] - ETA: 10:07 - loss: 17.6821

 66/420 [===>..........................] - ETA: 10:06 - loss: 17.6399

 67/420 [===>..........................] - ETA: 10:06 - loss: 17.6129

 68/420 [===>..........................] - ETA: 10:05 - loss: 17.6564

 69/420 [===>..........................] - ETA: 10:05 - loss: 17.6652

 70/420 [====>.........................] - ETA: 10:05 - loss: 17.6599

 71/420 [====>.........................] - ETA: 10:06 - loss: 17.6755

 72/420 [====>.........................] - ETA: 10:07 - loss: 17.6530

 73/420 [====>.........................] - ETA: 10:06 - loss: 17.6436

 74/420 [====>.........................] - ETA: 10:07 - loss: 17.6161

 75/420 [====>.........................] - ETA: 10:06 - loss: 17.6090

 76/420 [====>.........................] - ETA: 10:05 - loss: 17.6377

 77/420 [====>.........................] - ETA: 10:04 - loss: 17.6633

 78/420 [====>.........................] - ETA: 10:03 - loss: 17.6715

 79/420 [====>.........................] - ETA: 10:03 - loss: 17.6608

 80/420 [====>.........................] - ETA: 10:02 - loss: 17.6699

 81/420 [====>.........................] - ETA: 10:01 - loss: 17.6269

 82/420 [====>.........................] - ETA: 10:01 - loss: 17.6590

 83/420 [====>.........................] - ETA: 10:00 - loss: 17.6194

 84/420 [=====>........................] - ETA: 9:59 - loss: 17.6506 

 85/420 [=====>........................] - ETA: 9:58 - loss: 17.6208

 86/420 [=====>........................] - ETA: 9:56 - loss: 17.5950

 87/420 [=====>........................] - ETA: 9:57 - loss: 17.6787

 88/420 [=====>........................] - ETA: 9:56 - loss: 17.7093

 89/420 [=====>........................] - ETA: 9:54 - loss: 17.7106

 90/420 [=====>........................] - ETA: 9:53 - loss: 17.7178

 91/420 [=====>........................] - ETA: 9:52 - loss: 17.7279

 92/420 [=====>........................] - ETA: 9:50 - loss: 17.7064

 93/420 [=====>........................] - ETA: 9:50 - loss: 17.6867

 94/420 [=====>........................] - ETA: 9:48 - loss: 17.6610

 95/420 [=====>........................] - ETA: 9:47 - loss: 17.6718

 96/420 [=====>........................] - ETA: 9:48 - loss: 17.6703

 97/420 [=====>........................] - ETA: 9:46 - loss: 17.6712

 98/420 [======>.......................] - ETA: 9:45 - loss: 17.6460

 99/420 [======>.......................] - ETA: 9:43 - loss: 17.6237

100/420 [======>.......................] - ETA: 9:42 - loss: 17.6049

101/420 [======>.......................] - ETA: 9:41 - loss: 17.6579

102/420 [======>.......................] - ETA: 9:39 - loss: 17.6838

103/420 [======>.......................] - ETA: 9:38 - loss: 17.6556

104/420 [======>.......................] - ETA: 9:37 - loss: 17.6797

105/420 [======>.......................] - ETA: 9:36 - loss: 17.6538

106/420 [======>.......................] - ETA: 9:34 - loss: 17.6397

107/420 [======>.......................] - ETA: 9:33 - loss: 17.6479

108/420 [======>.......................] - ETA: 9:31 - loss: 17.6621

109/420 [======>.......................] - ETA: 9:30 - loss: 17.7415

110/420 [======>.......................] - ETA: 9:28 - loss: 17.7278

111/420 [======>.......................] - ETA: 9:27 - loss: 17.7178

112/420 [=======>......................] - ETA: 9:25 - loss: 17.7059

113/420 [=======>......................] - ETA: 9:24 - loss: 17.7182

114/420 [=======>......................] - ETA: 9:22 - loss: 17.7445

115/420 [=======>......................] - ETA: 9:21 - loss: 17.7450

116/420 [=======>......................] - ETA: 9:19 - loss: 17.7762

117/420 [=======>......................] - ETA: 9:18 - loss: 17.7474

118/420 [=======>......................] - ETA: 9:16 - loss: 17.7299

119/420 [=======>......................] - ETA: 9:14 - loss: 17.7126

120/420 [=======>......................] - ETA: 9:13 - loss: 17.6995

121/420 [=======>......................] - ETA: 9:11 - loss: 17.7060

122/420 [=======>......................] - ETA: 9:09 - loss: 17.6817

123/420 [=======>......................] - ETA: 9:08 - loss: 17.6922

124/420 [=======>......................] - ETA: 9:06 - loss: 17.7381

125/420 [=======>......................] - ETA: 9:05 - loss: 17.7329

126/420 [========>.....................] - ETA: 9:04 - loss: 17.7082

127/420 [========>.....................] - ETA: 9:03 - loss: 17.6901

128/420 [========>.....................] - ETA: 9:01 - loss: 17.6922

129/420 [========>.....................] - ETA: 9:00 - loss: 17.6643

130/420 [========>.....................] - ETA: 8:58 - loss: 17.6323

131/420 [========>.....................] - ETA: 8:57 - loss: 17.6211

132/420 [========>.....................] - ETA: 8:55 - loss: 17.6196

133/420 [========>.....................] - ETA: 8:53 - loss: 17.6023

134/420 [========>.....................] - ETA: 8:52 - loss: 17.5932

135/420 [========>.....................] - ETA: 8:50 - loss: 17.6231

136/420 [========>.....................] - ETA: 8:49 - loss: 17.6245

137/420 [========>.....................] - ETA: 8:47 - loss: 17.6068

138/420 [========>.....................] - ETA: 8:46 - loss: 17.6057

139/420 [========>.....................] - ETA: 8:44 - loss: 17.5904

140/420 [=========>....................] - ETA: 8:42 - loss: 17.6503

141/420 [=========>....................] - ETA: 8:41 - loss: 17.6363

142/420 [=========>....................] - ETA: 8:39 - loss: 17.6104

143/420 [=========>....................] - ETA: 8:37 - loss: 17.6305

144/420 [=========>....................] - ETA: 8:36 - loss: 17.6245

145/420 [=========>....................] - ETA: 8:34 - loss: 17.6543

146/420 [=========>....................] - ETA: 8:32 - loss: 17.6592

147/420 [=========>....................] - ETA: 8:31 - loss: 17.6719

148/420 [=========>....................] - ETA: 8:29 - loss: 17.6835

149/420 [=========>....................] - ETA: 8:27 - loss: 17.6740

150/420 [=========>....................] - ETA: 8:26 - loss: 17.6728

151/420 [=========>....................] - ETA: 8:24 - loss: 17.6697

152/420 [=========>....................] - ETA: 8:22 - loss: 17.6640

153/420 [=========>....................] - ETA: 8:20 - loss: 17.6718

154/420 [==========>...................] - ETA: 8:19 - loss: 17.6634

155/420 [==========>...................] - ETA: 8:17 - loss: 17.6540

156/420 [==========>...................] - ETA: 8:15 - loss: 17.6298

157/420 [==========>...................] - ETA: 8:14 - loss: 17.6507

158/420 [==========>...................] - ETA: 8:12 - loss: 17.6619

159/420 [==========>...................] - ETA: 8:10 - loss: 17.6702

160/420 [==========>...................] - ETA: 8:09 - loss: 17.6659

161/420 [==========>...................] - ETA: 8:07 - loss: 17.6564

162/420 [==========>...................] - ETA: 8:05 - loss: 17.6355

163/420 [==========>...................] - ETA: 8:03 - loss: 17.6250

164/420 [==========>...................] - ETA: 8:02 - loss: 17.6081

165/420 [==========>...................] - ETA: 8:00 - loss: 17.6149

166/420 [==========>...................] - ETA: 7:58 - loss: 17.6159

167/420 [==========>...................] - ETA: 7:57 - loss: 17.6334

168/420 [===========>..................] - ETA: 7:55 - loss: 17.6539

169/420 [===========>..................] - ETA: 7:53 - loss: 17.6519

170/420 [===========>..................] - ETA: 7:52 - loss: 17.6496

171/420 [===========>..................] - ETA: 7:50 - loss: 17.6317

172/420 [===========>..................] - ETA: 7:48 - loss: 17.6333

173/420 [===========>..................] - ETA: 7:46 - loss: 17.6160

174/420 [===========>..................] - ETA: 7:44 - loss: 17.5985

175/420 [===========>..................] - ETA: 7:43 - loss: 17.5950

176/420 [===========>..................] - ETA: 7:41 - loss: 17.5858

177/420 [===========>..................] - ETA: 7:39 - loss: 17.5825

178/420 [===========>..................] - ETA: 7:37 - loss: 17.5756

179/420 [===========>..................] - ETA: 7:36 - loss: 17.5573

180/420 [===========>..................] - ETA: 7:34 - loss: 17.5459

181/420 [===========>..................] - ETA: 7:32 - loss: 17.5623

182/420 [============>.................] - ETA: 7:31 - loss: 17.5613

183/420 [============>.................] - ETA: 7:29 - loss: 17.5565

184/420 [============>.................] - ETA: 7:27 - loss: 17.5607

185/420 [============>.................] - ETA: 7:26 - loss: 17.5648

186/420 [============>.................] - ETA: 7:24 - loss: 17.5515

187/420 [============>.................] - ETA: 7:22 - loss: 17.5523

188/420 [============>.................] - ETA: 7:20 - loss: 17.5423

189/420 [============>.................] - ETA: 7:18 - loss: 17.5345

190/420 [============>.................] - ETA: 7:16 - loss: 17.5427

191/420 [============>.................] - ETA: 7:15 - loss: 17.5267

192/420 [============>.................] - ETA: 7:13 - loss: 17.5382

193/420 [============>.................] - ETA: 7:11 - loss: 17.5370

194/420 [============>.................] - ETA: 7:10 - loss: 17.5292

195/420 [============>.................] - ETA: 7:08 - loss: 17.5552

196/420 [=============>................] - ETA: 7:06 - loss: 17.5504

197/420 [=============>................] - ETA: 7:04 - loss: 17.5789

198/420 [=============>................] - ETA: 7:02 - loss: 17.5774

199/420 [=============>................] - ETA: 7:01 - loss: 17.5749

200/420 [=============>................] - ETA: 6:59 - loss: 17.5639

201/420 [=============>................] - ETA: 6:58 - loss: 17.5524

202/420 [=============>................] - ETA: 6:56 - loss: 17.5778

203/420 [=============>................] - ETA: 6:54 - loss: 17.5822

204/420 [=============>................] - ETA: 6:52 - loss: 17.5940

205/420 [=============>................] - ETA: 6:51 - loss: 17.5882

206/420 [=============>................] - ETA: 6:49 - loss: 17.5833

207/420 [=============>................] - ETA: 6:47 - loss: 17.6033

208/420 [=============>................] - ETA: 6:45 - loss: 17.6090

209/420 [=============>................] - ETA: 6:44 - loss: 17.6041

210/420 [==============>...............] - ETA: 6:42 - loss: 17.5981

211/420 [==============>...............] - ETA: 6:40 - loss: 17.5890

212/420 [==============>...............] - ETA: 6:38 - loss: 17.5718

213/420 [==============>...............] - ETA: 6:37 - loss: 17.5709

214/420 [==============>...............] - ETA: 6:35 - loss: 17.5791

215/420 [==============>...............] - ETA: 6:33 - loss: 17.5735

216/420 [==============>...............] - ETA: 6:31 - loss: 17.5802

217/420 [==============>...............] - ETA: 6:29 - loss: 17.5741

218/420 [==============>...............] - ETA: 6:27 - loss: 17.5936

219/420 [==============>...............] - ETA: 6:26 - loss: 17.5868

220/420 [==============>...............] - ETA: 6:24 - loss: 17.5901

221/420 [==============>...............] - ETA: 6:22 - loss: 17.6003

222/420 [==============>...............] - ETA: 6:20 - loss: 17.5995

223/420 [==============>...............] - ETA: 6:18 - loss: 17.6113

224/420 [===============>..............] - ETA: 6:16 - loss: 17.6068

225/420 [===============>..............] - ETA: 6:14 - loss: 17.6029

226/420 [===============>..............] - ETA: 6:12 - loss: 17.5965

227/420 [===============>..............] - ETA: 6:11 - loss: 17.5887

228/420 [===============>..............] - ETA: 6:09 - loss: 17.5917

229/420 [===============>..............] - ETA: 6:07 - loss: 17.5936

230/420 [===============>..............] - ETA: 6:05 - loss: 17.5993

231/420 [===============>..............] - ETA: 6:03 - loss: 17.5853

232/420 [===============>..............] - ETA: 6:01 - loss: 17.5905

233/420 [===============>..............] - ETA: 6:00 - loss: 17.5860

234/420 [===============>..............] - ETA: 5:58 - loss: 17.5906

235/420 [===============>..............] - ETA: 5:56 - loss: 17.5839

236/420 [===============>..............] - ETA: 5:54 - loss: 17.5649

237/420 [===============>..............] - ETA: 5:52 - loss: 17.5584

238/420 [================>.............] - ETA: 5:50 - loss: 17.54732020-05-13 00:14:49.552234: E tensorflow/stream_executor/cuda/cuda_event.cc:48] Error polling for event status: failed to query event: CUDA_ERROR_ECC_UNCORRECTABLE: uncorrectable ECC error encountered

2020-05-13 00:14:49.552296: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:274] Unexpected Event status: 1

[Modelarts Service Log]Training end with return code: 134

[Modelarts Service Log]Training completed.


训练GPU报错

上一篇>训练作业，无法显示日志
下一篇>批量预测失败, run batch task failed. 评分失败

回复 (1)

云上玩AI
0 0
2020/5/20 12:52

问题解决了吗

上划加载中
说点什么...
联盟首页 发表话题
返回列表 上一篇 下一篇
回复话题 发表话题
标签
您还可以添加5个标签
  • 没有搜索到和“关键字”相关的标签
  • 云产品
  • 解决方案
  • 技术领域
  • 通用技术
  • 平台功能
取消

哈比

角色：成员

话题：1

发消息
发表于2020年05月13日 10:24:30 8961
直达本楼层的链接
楼主 正序浏览 只看该作者
训练40轮次，第34轮 GPU报错 CUDA_ERROR_ECC_UNCORRECTABLE

Epoch 34/40


  1/420 [..............................] - ETA: 5:25 - loss: 18.5104

  2/420 [..............................] - ETA: 5:15 - loss: 18.3590

  3/420 [..............................] - ETA: 5:11 - loss: 17.5936

  4/420 [..............................] - ETA: 5:08 - loss: 16.8200

  5/420 [..............................] - ETA: 5:07 - loss: 17.1277

  6/420 [..............................] - ETA: 5:05 - loss: 17.3498

  7/420 [..............................] - ETA: 5:04 - loss: 17.8328

  8/420 [..............................] - ETA: 5:03 - loss: 17.6816

  9/420 [..............................] - ETA: 5:02 - loss: 18.1138

 10/420 [..............................] - ETA: 5:01 - loss: 18.3598

 11/420 [..............................] - ETA: 5:00 - loss: 18.1559

 12/420 [..............................] - ETA: 4:59 - loss: 17.9971

 13/420 [..............................] - ETA: 4:58 - loss: 18.5058

 14/420 [>.............................] - ETA: 4:58 - loss: 18.3141

 15/420 [>.............................] - ETA: 4:57 - loss: 18.0721

 16/420 [>.............................] - ETA: 5:05 - loss: 18.3228

 17/420 [>.............................] - ETA: 5:36 - loss: 18.3057

 18/420 [>.............................] - ETA: 6:02 - loss: 18.1159

 19/420 [>.............................] - ETA: 6:26 - loss: 18.1399

 20/420 [>.............................] - ETA: 6:52 - loss: 18.0465

 21/420 [>.............................] - ETA: 7:11 - loss: 17.9300

 22/420 [>.............................] - ETA: 7:31 - loss: 17.8470

 23/420 [>.............................] - ETA: 7:45 - loss: 18.1962

 24/420 [>.............................] - ETA: 7:59 - loss: 18.3780

 25/420 [>.............................] - ETA: 8:10 - loss: 18.3227

 26/420 [>.............................] - ETA: 8:19 - loss: 18.1642

 27/420 [>.............................] - ETA: 8:27 - loss: 18.1588

 28/420 [=>............................] - ETA: 8:36 - loss: 18.1718

 29/420 [=>............................] - ETA: 8:45 - loss: 18.2434

 30/420 [=>............................] - ETA: 8:52 - loss: 18.2517

 31/420 [=>............................] - ETA: 8:59 - loss: 18.1228

 32/420 [=>............................] - ETA: 9:04 - loss: 18.1457

 33/420 [=>............................] - ETA: 9:12 - loss: 18.0834

 34/420 [=>............................] - ETA: 9:17 - loss: 18.0259

 35/420 [=>............................] - ETA: 9:21 - loss: 18.0105

 36/420 [=>............................] - ETA: 9:24 - loss: 17.9040

 37/420 [=>............................] - ETA: 9:30 - loss: 17.9191

 38/420 [=>............................] - ETA: 9:32 - loss: 17.9225

 39/420 [=>............................] - ETA: 9:35 - loss: 17.8731

 40/420 [=>............................] - ETA: 9:37 - loss: 17.8323

 41/420 [=>............................] - ETA: 9:40 - loss: 17.7485

 42/420 [==>...........................] - ETA: 9:42 - loss: 17.6677

 43/420 [==>...........................] - ETA: 9:44 - loss: 17.6094

 44/420 [==>...........................] - ETA: 9:47 - loss: 17.5825

 45/420 [==>...........................] - ETA: 9:50 - loss: 17.5997

 46/420 [==>...........................] - ETA: 9:53 - loss: 17.6347

 47/420 [==>...........................] - ETA: 9:55 - loss: 17.6082

 48/420 [==>...........................] - ETA: 9:57 - loss: 17.5481

 49/420 [==>...........................] - ETA: 9:58 - loss: 17.5517

 50/420 [==>...........................] - ETA: 10:01 - loss: 17.5625

 51/420 [==>...........................] - ETA: 10:02 - loss: 17.5972

 52/420 [==>...........................] - ETA: 10:03 - loss: 17.5366

 53/420 [==>...........................] - ETA: 10:03 - loss: 17.5583

 54/420 [==>...........................] - ETA: 10:04 - loss: 17.5761

 55/420 [==>...........................] - ETA: 10:04 - loss: 17.5531

 56/420 [===>..........................] - ETA: 10:05 - loss: 17.6171

 57/420 [===>..........................] - ETA: 10:07 - loss: 17.5667

 58/420 [===>..........................] - ETA: 10:08 - loss: 17.5386

 59/420 [===>..........................] - ETA: 10:08 - loss: 17.6176

 60/420 [===>..........................] - ETA: 10:08 - loss: 17.6191

 61/420 [===>..........................] - ETA: 10:08 - loss: 17.5991

 62/420 [===>..........................] - ETA: 10:08 - loss: 17.6671

 63/420 [===>..........................] - ETA: 10:08 - loss: 17.6406

 64/420 [===>..........................] - ETA: 10:07 - loss: 17.6188

 65/420 [===>..........................] - ETA: 10:07 - loss: 17.6821

 66/420 [===>..........................] - ETA: 10:06 - loss: 17.6399

 67/420 [===>..........................] - ETA: 10:06 - loss: 17.6129

 68/420 [===>..........................] - ETA: 10:05 - loss: 17.6564

 69/420 [===>..........................] - ETA: 10:05 - loss: 17.6652

 70/420 [====>.........................] - ETA: 10:05 - loss: 17.6599

 71/420 [====>.........................] - ETA: 10:06 - loss: 17.6755

 72/420 [====>.........................] - ETA: 10:07 - loss: 17.6530

 73/420 [====>.........................] - ETA: 10:06 - loss: 17.6436

 74/420 [====>.........................] - ETA: 10:07 - loss: 17.6161

 75/420 [====>.........................] - ETA: 10:06 - loss: 17.6090

 76/420 [====>.........................] - ETA: 10:05 - loss: 17.6377

 77/420 [====>.........................] - ETA: 10:04 - loss: 17.6633

 78/420 [====>.........................] - ETA: 10:03 - loss: 17.6715

 79/420 [====>.........................] - ETA: 10:03 - loss: 17.6608

 80/420 [====>.........................] - ETA: 10:02 - loss: 17.6699

 81/420 [====>.........................] - ETA: 10:01 - loss: 17.6269

 82/420 [====>.........................] - ETA: 10:01 - loss: 17.6590

 83/420 [====>.........................] - ETA: 10:00 - loss: 17.6194

 84/420 [=====>........................] - ETA: 9:59 - loss: 17.6506 

 85/420 [=====>........................] - ETA: 9:58 - loss: 17.6208

 86/420 [=====>........................] - ETA: 9:56 - loss: 17.5950

 87/420 [=====>........................] - ETA: 9:57 - loss: 17.6787

 88/420 [=====>........................] - ETA: 9:56 - loss: 17.7093

 89/420 [=====>........................] - ETA: 9:54 - loss: 17.7106

 90/420 [=====>........................] - ETA: 9:53 - loss: 17.7178

 91/420 [=====>........................] - ETA: 9:52 - loss: 17.7279

 92/420 [=====>........................] - ETA: 9:50 - loss: 17.7064

 93/420 [=====>........................] - ETA: 9:50 - loss: 17.6867

 94/420 [=====>........................] - ETA: 9:48 - loss: 17.6610

 95/420 [=====>........................] - ETA: 9:47 - loss: 17.6718

 96/420 [=====>........................] - ETA: 9:48 - loss: 17.6703

 97/420 [=====>........................] - ETA: 9:46 - loss: 17.6712

 98/420 [======>.......................] - ETA: 9:45 - loss: 17.6460

 99/420 [======>.......................] - ETA: 9:43 - loss: 17.6237

100/420 [======>.......................] - ETA: 9:42 - loss: 17.6049

101/420 [======>.......................] - ETA: 9:41 - loss: 17.6579

102/420 [======>.......................] - ETA: 9:39 - loss: 17.6838

103/420 [======>.......................] - ETA: 9:38 - loss: 17.6556

104/420 [======>.......................] - ETA: 9:37 - loss: 17.6797

105/420 [======>.......................] - ETA: 9:36 - loss: 17.6538

106/420 [======>.......................] - ETA: 9:34 - loss: 17.6397

107/420 [======>.......................] - ETA: 9:33 - loss: 17.6479

108/420 [======>.......................] - ETA: 9:31 - loss: 17.6621

109/420 [======>.......................] - ETA: 9:30 - loss: 17.7415

110/420 [======>.......................] - ETA: 9:28 - loss: 17.7278

111/420 [======>.......................] - ETA: 9:27 - loss: 17.7178

112/420 [=======>......................] - ETA: 9:25 - loss: 17.7059

113/420 [=======>......................] - ETA: 9:24 - loss: 17.7182

114/420 [=======>......................] - ETA: 9:22 - loss: 17.7445

115/420 [=======>......................] - ETA: 9:21 - loss: 17.7450

116/420 [=======>......................] - ETA: 9:19 - loss: 17.7762

117/420 [=======>......................] - ETA: 9:18 - loss: 17.7474

118/420 [=======>......................] - ETA: 9:16 - loss: 17.7299

119/420 [=======>......................] - ETA: 9:14 - loss: 17.7126

120/420 [=======>......................] - ETA: 9:13 - loss: 17.6995

121/420 [=======>......................] - ETA: 9:11 - loss: 17.7060

122/420 [=======>......................] - ETA: 9:09 - loss: 17.6817

123/420 [=======>......................] - ETA: 9:08 - loss: 17.6922

124/420 [=======>......................] - ETA: 9:06 - loss: 17.7381

125/420 [=======>......................] - ETA: 9:05 - loss: 17.7329

126/420 [========>.....................] - ETA: 9:04 - loss: 17.7082

127/420 [========>.....................] - ETA: 9:03 - loss: 17.6901

128/420 [========>.....................] - ETA: 9:01 - loss: 17.6922

129/420 [========>.....................] - ETA: 9:00 - loss: 17.6643

130/420 [========>.....................] - ETA: 8:58 - loss: 17.6323

131/420 [========>.....................] - ETA: 8:57 - loss: 17.6211

132/420 [========>.....................] - ETA: 8:55 - loss: 17.6196

133/420 [========>.....................] - ETA: 8:53 - loss: 17.6023

134/420 [========>.....................] - ETA: 8:52 - loss: 17.5932

135/420 [========>.....................] - ETA: 8:50 - loss: 17.6231

136/420 [========>.....................] - ETA: 8:49 - loss: 17.6245

137/420 [========>.....................] - ETA: 8:47 - loss: 17.6068

138/420 [========>.....................] - ETA: 8:46 - loss: 17.6057

139/420 [========>.....................] - ETA: 8:44 - loss: 17.5904

140/420 [=========>....................] - ETA: 8:42 - loss: 17.6503

141/420 [=========>....................] - ETA: 8:41 - loss: 17.6363

142/420 [=========>....................] - ETA: 8:39 - loss: 17.6104

143/420 [=========>....................] - ETA: 8:37 - loss: 17.6305

144/420 [=========>....................] - ETA: 8:36 - loss: 17.6245

145/420 [=========>....................] - ETA: 8:34 - loss: 17.6543

146/420 [=========>....................] - ETA: 8:32 - loss: 17.6592

147/420 [=========>....................] - ETA: 8:31 - loss: 17.6719

148/420 [=========>....................] - ETA: 8:29 - loss: 17.6835

149/420 [=========>....................] - ETA: 8:27 - loss: 17.6740

150/420 [=========>....................] - ETA: 8:26 - loss: 17.6728

151/420 [=========>....................] - ETA: 8:24 - loss: 17.6697

152/420 [=========>....................] - ETA: 8:22 - loss: 17.6640

153/420 [=========>....................] - ETA: 8:20 - loss: 17.6718

154/420 [==========>...................] - ETA: 8:19 - loss: 17.6634

155/420 [==========>...................] - ETA: 8:17 - loss: 17.6540

156/420 [==========>...................] - ETA: 8:15 - loss: 17.6298

157/420 [==========>...................] - ETA: 8:14 - loss: 17.6507

158/420 [==========>...................] - ETA: 8:12 - loss: 17.6619

159/420 [==========>...................] - ETA: 8:10 - loss: 17.6702

160/420 [==========>...................] - ETA: 8:09 - loss: 17.6659

161/420 [==========>...................] - ETA: 8:07 - loss: 17.6564

162/420 [==========>...................] - ETA: 8:05 - loss: 17.6355

163/420 [==========>...................] - ETA: 8:03 - loss: 17.6250

164/420 [==========>...................] - ETA: 8:02 - loss: 17.6081

165/420 [==========>...................] - ETA: 8:00 - loss: 17.6149

166/420 [==========>...................] - ETA: 7:58 - loss: 17.6159

167/420 [==========>...................] - ETA: 7:57 - loss: 17.6334

168/420 [===========>..................] - ETA: 7:55 - loss: 17.6539

169/420 [===========>..................] - ETA: 7:53 - loss: 17.6519

170/420 [===========>..................] - ETA: 7:52 - loss: 17.6496

171/420 [===========>..................] - ETA: 7:50 - loss: 17.6317

172/420 [===========>..................] - ETA: 7:48 - loss: 17.6333

173/420 [===========>..................] - ETA: 7:46 - loss: 17.6160

174/420 [===========>..................] - ETA: 7:44 - loss: 17.5985

175/420 [===========>..................] - ETA: 7:43 - loss: 17.5950

176/420 [===========>..................] - ETA: 7:41 - loss: 17.5858

177/420 [===========>..................] - ETA: 7:39 - loss: 17.5825

178/420 [===========>..................] - ETA: 7:37 - loss: 17.5756

179/420 [===========>..................] - ETA: 7:36 - loss: 17.5573

180/420 [===========>..................] - ETA: 7:34 - loss: 17.5459

181/420 [===========>..................] - ETA: 7:32 - loss: 17.5623

182/420 [============>.................] - ETA: 7:31 - loss: 17.5613

183/420 [============>.................] - ETA: 7:29 - loss: 17.5565

184/420 [============>.................] - ETA: 7:27 - loss: 17.5607

185/420 [============>.................] - ETA: 7:26 - loss: 17.5648

186/420 [============>.................] - ETA: 7:24 - loss: 17.5515

187/420 [============>.................] - ETA: 7:22 - loss: 17.5523

188/420 [============>.................] - ETA: 7:20 - loss: 17.5423

189/420 [============>.................] - ETA: 7:18 - loss: 17.5345

190/420 [============>.................] - ETA: 7:16 - loss: 17.5427

191/420 [============>.................] - ETA: 7:15 - loss: 17.5267

192/420 [============>.................] - ETA: 7:13 - loss: 17.5382

193/420 [============>.................] - ETA: 7:11 - loss: 17.5370

194/420 [============>.................] - ETA: 7:10 - loss: 17.5292

195/420 [============>.................] - ETA: 7:08 - loss: 17.5552

196/420 [=============>................] - ETA: 7:06 - loss: 17.5504

197/420 [=============>................] - ETA: 7:04 - loss: 17.5789

198/420 [=============>................] - ETA: 7:02 - loss: 17.5774

199/420 [=============>................] - ETA: 7:01 - loss: 17.5749

200/420 [=============>................] - ETA: 6:59 - loss: 17.5639

201/420 [=============>................] - ETA: 6:58 - loss: 17.5524

202/420 [=============>................] - ETA: 6:56 - loss: 17.5778

203/420 [=============>................] - ETA: 6:54 - loss: 17.5822

204/420 [=============>................] - ETA: 6:52 - loss: 17.5940

205/420 [=============>................] - ETA: 6:51 - loss: 17.5882

206/420 [=============>................] - ETA: 6:49 - loss: 17.5833

207/420 [=============>................] - ETA: 6:47 - loss: 17.6033

208/420 [=============>................] - ETA: 6:45 - loss: 17.6090

209/420 [=============>................] - ETA: 6:44 - loss: 17.6041

210/420 [==============>...............] - ETA: 6:42 - loss: 17.5981

211/420 [==============>...............] - ETA: 6:40 - loss: 17.5890

212/420 [==============>...............] - ETA: 6:38 - loss: 17.5718

213/420 [==============>...............] - ETA: 6:37 - loss: 17.5709

214/420 [==============>...............] - ETA: 6:35 - loss: 17.5791

215/420 [==============>...............] - ETA: 6:33 - loss: 17.5735

216/420 [==============>...............] - ETA: 6:31 - loss: 17.5802

217/420 [==============>...............] - ETA: 6:29 - loss: 17.5741

218/420 [==============>...............] - ETA: 6:27 - loss: 17.5936

219/420 [==============>...............] - ETA: 6:26 - loss: 17.5868

220/420 [==============>...............] - ETA: 6:24 - loss: 17.5901

221/420 [==============>...............] - ETA: 6:22 - loss: 17.6003

222/420 [==============>...............] - ETA: 6:20 - loss: 17.5995

223/420 [==============>...............] - ETA: 6:18 - loss: 17.6113

224/420 [===============>..............] - ETA: 6:16 - loss: 17.6068

225/420 [===============>..............] - ETA: 6:14 - loss: 17.6029

226/420 [===============>..............] - ETA: 6:12 - loss: 17.5965

227/420 [===============>..............] - ETA: 6:11 - loss: 17.5887

228/420 [===============>..............] - ETA: 6:09 - loss: 17.5917

229/420 [===============>..............] - ETA: 6:07 - loss: 17.5936

230/420 [===============>..............] - ETA: 6:05 - loss: 17.5993

231/420 [===============>..............] - ETA: 6:03 - loss: 17.5853

232/420 [===============>..............] - ETA: 6:01 - loss: 17.5905

233/420 [===============>..............] - ETA: 6:00 - loss: 17.5860

234/420 [===============>..............] - ETA: 5:58 - loss: 17.5906

235/420 [===============>..............] - ETA: 5:56 - loss: 17.5839

236/420 [===============>..............] - ETA: 5:54 - loss: 17.5649

237/420 [===============>..............] - ETA: 5:52 - loss: 17.5584

238/420 [================>.............] - ETA: 5:50 - loss: 17.54732020-05-13 00:14:49.552234: E tensorflow/stream_executor/cuda/cuda_event.cc:48] Error polling for event status: failed to query event: CUDA_ERROR_ECC_UNCORRECTABLE: uncorrectable ECC error encountered

2020-05-13 00:14:49.552296: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:274] Unexpected Event status: 1

[Modelarts Service Log]Training end with return code: 134

[Modelarts Service Log]Training completed.


训练GPU报错
点赞 举报
分享

分享文章到朋友圈

分享文章到微博
复制链接到剪贴板

云上玩AI

角色：成员

话题：0

发消息
发表于2020年05月20日 12:52:22
直达本楼层的链接
沙发 只看该作者

问题解决了吗
点赞 评论 引用 举报
返回列表 上一篇 下一篇

游客

您需要登录后才可以回帖 登录 | 立即注册

手机扫码分享给好友