-
账期、产品类型等查询ModelArts的消费详情在哪里查看?
-
prompt:There was a beautiful sunset on the sea of spray.
-
我的第一次使用AI作画
-
使用DPPO算法控制“倒立摆”实验目标通过本案例的学习和课后作业的练习:了解DPPO基本概念了解如何基于DPPO训练一个控制类问题了解强化学习训练推理控制类问题的整体流程你也可以将本案例相关的 ipynb 学习笔记分享到 AI Gallery Notebook 版块获得成长值,分享方法请查看此文档。案例内容介绍倒立摆(Pendulum)摆动问题是控制文献中的经典问题。在我们本节用DPPO解决的Pendulum-v0问题中,钟摆从一个随机位置开始围绕一个端点摆动,目标是把钟摆向上摆动,并且是钟摆保持直立。一个随机动作的倒立摆demo如下所示:整体流程:安装基础依赖->创建倒立摆环境->构建DPPO算法->训练->推理->可视化效果Distributed Proximal Policy Optimization (DPPO) 算法的基本结构DPPO算法是在Proximal Policy Optimization(PPO)算法基础上发展而来,相关PPO算法请看 使用PPO算法玩“超级马里奥兄弟”,我们在这一教程中有详细的介绍。DPPO借鉴A3C的并行方法,使用多个workers并行地在不同的环境中收集数据,并根据采集的数据计算梯度,将梯度发送给一个全局chief,全局chief在拿到一定数量的梯度数据之后进行网络更新,更新时workers停止采集等待等下完毕,更新完毕之后workers重新使用最新的网络采集数据。下面我们使用论文中的伪代码介绍DPPO的具体算法细节。上述算法所示为全局PPO的伪代码,其中 W 是workers的数目,D 是用于更新参数的works数量阈值,M,B是给定一批数据点的具有policy网络和critic网络更新的子迭代数,θ, Φ为policy网络,critic网络的参数。上述算法所示为workers的伪代码,其中T是在计算参数更新之前收集的每个工作节点的数据点数,K是计算K步返回和通过时间截断的反向道具的时间步数(对于RNN)。 该部分算法基于PPO,首先采集数据,根据PPO算法计算梯度并将梯度发送给全局chief,等待全局chief更新完毕参数再进行数据的采集。DPPO论文代码部分参考GitHub开源项目注意事项本案例运行环境为 TensorFlow-2.0.0,且需使用 GPU 运行,请查看《ModelAtrs JupyterLab 硬件规格使用指南》了解切换硬件规格的方法;如果您是第一次使用 JupyterLab,请查看《ModelAtrs JupyterLab使用指导》了解使用方法;如果您在使用 JupyterLab 过程中碰到报错,请参考《ModelAtrs JupyterLab常见问题解决办法》尝试解决问题。实验步骤1. 程序初始化第1步:安装基础依赖!pip install tensorflow==2.0.0 !pip install tensorflow-probability==0.7.0 !pip install tensorlayer==2.1.0 --ignore-installed !pip install h5py==2.10.0 !pip install gym第2步:导入相关的库import os import time import queue import threading import gym import matplotlib.pyplot as plt import numpy as np import tensorflow as tf import tensorflow_probability as tfp import tensorlayer as tl2. 训练参数初始化本案例设置的 训练最大局数 EP_MAX = 1000,可以达到较好的训练效果,训练耗时约20分钟。你也可以调小 EP_MAX 的值,以便快速跑通代码。RANDOMSEED = 1 # 随机数种子 EP_MAX = 100 # 训练总局数 EP_LEN = 200 # 一局最长长度 GAMMA = 0.9 # 折扣率 A_LR = 0.0001 # actor学习率 C_LR = 0.0002 # critic学习率 BATCH = 32 # batchsize大小 A_UPDATE_STEPS = 10 # actor更新步数 C_UPDATE_STEPS = 10 # critic更新步数 S_DIM, A_DIM = 3, 1 # state维度, action维度 EPS = 1e-8 # epsilon值 # PPO1 和PPO2 的参数,你可以选择用PPO1 (METHOD[0]),还是PPO2 (METHOD[1]) METHOD = [ dict(name='kl_pen', kl_target=0.01, lam=0.5), # KL penalty dict(name='clip', epsilon=0.2), # Clipped surrogate objective, find this is better ][1] # choose the method for optimization N_WORKER = 4 # 并行workers数目 MIN_BATCH_SIZE = 64 # 更新PPO的minibatch大小 UPDATE_STEP = 10 # 每隔10steps更新一次3. 创建环境本环境为gym内置的Pendulum,倒立摆倒下即失败。env_name = 'Pendulum-v0' # environment name4. 定义DPPO算法DPPO算法-PPO算法class PPO(object): ''' PPO class ''' def __init__(self): # 创建critic tfs = tl.layers.Input([None, S_DIM], tf.float32, 'state') l1 = tl.layers.Dense(100, tf.nn.relu)(tfs) v = tl.layers.Dense(1)(l1) self.critic = tl.models.Model(tfs, v) self.critic.train() # 创建actor self.actor = self._build_anet('pi', trainable=True) self.actor_old = self._build_anet('oldpi', trainable=False) self.actor_opt = tf.optimizers.Adam(A_LR) self.critic_opt = tf.optimizers.Adam(C_LR) # 更新actor def a_train(self, tfs, tfa, tfadv): ''' Update policy network :param tfs: state :param tfa: act :param tfadv: advantage :return: ''' tfs = np.array(tfs, np.float32) tfa = np.array(tfa, np.float32) tfadv = np.array(tfadv, np.float32) # td-error with tf.GradientTape() as tape: mu, sigma = self.actor(tfs) pi = tfp.distributions.Normal(mu, sigma) mu_old, sigma_old = self.actor_old(tfs) oldpi = tfp.distributions.Normal(mu_old, sigma_old) ratio = pi.prob(tfa) / (oldpi.prob(tfa) + EPS) surr = ratio * tfadv ## PPO1 if METHOD['name'] == 'kl_pen': tflam = METHOD['lam'] kl = tfp.distributions.kl_divergence(oldpi, pi) kl_mean = tf.reduce_mean(kl) aloss = -(tf.reduce_mean(surr - tflam * kl)) ## PPO2 else: aloss = -tf.reduce_mean( tf.minimum(surr, tf.clip_by_value(ratio, 1. - METHOD['epsilon'], 1. + METHOD['epsilon']) * tfadv) ) a_gard = tape.gradient(aloss, self.actor.trainable_weights) self.actor_opt.apply_gradients(zip(a_gard, self.actor.trainable_weights)) if METHOD['name'] == 'kl_pen': return kl_mean # 更新old_pi def update_old_pi(self): ''' Update old policy parameter :return: None ''' for p, oldp in zip(self.actor.trainable_weights, self.actor_old.trainable_weights): oldp.assign(p) # 更新critic def c_train(self, tfdc_r, s): ''' Update actor network :param tfdc_r: cumulative reward :param s: state :return: None ''' tfdc_r = np.array(tfdc_r, dtype=np.float32) with tf.GradientTape() as tape: advantage = tfdc_r - self.critic(s) # 计算advantage:V(s') * gamma + r - V(s) closs = tf.reduce_mean(tf.square(advantage)) grad = tape.gradient(closs, self.critic.trainable_weights) self.critic_opt.apply_gradients(zip(grad, self.critic.trainable_weights)) # 计算advantage:V(s') * gamma + r - V(s) def cal_adv(self, tfs, tfdc_r): ''' Calculate advantage :param tfs: state :param tfdc_r: cumulative reward :return: advantage ''' tfdc_r = np.array(tfdc_r, dtype=np.float32) advantage = tfdc_r - self.critic(tfs) return advantage.numpy() def update(self): ''' Update parameter with the constraint of KL divergent :return: None ''' global GLOBAL_UPDATE_COUNTER while not COORD.should_stop(): # 如果协调器没有停止 if GLOBAL_EP < EP_MAX: # EP_MAX是最大更新次数 UPDATE_EVENT.wait() # PPO进程的等待位置 self.update_old_pi() # copy pi to old pi data = [QUEUE.get() for _ in range(QUEUE.qsize())] # collect data from all workers data = np.vstack(data) s, a, r = data[:, :S_DIM].astype(np.float32), \ data[:, S_DIM: S_DIM + A_DIM].astype(np.float32), \ data[:, -1:].astype(np.float32) adv = self.cal_adv(s, r) # update actor ## PPO1 if METHOD['name'] == 'kl_pen': for _ in range(A_UPDATE_STEPS): kl = self.a_train(s, a, adv) if kl > 4 * METHOD['kl_target']: # this in in google's paper break if kl < METHOD['kl_target'] / 1.5: # adaptive lambda, this is in OpenAI's paper METHOD['lam'] /= 2 elif kl > METHOD['kl_target'] * 1.5: METHOD['lam'] *= 2 # sometimes explode, this clipping is MorvanZhou's solution METHOD['lam'] = np.clip(METHOD['lam'], 1e-4, 10) ## PPO2 else: # clipping method, find this is better (OpenAI's paper) for _ in range(A_UPDATE_STEPS): self.a_train(s, a, adv) # 更新critic for _ in range(C_UPDATE_STEPS): self.c_train(r, s) UPDATE_EVENT.clear() # updating finished GLOBAL_UPDATE_COUNTER = 0 # reset counter ROLLING_EVENT.set() # set roll-out available # 构建actor网络 def _build_anet(self, name, trainable): ''' Build policy network :param name: name :param trainable: trainable flag :return: policy network ''' tfs = tl.layers.Input([None, S_DIM], tf.float32, name + '_state') l1 = tl.layers.Dense(100, tf.nn.relu, name=name + '_l1')(tfs) a = tl.layers.Dense(A_DIM, tf.nn.tanh, name=name + '_a')(l1) mu = tl.layers.Lambda(lambda x: x * 2, name=name + '_lambda')(a) sigma = tl.layers.Dense(A_DIM, tf.nn.softplus, name=name + '_sigma')(l1) model = tl.models.Model(tfs, [mu, sigma], name) if trainable: model.train() else: model.eval() return model # 选择动作 def choose_action(self, s): ''' Choose action :param s: state :return: clipped act ''' s = s[np.newaxis, :].astype(np.float32) mu, sigma = self.actor(s) pi = tfp.distributions.Normal(mu, sigma) a = tf.squeeze(pi.sample(1), axis=0)[0] # choosing action return np.clip(a, -2, 2) # 计算V() def get_v(self, s): ''' Compute value :param s: state :return: value ''' s = s.astype(np.float32) if s.ndim < 2: s = s[np.newaxis, :] return self.critic(s)[0, 0] def save_ckpt(self): """ save trained weights :return: None """ if not os.path.exists('model_Pendulum'): os.makedirs('model_Pendulum') tl.files.save_weights_to_hdf5('model_Pendulum/dppo_actor.hdf5', self.actor) tl.files.save_weights_to_hdf5('model_Pendulum/dppo_actor_old.hdf5', self.actor_old) tl.files.save_weights_to_hdf5('model_Pendulum/dppo_critic.hdf5', self.critic) def load_ckpt(self): """ load trained weights :return: None """ tl.files.load_hdf5_to_weights_in_order('model_Pendulum/dppo_actor.hdf5', self.actor) tl.files.load_hdf5_to_weights_in_order('model_Pendulum/dppo_actor_old.hdf5', self.actor_old) tl.files.load_hdf5_to_weights_in_order('model_Pendulum/dppo_critic.hdf5', self.critic)workers构建class Worker(object): ''' Worker class for distributional running ''' def __init__(self, wid): self.wid = wid # 工号 self.env = gym.make(env_name).unwrapped # 创建环境 self.env.seed(wid * 100 + RANDOMSEED) # 设置不同的随机种子,因为不希望每个worker的都一致 self.ppo = GLOBAL_PPO # 算法 def work(self): ''' Define a worker :return: None ''' global GLOBAL_EP, GLOBAL_RUNNING_R, GLOBAL_UPDATE_COUNTER while not COORD.should_stop(): # 从COORD接受消息,看看是否应该should_stop s = self.env.reset() ep_r = 0 buffer_s, buffer_a, buffer_r = [], [], [] # 记录data t0 = time.time() for t in range(EP_LEN): # 看是否正在被更新。PPO进程正在工作,那么就在这里等待 if not ROLLING_EVENT.is_set(): # 查询进程是否被阻塞,如果在阻塞状态,就证明如果global PPO正在更新。否则就可以继续。 ROLLING_EVENT.wait() # worker进程的等待位置。wait until PPO is updated buffer_s, buffer_a, buffer_r = [], [], [] # clear history buffer, use new policy to collect data # 正常跑游戏,并搜集数据 a = self.ppo.choose_action(s) s_, r, done, _ = self.env.step(a) buffer_s.append(s) buffer_a.append(a) buffer_r.append((r + 8) / 8) # normalize reward, find to be useful s = s_ ep_r += r # GLOBAL_UPDATE_COUNTER是每个work的在游戏中进行一步,也就是产生一条数据就会+1. # 当GLOBAL_UPDATE_COUNTER大于batch(64)的时候,就可以进行更新。 GLOBAL_UPDATE_COUNTER += 1 # count to minimum batch size, no need to wait other workers if t == EP_LEN - 1 or GLOBAL_UPDATE_COUNTER >= MIN_BATCH_SIZE: # t == EP_LEN - 1 是最后一步 ## 计算每个状态对应的V(s') ## 要注意,这里的len(buffer) < GLOBAL_UPDATE_COUNTER。所以数据是每个worker各自计算的。 v_s_ = self.ppo.get_v(s_) discounted_r = [] # compute discounted reward for r in buffer_r[::-1]: v_s_ = r + GAMMA * v_s_ discounted_r.append(v_s_) discounted_r.reverse() ## 堆叠成数据,并保存到公共队列中。 bs, ba, br = np.vstack(buffer_s), np.vstack(buffer_a), np.array(discounted_r)[:, np.newaxis] buffer_s, buffer_a, buffer_r = [], [], [] QUEUE.put(np.hstack((bs, ba, br))) # put data in the queue # 如果数据足够,就开始更新 if GLOBAL_UPDATE_COUNTER >= MIN_BATCH_SIZE: ROLLING_EVENT.clear() # stop collecting data UPDATE_EVENT.set() # global PPO update if GLOBAL_EP >= EP_MAX: # stop training COORD.request_stop() # 停止更新 break # record reward changes, plot later if len(GLOBAL_RUNNING_R) == 0: GLOBAL_RUNNING_R.append(ep_r) else: GLOBAL_RUNNING_R.append(GLOBAL_RUNNING_R[-1] * 0.9 + ep_r * 0.1) GLOBAL_EP += 1 print( 'Episode: {}/{} | Worker: {} | Episode Reward: {:.4f} | Running Time: {:.4f}'.format( GLOBAL_EP, EP_MAX, self.wid, ep_r, time.time() - t0 ) )5. 模型训练np.random.seed(RANDOMSEED) tf.random.set_seed(RANDOMSEED) GLOBAL_PPO = PPO()[TL] Input state: [None, 3] [TL] Dense dense_1: 100 relu [TL] Dense dense_2: 1 No Activation [TL] Input pi_state: [None, 3] [TL] Dense pi_l1: 100 relu [TL] Dense pi_a: 1 tanh [TL] Lambda pi_lambda: func: <function PPO._build_anet.<locals>.<lambda> at 0x7fb65d633950>, len_weights: 0 [TL] Dense pi_sigma: 1 softplus [TL] Input oldpi_state: [None, 3] [TL] Dense oldpi_l1: 100 relu [TL] Dense oldpi_a: 1 tanh [TL] Lambda oldpi_lambda: func: <function PPO._build_anet.<locals>.<lambda> at 0x7fb65d633a70>, len_weights: 0 [TL] Dense oldpi_sigma: 1 softplus# 定义两组不同的事件,update 和 rolling UPDATE_EVENT, ROLLING_EVENT = threading.Event(), threading.Event() UPDATE_EVENT.clear() # not update now,相当于把标志位设置为False ROLLING_EVENT.set() # start to roll out,相当于把标志位设置为True,并通知所有处于等待阻塞状态的线程恢复运行状态。 # 创建workers workers = [Worker(wid=i) for i in range(N_WORKER)] GLOBAL_UPDATE_COUNTER, GLOBAL_EP = 0, 0 # 全局更新次数计数器,全局EP计数器 GLOBAL_RUNNING_R = [] # 记录动态的reward,看成绩 COORD = tf.train.Coordinator() # 创建tensorflow的协调器 QUEUE = queue.Queue() # workers putting data in this queue threads = [] # 为每个worker创建进程 for worker in workers: # worker threads t = threading.Thread(target=worker.work, args=()) # 创建进程 t.start() # 开始进程 threads.append(t) # 把进程放到进程列表中,方便管理 # add a PPO updating thread # 把一个全局的PPO更新放到进程列表最后。 threads.append(threading.Thread(target=GLOBAL_PPO.update, )) threads[-1].start() COORD.join(threads) # 把进程列表交给协调器管理 GLOBAL_PPO.save_ckpt() # 保存全局参数 # plot reward change and test plt.title('DPPO') plt.plot(np.arange(len(GLOBAL_RUNNING_R)), GLOBAL_RUNNING_R) plt.xlabel('Episode') plt.ylabel('Moving reward') plt.ylim(-2000, 0) plt.show()Episode: 1/100 | Worker: 1 | Episode Reward: -965.6343 | Running Time: 3.1675 Episode: 2/100 | Worker: 2 | Episode Reward: -1443.1138 | Running Time: 3.1689 Episode: 3/100 | Worker: 3 | Episode Reward: -1313.6248 | Running Time: 3.1734 Episode: 4/100 | Worker: 0 | Episode Reward: -1403.1952 | Running Time: 3.1819 Episode: 5/100 | Worker: 1 | Episode Reward: -1399.3963 | Running Time: 3.2429 Episode: 6/100 | Worker: 2 | Episode Reward: -1480.8439 | Running Time: 3.2453 Episode: 7/100 | Worker: 0 | Episode Reward: -1489.4195 | Running Time: 3.2373 Episode: 8/100 | Worker: 3 | Episode Reward: -1339.0517 | Running Time: 3.2583 Episode: 9/100 | Worker: 1 | Episode Reward: -1600.1292 | Running Time: 3.2478 Episode: 10/100 | Worker: 0 | Episode Reward: -1513.2170 | Running Time: 3.2584 Episode: 11/100 | Worker: 2 | Episode Reward: -1461.7279 | Running Time: 3.2697 Episode: 12/100 | Worker: 3 | Episode Reward: -1480.2685 | Running Time: 3.2598 Episode: 13/100 | Worker: 0 | Episode Reward: -1831.5374 | Running Time: 3.2423Episode: 14/100 | Worker: 1 | Episode Reward: -1524.8253 | Running Time: 3.2635 Episode: 15/100 | Worker: 2 | Episode Reward: -1383.4878 | Running Time: 3.2556 Episode: 16/100 | Worker: 3 | Episode Reward: -1288.9392 | Running Time: 3.2588 Episode: 17/100 | Worker: 1 | Episode Reward: -1657.2223 | Running Time: 3.2377 Episode: 18/100 | Worker: 0 | Episode Reward: -1472.2335 | Running Time: 3.2678 Episode: 19/100 | Worker: 2 | Episode Reward: -1475.5421 | Running Time: 3.2667 Episode: 20/100 | Worker: 3 | Episode Reward: -1532.7678 | Running Time: 3.2739 Episode: 21/100 | Worker: 1 | Episode Reward: -1575.5706 | Running Time: 3.2688 Episode: 22/100 | Worker: 2 | Episode Reward: -1238.4006 | Running Time: 3.2303 Episode: 23/100 | Worker: 0 | Episode Reward: -1630.9554 | Running Time: 3.2584 Episode: 24/100 | Worker: 3 | Episode Reward: -1610.7237 | Running Time: 3.2601 Episode: 25/100 | Worker: 1 | Episode Reward: -1516.5440 | Running Time: 3.2683 Episode: 26/100 | Worker: 0 | Episode Reward: -1547.6209 | Running Time: 3.2589 Episode: 27/100 | Worker: 2 | Episode Reward: -1328.2584 | Running Time: 3.2762 Episode: 28/100 | Worker: 3 | Episode Reward: -1191.0914 | Running Time: 3.2552 Episode: 29/100 | Worker: 1 | Episode Reward: -1415.3608 | Running Time: 3.2804 Episode: 30/100 | Worker: 0 | Episode Reward: -1765.8007 | Running Time: 3.2767 Episode: 31/100 | Worker: 2 | Episode Reward: -1756.5872 | Running Time: 3.3078 Episode: 32/100 | Worker: 3 | Episode Reward: -1428.0094 | Running Time: 3.2815 Episode: 33/100 | Worker: 1 | Episode Reward: -1605.7720 | Running Time: 3.3010 Episode: 34/100 | Worker: 0 | Episode Reward: -1247.7492 | Running Time: 3.3115Episode: 35/100 | Worker: 2 | Episode Reward: -1333.9553 | Running Time: 3.2759 Episode: 36/100 | Worker: 3 | Episode Reward: -1485.7453 | Running Time: 3.2749 Episode: 37/100 | Worker: 3 | Episode Reward: -1341.3090 | Running Time: 3.2323 Episode: 38/100 | Worker: 2 | Episode Reward: -1472.5245 | Running Time: 3.2595 Episode: 39/100 | Worker: 0 | Episode Reward: -1583.6614 | Running Time: 3.2721 Episode: 40/100 | Worker: 1 | Episode Reward: -1358.4421 | Running Time: 3.2925 Episode: 41/100 | Worker: 3 | Episode Reward: -1744.7500 | Running Time: 3.2391 Episode: 42/100 | Worker: 2 | Episode Reward: -1684.8821 | Running Time: 3.2527 Episode: 43/100 | Worker: 1 | Episode Reward: -1412.0231 | Running Time: 3.2400 Episode: 44/100 | Worker: 0 | Episode Reward: -1437.6130 | Running Time: 3.2458 Episode: 45/100 | Worker: 3 | Episode Reward: -1461.7901 | Running Time: 3.2872 Episode: 46/100 | Worker: 2 | Episode Reward: -1572.6255 | Running Time: 3.2710 Episode: 47/100 | Worker: 0 | Episode Reward: -1704.6351 | Running Time: 3.2762 Episode: 48/100 | Worker: 1 | Episode Reward: -1538.4030 | Running Time: 3.3117 Episode: 49/100 | Worker: 3 | Episode Reward: -1554.7941 | Running Time: 3.2881 Episode: 50/100 | Worker: 2 | Episode Reward: -1796.0786 | Running Time: 3.2718 Episode: 51/100 | Worker: 0 | Episode Reward: -1877.3152 | Running Time: 3.2804 Episode: 52/100 | Worker: 1 | Episode Reward: -1749.8780 | Running Time: 3.2779 Episode: 53/100 | Worker: 3 | Episode Reward: -1486.8338 | Running Time: 3.1559 Episode: 54/100 | Worker: 2 | Episode Reward: -1540.8134 | Running Time: 3.2903 Episode: 55/100 | Worker: 0 | Episode Reward: -1596.7365 | Running Time: 3.3156 Episode: 56/100 | Worker: 1 | Episode Reward: -1644.7888 | Running Time: 3.3065 Episode: 57/100 | Worker: 3 | Episode Reward: -1514.0685 | Running Time: 3.2920 Episode: 58/100 | Worker: 2 | Episode Reward: -1411.2714 | Running Time: 3.1554 Episode: 59/100 | Worker: 0 | Episode Reward: -1602.3725 | Running Time: 3.2737 Episode: 60/100 | Worker: 1 | Episode Reward: -1579.8769 | Running Time: 3.3140 Episode: 61/100 | Worker: 3 | Episode Reward: -1360.7916 | Running Time: 3.2856 Episode: 62/100 | Worker: 2 | Episode Reward: -1490.7107 | Running Time: 3.2861 Episode: 63/100 | Worker: 0 | Episode Reward: -1775.7557 | Running Time: 3.2644 Episode: 64/100 | Worker: 1 | Episode Reward: -1491.0894 | Running Time: 3.2828 Episode: 65/100 | Worker: 0 | Episode Reward: -1428.8124 | Running Time: 3.1239 Episode: 66/100 | Worker: 2 | Episode Reward: -1493.7703 | Running Time: 3.2680 Episode: 67/100 | Worker: 3 | Episode Reward: -1658.3558 | Running Time: 3.2853 Episode: 68/100 | Worker: 1 | Episode Reward: -1605.9077 | Running Time: 3.2911 Episode: 69/100 | Worker: 2 | Episode Reward: -1374.3309 | Running Time: 3.3644 Episode: 70/100 | Worker: 0 | Episode Reward: -1283.5023 | Running Time: 3.3819 Episode: 71/100 | Worker: 3 | Episode Reward: -1346.1850 | Running Time: 3.3860 Episode: 72/100 | Worker: 1 | Episode Reward: -1222.1988 | Running Time: 3.3724 Episode: 73/100 | Worker: 2 | Episode Reward: -1199.1266 | Running Time: 3.2739 Episode: 74/100 | Worker: 0 | Episode Reward: -1207.3161 | Running Time: 3.2670 Episode: 75/100 | Worker: 3 | Episode Reward: -1302.0207 | Running Time: 3.2562 Episode: 76/100 | Worker: 1 | Episode Reward: -1233.3584 | Running Time: 3.2892 Episode: 77/100 | Worker: 2 | Episode Reward: -964.8099 | Running Time: 3.2339 Episode: 78/100 | Worker: 0 | Episode Reward: -1208.2836 | Running Time: 3.2602 Episode: 79/100 | Worker: 3 | Episode Reward: -1149.2154 | Running Time: 3.2579 Episode: 80/100 | Worker: 1 | Episode Reward: -1219.3229 | Running Time: 3.2321 Episode: 81/100 | Worker: 2 | Episode Reward: -1097.7572 | Running Time: 3.2995 Episode: 82/100 | Worker: 3 | Episode Reward: -940.7949 | Running Time: 3.2981 Episode: 83/100 | Worker: 0 | Episode Reward: -1395.6272 | Running Time: 3.3076 Episode: 84/100 | Worker: 1 | Episode Reward: -1092.5180 | Running Time: 3.2936 Episode: 85/100 | Worker: 2 | Episode Reward: -1369.8868 | Running Time: 3.2517 Episode: 86/100 | Worker: 0 | Episode Reward: -1380.5247 | Running Time: 3.2390 Episode: 87/100 | Worker: 3 | Episode Reward: -1413.2114 | Running Time: 3.2740 Episode: 88/100 | Worker: 1 | Episode Reward: -1403.9904 | Running Time: 3.2643 Episode: 89/100 | Worker: 2 | Episode Reward: -1098.8470 | Running Time: 3.3078 Episode: 90/100 | Worker: 0 | Episode Reward: -983.4387 | Running Time: 3.3224 Episode: 91/100 | Worker: 3 | Episode Reward: -1056.6701 | Running Time: 3.3059 Episode: 92/100 | Worker: 1 | Episode Reward: -1357.6828 | Running Time: 3.2980 Episode: 93/100 | Worker: 2 | Episode Reward: -1082.3377 | Running Time: 3.3248 Episode: 94/100 | Worker: 3 | Episode Reward: -1052.0146 | Running Time: 3.3291 Episode: 95/100 | Worker: 0 | Episode Reward: -1373.0590 | Running Time: 3.3660 Episode: 96/100 | Worker: 1 | Episode Reward: -1044.4578 | Running Time: 3.3311 Episode: 97/100 | Worker: 2 | Episode Reward: -1179.2926 | Running Time: 3.3593 Episode: 98/100 | Worker: 3 | Episode Reward: -1039.1825 | Running Time: 3.3540 Episode: 99/100 | Worker: 0 | Episode Reward: -1193.3356 | Running Time: 3.3599 Episode: 100/100 | Worker: 1 | Episode Reward: -1378.5094 | Running Time: 3.2025 Episode: 101/100 | Worker: 2 | Episode Reward: -30.6317 | Running Time: 0.1128 Episode: 102/100 | Worker: 0 | Episode Reward: -141.0568 | Running Time: 0.2976 Episode: 103/100 | Worker: 3 | Episode Reward: -166.4818 | Running Time: 0.3256 Episode: 104/100 | Worker: 1 | Episode Reward: -123.2953 | Running Time: 0.2683 [TL] [*] Saving TL weights into model_Pendulum/dppo_actor.hdf5 [TL] [*] Saved [TL] [*] Saving TL weights into model_Pendulum/dppo_actor_old.hdf5 [TL] [*] Saved [TL] [*] Saving TL weights into model_Pendulum/dppo_critic.hdf5 [TL] [*] Saved6. 模型推理Notebook暂时不支持Pendulum可视化,请将下面代码下载到本地,可查看可视化效果。from matplotlib import animation GLOBAL_PPO.load_ckpt() env = gym.make(env_name) s = env.reset() def display_frames_as_gif(frames): patch = plt.imshow(frames[0]) plt.axis('off') def animate(i): patch.set_data(frames[i]) anim = animation.FuncAnimation(plt.gcf(), animate, frames=len(frames), interval=5) anim.save('./DPPO_Pendulum.gif', writer='imagemagick', fps=30) total_reward = 0 frames = [] while True: env.render() frames.append(env.render(mode='rgb_array')) s, r, done, info = env.step(GLOBAL_PPO.choose_action(s)) if done: print('It is over, the window will be closed after 1 seconds.') time.sleep(1) break env.close() print('Total Reward : %.2f' % total_reward) display_frames_as_gif(frames)7. 模型推理效果如下视频是训练1000 Episode模型的推理效果8. 作业请你调整步骤2中的训练参数,重新训练一个模型,使它在游戏中获得更好的表现。
-
使用DPPO算法控制“倒立摆”实验目标通过本案例的学习和课后作业的练习:了解DPPO基本概念了解如何基于DPPO训练一个控制类问题了解强化学习训练推理控制类问题的整体流程你也可以将本案例相关的 ipynb 学习笔记分享到 AI Gallery Notebook 版块获得成长值,分享方法请查看此文档。案例内容介绍倒立摆(Pendulum)摆动问题是控制文献中的经典问题。在我们本节用DPPO解决的Pendulum-v0问题中,钟摆从一个随机位置开始围绕一个端点摆动,目标是把钟摆向上摆动,并且是钟摆保持直立。一个随机动作的倒立摆demo如下所示:整体流程:安装基础依赖->创建倒立摆环境->构建DPPO算法->训练->推理->可视化效果Distributed Proximal Policy Optimization (DPPO) 算法的基本结构DPPO算法是在Proximal Policy Optimization(PPO)算法基础上发展而来,相关PPO算法请看 使用PPO算法玩“超级马里奥兄弟”,我们在这一教程中有详细的介绍。DPPO借鉴A3C的并行方法,使用多个workers并行地在不同的环境中收集数据,并根据采集的数据计算梯度,将梯度发送给一个全局chief,全局chief在拿到一定数量的梯度数据之后进行网络更新,更新时workers停止采集等待等下完毕,更新完毕之后workers重新使用最新的网络采集数据。下面我们使用论文中的伪代码介绍DPPO的具体算法细节。上述算法所示为全局PPO的伪代码,其中 W 是workers的数目,D 是用于更新参数的works数量阈值,M,B是给定一批数据点的具有policy网络和critic网络更新的子迭代数,θ, Φ为policy网络,critic网络的参数。上述算法所示为workers的伪代码,其中T是在计算参数更新之前收集的每个工作节点的数据点数,K是计算K步返回和通过时间截断的反向道具的时间步数(对于RNN)。 该部分算法基于PPO,首先采集数据,根据PPO算法计算梯度并将梯度发送给全局chief,等待全局chief更新完毕参数再进行数据的采集。DPPO论文代码部分参考GitHub开源项目注意事项本案例运行环境为 TensorFlow-2.0.0,且需使用 GPU 运行,请查看《ModelAtrs JupyterLab 硬件规格使用指南》了解切换硬件规格的方法;如果您是第一次使用 JupyterLab,请查看《ModelAtrs JupyterLab使用指导》了解使用方法;如果您在使用 JupyterLab 过程中碰到报错,请参考《ModelAtrs JupyterLab常见问题解决办法》尝试解决问题。实验步骤1. 程序初始化第1步:安装基础依赖!pip install tensorflow==2.0.0 !pip install tensorflow-probability==0.7.0 !pip install tensorlayer==2.1.0 --ignore-installed !pip install h5py==2.10.0 !pip install gym第2步:导入相关的库import os import time import queue import threading import gym import matplotlib.pyplot as plt import numpy as np import tensorflow as tf import tensorflow_probability as tfp import tensorlayer as tl2. 训练参数初始化本案例设置的 训练最大局数 EP_MAX = 1000,可以达到较好的训练效果,训练耗时约20分钟。你也可以调小 EP_MAX 的值,以便快速跑通代码。RANDOMSEED = 1 # 随机数种子 EP_MAX = 100 # 训练总局数 EP_LEN = 200 # 一局最长长度 GAMMA = 0.9 # 折扣率 A_LR = 0.0001 # actor学习率 C_LR = 0.0002 # critic学习率 BATCH = 32 # batchsize大小 A_UPDATE_STEPS = 10 # actor更新步数 C_UPDATE_STEPS = 10 # critic更新步数 S_DIM, A_DIM = 3, 1 # state维度, action维度 EPS = 1e-8 # epsilon值 # PPO1 和PPO2 的参数,你可以选择用PPO1 (METHOD[0]),还是PPO2 (METHOD[1]) METHOD = [ dict(name='kl_pen', kl_target=0.01, lam=0.5), # KL penalty dict(name='clip', epsilon=0.2), # Clipped surrogate objective, find this is better ][1] # choose the method for optimization N_WORKER = 4 # 并行workers数目 MIN_BATCH_SIZE = 64 # 更新PPO的minibatch大小 UPDATE_STEP = 10 # 每隔10steps更新一次 3. 创建环境本环境为gym内置的Pendulum,倒立摆倒下即失败。env_name = 'Pendulum-v0' # environment name 4. 定义DPPO算法DPPO算法-PPO算法class PPO(object): ''' PPO class ''' def __init__(self): # 创建critic tfs = tl.layers.Input([None, S_DIM], tf.float32, 'state') l1 = tl.layers.Dense(100, tf.nn.relu)(tfs) v = tl.layers.Dense(1)(l1) self.critic = tl.models.Model(tfs, v) self.critic.train() # 创建actor self.actor = self._build_anet('pi', trainable=True) self.actor_old = self._build_anet('oldpi', trainable=False) self.actor_opt = tf.optimizers.Adam(A_LR) self.critic_opt = tf.optimizers.Adam(C_LR) # 更新actor def a_train(self, tfs, tfa, tfadv): ''' Update policy network :param tfs: state :param tfa: act :param tfadv: advantage :return: ''' tfs = np.array(tfs, np.float32) tfa = np.array(tfa, np.float32) tfadv = np.array(tfadv, np.float32) # td-error with tf.GradientTape() as tape: mu, sigma = self.actor(tfs) pi = tfp.distributions.Normal(mu, sigma) mu_old, sigma_old = self.actor_old(tfs) oldpi = tfp.distributions.Normal(mu_old, sigma_old) ratio = pi.prob(tfa) / (oldpi.prob(tfa) + EPS) surr = ratio * tfadv ## PPO1 if METHOD['name'] == 'kl_pen': tflam = METHOD['lam'] kl = tfp.distributions.kl_divergence(oldpi, pi) kl_mean = tf.reduce_mean(kl) aloss = -(tf.reduce_mean(surr - tflam * kl)) ## PPO2 else: aloss = -tf.reduce_mean( tf.minimum(surr, tf.clip_by_value(ratio, 1. - METHOD['epsilon'], 1. + METHOD['epsilon']) * tfadv) ) a_gard = tape.gradient(aloss, self.actor.trainable_weights) self.actor_opt.apply_gradients(zip(a_gard, self.actor.trainable_weights)) if METHOD['name'] == 'kl_pen': return kl_mean # 更新old_pi def update_old_pi(self): ''' Update old policy parameter :return: None ''' for p, oldp in zip(self.actor.trainable_weights, self.actor_old.trainable_weights): oldp.assign(p) # 更新critic def c_train(self, tfdc_r, s): ''' Update actor network :param tfdc_r: cumulative reward :param s: state :return: None ''' tfdc_r = np.array(tfdc_r, dtype=np.float32) with tf.GradientTape() as tape: advantage = tfdc_r - self.critic(s) # 计算advantage:V(s') * gamma + r - V(s) closs = tf.reduce_mean(tf.square(advantage)) grad = tape.gradient(closs, self.critic.trainable_weights) self.critic_opt.apply_gradients(zip(grad, self.critic.trainable_weights)) # 计算advantage:V(s') * gamma + r - V(s) def cal_adv(self, tfs, tfdc_r): ''' Calculate advantage :param tfs: state :param tfdc_r: cumulative reward :return: advantage ''' tfdc_r = np.array(tfdc_r, dtype=np.float32) advantage = tfdc_r - self.critic(tfs) return advantage.numpy() def update(self): ''' Update parameter with the constraint of KL divergent :return: None ''' global GLOBAL_UPDATE_COUNTER while not COORD.should_stop(): # 如果协调器没有停止 if GLOBAL_EP < EP_MAX: # EP_MAX是最大更新次数 UPDATE_EVENT.wait() # PPO进程的等待位置 self.update_old_pi() # copy pi to old pi data = [QUEUE.get() for _ in range(QUEUE.qsize())] # collect data from all workers data = np.vstack(data) s, a, r = data[:, :S_DIM].astype(np.float32), \ data[:, S_DIM: S_DIM + A_DIM].astype(np.float32), \ data[:, -1:].astype(np.float32) adv = self.cal_adv(s, r) # update actor ## PPO1 if METHOD['name'] == 'kl_pen': for _ in range(A_UPDATE_STEPS): kl = self.a_train(s, a, adv) if kl > 4 * METHOD['kl_target']: # this in in google's paper break if kl < METHOD['kl_target'] / 1.5: # adaptive lambda, this is in OpenAI's paper METHOD['lam'] /= 2 elif kl > METHOD['kl_target'] * 1.5: METHOD['lam'] *= 2 # sometimes explode, this clipping is MorvanZhou's solution METHOD['lam'] = np.clip(METHOD['lam'], 1e-4, 10) ## PPO2 else: # clipping method, find this is better (OpenAI's paper) for _ in range(A_UPDATE_STEPS): self.a_train(s, a, adv) # 更新critic for _ in range(C_UPDATE_STEPS): self.c_train(r, s) UPDATE_EVENT.clear() # updating finished GLOBAL_UPDATE_COUNTER = 0 # reset counter ROLLING_EVENT.set() # set roll-out available # 构建actor网络 def _build_anet(self, name, trainable): ''' Build policy network :param name: name :param trainable: trainable flag :return: policy network ''' tfs = tl.layers.Input([None, S_DIM], tf.float32, name + '_state') l1 = tl.layers.Dense(100, tf.nn.relu, name=name + '_l1')(tfs) a = tl.layers.Dense(A_DIM, tf.nn.tanh, name=name + '_a')(l1) mu = tl.layers.Lambda(lambda x: x * 2, name=name + '_lambda')(a) sigma = tl.layers.Dense(A_DIM, tf.nn.softplus, name=name + '_sigma')(l1) model = tl.models.Model(tfs, [mu, sigma], name) if trainable: model.train() else: model.eval() return model # 选择动作 def choose_action(self, s): ''' Choose action :param s: state :return: clipped act ''' s = s[np.newaxis, :].astype(np.float32) mu, sigma = self.actor(s) pi = tfp.distributions.Normal(mu, sigma) a = tf.squeeze(pi.sample(1), axis=0)[0] # choosing action return np.clip(a, -2, 2) # 计算V() def get_v(self, s): ''' Compute value :param s: state :return: value ''' s = s.astype(np.float32) if s.ndim < 2: s = s[np.newaxis, :] return self.critic(s)[0, 0] def save_ckpt(self): """ save trained weights :return: None """ if not os.path.exists('model_Pendulum'): os.makedirs('model_Pendulum') tl.files.save_weights_to_hdf5('model_Pendulum/dppo_actor.hdf5', self.actor) tl.files.save_weights_to_hdf5('model_Pendulum/dppo_actor_old.hdf5', self.actor_old) tl.files.save_weights_to_hdf5('model_Pendulum/dppo_critic.hdf5', self.critic) def load_ckpt(self): """ load trained weights :return: None """ tl.files.load_hdf5_to_weights_in_order('model_Pendulum/dppo_actor.hdf5', self.actor) tl.files.load_hdf5_to_weights_in_order('model_Pendulum/dppo_actor_old.hdf5', self.actor_old) tl.files.load_hdf5_to_weights_in_order('model_Pendulum/dppo_critic.hdf5', self.critic) workers构建class Worker(object): ''' Worker class for distributional running ''' def __init__(self, wid): self.wid = wid # 工号 self.env = gym.make(env_name).unwrapped # 创建环境 self.env.seed(wid * 100 + RANDOMSEED) # 设置不同的随机种子,因为不希望每个worker的都一致 self.ppo = GLOBAL_PPO # 算法 def work(self): ''' Define a worker :return: None ''' global GLOBAL_EP, GLOBAL_RUNNING_R, GLOBAL_UPDATE_COUNTER while not COORD.should_stop(): # 从COORD接受消息,看看是否应该should_stop s = self.env.reset() ep_r = 0 buffer_s, buffer_a, buffer_r = [], [], [] # 记录data t0 = time.time() for t in range(EP_LEN): # 看是否正在被更新。PPO进程正在工作,那么就在这里等待 if not ROLLING_EVENT.is_set(): # 查询进程是否被阻塞,如果在阻塞状态,就证明如果global PPO正在更新。否则就可以继续。 ROLLING_EVENT.wait() # worker进程的等待位置。wait until PPO is updated buffer_s, buffer_a, buffer_r = [], [], [] # clear history buffer, use new policy to collect data # 正常跑游戏,并搜集数据 a = self.ppo.choose_action(s) s_, r, done, _ = self.env.step(a) buffer_s.append(s) buffer_a.append(a) buffer_r.append((r + 8) / 8) # normalize reward, find to be useful s = s_ ep_r += r # GLOBAL_UPDATE_COUNTER是每个work的在游戏中进行一步,也就是产生一条数据就会+1. # 当GLOBAL_UPDATE_COUNTER大于batch(64)的时候,就可以进行更新。 GLOBAL_UPDATE_COUNTER += 1 # count to minimum batch size, no need to wait other workers if t == EP_LEN - 1 or GLOBAL_UPDATE_COUNTER >= MIN_BATCH_SIZE: # t == EP_LEN - 1 是最后一步 ## 计算每个状态对应的V(s') ## 要注意,这里的len(buffer) < GLOBAL_UPDATE_COUNTER。所以数据是每个worker各自计算的。 v_s_ = self.ppo.get_v(s_) discounted_r = [] # compute discounted reward for r in buffer_r[::-1]: v_s_ = r + GAMMA * v_s_ discounted_r.append(v_s_) discounted_r.reverse() ## 堆叠成数据,并保存到公共队列中。 bs, ba, br = np.vstack(buffer_s), np.vstack(buffer_a), np.array(discounted_r)[:, np.newaxis] buffer_s, buffer_a, buffer_r = [], [], [] QUEUE.put(np.hstack((bs, ba, br))) # put data in the queue # 如果数据足够,就开始更新 if GLOBAL_UPDATE_COUNTER >= MIN_BATCH_SIZE: ROLLING_EVENT.clear() # stop collecting data UPDATE_EVENT.set() # global PPO update if GLOBAL_EP >= EP_MAX: # stop training COORD.request_stop() # 停止更新 break # record reward changes, plot later if len(GLOBAL_RUNNING_R) == 0: GLOBAL_RUNNING_R.append(ep_r) else: GLOBAL_RUNNING_R.append(GLOBAL_RUNNING_R[-1] * 0.9 + ep_r * 0.1) GLOBAL_EP += 1 print( 'Episode: {}/{} | Worker: {} | Episode Reward: {:.4f} | Running Time: {:.4f}'.format( GLOBAL_EP, EP_MAX, self.wid, ep_r, time.time() - t0 ) ) 5. 模型训练np.random.seed(RANDOMSEED) tf.random.set_seed(RANDOMSEED) GLOBAL_PPO = PPO() [TL] Input state: [None, 3] [TL] Dense dense_1: 100 relu [TL] Dense dense_2: 1 No Activation [TL] Input pi_state: [None, 3] [TL] Dense pi_l1: 100 relu [TL] Dense pi_a: 1 tanh [TL] Lambda pi_lambda: func: <function ppo._build_anet.<locals>.<lambda> at 0x7fb65d633950>, len_weights: 0 [TL] Dense pi_sigma: 1 softplus [TL] Input oldpi_state: [None, 3] [TL] Dense oldpi_l1: 100 relu [TL] Dense oldpi_a: 1 tanh [TL] Lambda oldpi_lambda: func: <function ppo._build_anet.<locals>.<lambda> at 0x7fb65d633a70>, len_weights: 0 [TL] Dense oldpi_sigma: 1 softplus# 定义两组不同的事件,update 和 rolling UPDATE_EVENT, ROLLING_EVENT = threading.Event(), threading.Event() UPDATE_EVENT.clear() # not update now,相当于把标志位设置为False ROLLING_EVENT.set() # start to roll out,相当于把标志位设置为True,并通知所有处于等待阻塞状态的线程恢复运行状态。 # 创建workers workers = [Worker(wid=i) for i in range(N_WORKER)] GLOBAL_UPDATE_COUNTER, GLOBAL_EP = 0, 0 # 全局更新次数计数器,全局EP计数器 GLOBAL_RUNNING_R = [] # 记录动态的reward,看成绩 COORD = tf.train.Coordinator() # 创建tensorflow的协调器 QUEUE = queue.Queue() # workers putting data in this queue threads = [] # 为每个worker创建进程 for worker in workers: # worker threads t = threading.Thread(target=worker.work, args=()) # 创建进程 t.start() # 开始进程 threads.append(t) # 把进程放到进程列表中,方便管理 # add a PPO updating thread # 把一个全局的PPO更新放到进程列表最后。 threads.append(threading.Thread(target=GLOBAL_PPO.update, )) threads[-1].start() COORD.join(threads) # 把进程列表交给协调器管理 GLOBAL_PPO.save_ckpt() # 保存全局参数 # plot reward change and test plt.title('DPPO') plt.plot(np.arange(len(GLOBAL_RUNNING_R)), GLOBAL_RUNNING_R) plt.xlabel('Episode') plt.ylabel('Moving reward') plt.ylim(-2000, 0) plt.show() Episode: 1/100 | Worker: 1 | Episode Reward: -965.6343 | Running Time: 3.1675 Episode: 2/100 | Worker: 2 | Episode Reward: -1443.1138 | Running Time: 3.1689 Episode: 3/100 | Worker: 3 | Episode Reward: -1313.6248 | Running Time: 3.1734 Episode: 4/100 | Worker: 0 | Episode Reward: -1403.1952 | Running Time: 3.1819 Episode: 5/100 | Worker: 1 | Episode Reward: -1399.3963 | Running Time: 3.2429 Episode: 6/100 | Worker: 2 | Episode Reward: -1480.8439 | Running Time: 3.2453 Episode: 7/100 | Worker: 0 | Episode Reward: -1489.4195 | Running Time: 3.2373 Episode: 8/100 | Worker: 3 | Episode Reward: -1339.0517 | Running Time: 3.2583 Episode: 9/100 | Worker: 1 | Episode Reward: -1600.1292 | Running Time: 3.2478 Episode: 10/100 | Worker: 0 | Episode Reward: -1513.2170 | Running Time: 3.2584 Episode: 11/100 | Worker: 2 | Episode Reward: -1461.7279 | Running Time: 3.2697 Episode: 12/100 | Worker: 3 | Episode Reward: -1480.2685 | Running Time: 3.2598 Episode: 13/100 | Worker: 0 | Episode Reward: -1831.5374 | Running Time: 3.2423Episode: 14/100 | Worker: 1 | Episode Reward: -1524.8253 | Running Time: 3.2635 Episode: 15/100 | Worker: 2 | Episode Reward: -1383.4878 | Running Time: 3.2556 Episode: 16/100 | Worker: 3 | Episode Reward: -1288.9392 | Running Time: 3.2588 Episode: 17/100 | Worker: 1 | Episode Reward: -1657.2223 | Running Time: 3.2377 Episode: 18/100 | Worker: 0 | Episode Reward: -1472.2335 | Running Time: 3.2678 Episode: 19/100 | Worker: 2 | Episode Reward: -1475.5421 | Running Time: 3.2667 Episode: 20/100 | Worker: 3 | Episode Reward: -1532.7678 | Running Time: 3.2739 Episode: 21/100 | Worker: 1 | Episode Reward: -1575.5706 | Running Time: 3.2688 Episode: 22/100 | Worker: 2 | Episode Reward: -1238.4006 | Running Time: 3.2303 Episode: 23/100 | Worker: 0 | Episode Reward: -1630.9554 | Running Time: 3.2584 Episode: 24/100 | Worker: 3 | Episode Reward: -1610.7237 | Running Time: 3.2601 Episode: 25/100 | Worker: 1 | Episode Reward: -1516.5440 | Running Time: 3.2683 Episode: 26/100 | Worker: 0 | Episode Reward: -1547.6209 | Running Time: 3.2589 Episode: 27/100 | Worker: 2 | Episode Reward: -1328.2584 | Running Time: 3.2762 Episode: 28/100 | Worker: 3 | Episode Reward: -1191.0914 | Running Time: 3.2552 Episode: 29/100 | Worker: 1 | Episode Reward: -1415.3608 | Running Time: 3.2804 Episode: 30/100 | Worker: 0 | Episode Reward: -1765.8007 | Running Time: 3.2767 Episode: 31/100 | Worker: 2 | Episode Reward: -1756.5872 | Running Time: 3.3078 Episode: 32/100 | Worker: 3 | Episode Reward: -1428.0094 | Running Time: 3.2815 Episode: 33/100 | Worker: 1 | Episode Reward: -1605.7720 | Running Time: 3.3010 Episode: 34/100 | Worker: 0 | Episode Reward: -1247.7492 | Running Time: 3.3115Episode: 35/100 | Worker: 2 | Episode Reward: -1333.9553 | Running Time: 3.2759 Episode: 36/100 | Worker: 3 | Episode Reward: -1485.7453 | Running Time: 3.2749 Episode: 37/100 | Worker: 3 | Episode Reward: -1341.3090 | Running Time: 3.2323 Episode: 38/100 | Worker: 2 | Episode Reward: -1472.5245 | Running Time: 3.2595 Episode: 39/100 | Worker: 0 | Episode Reward: -1583.6614 | Running Time: 3.2721 Episode: 40/100 | Worker: 1 | Episode Reward: -1358.4421 | Running Time: 3.2925 Episode: 41/100 | Worker: 3 | Episode Reward: -1744.7500 | Running Time: 3.2391 Episode: 42/100 | Worker: 2 | Episode Reward: -1684.8821 | Running Time: 3.2527 Episode: 43/100 | Worker: 1 | Episode Reward: -1412.0231 | Running Time: 3.2400 Episode: 44/100 | Worker: 0 | Episode Reward: -1437.6130 | Running Time: 3.2458 Episode: 45/100 | Worker: 3 | Episode Reward: -1461.7901 | Running Time: 3.2872 Episode: 46/100 | Worker: 2 | Episode Reward: -1572.6255 | Running Time: 3.2710 Episode: 47/100 | Worker: 0 | Episode Reward: -1704.6351 | Running Time: 3.2762 Episode: 48/100 | Worker: 1 | Episode Reward: -1538.4030 | Running Time: 3.3117 Episode: 49/100 | Worker: 3 | Episode Reward: -1554.7941 | Running Time: 3.2881 Episode: 50/100 | Worker: 2 | Episode Reward: -1796.0786 | Running Time: 3.2718 Episode: 51/100 | Worker: 0 | Episode Reward: -1877.3152 | Running Time: 3.2804 Episode: 52/100 | Worker: 1 | Episode Reward: -1749.8780 | Running Time: 3.2779 Episode: 53/100 | Worker: 3 | Episode Reward: -1486.8338 | Running Time: 3.1559 Episode: 54/100 | Worker: 2 | Episode Reward: -1540.8134 | Running Time: 3.2903 Episode: 55/100 | Worker: 0 | Episode Reward: -1596.7365 | Running Time: 3.3156 Episode: 56/100 | Worker: 1 | Episode Reward: -1644.7888 | Running Time: 3.3065 Episode: 57/100 | Worker: 3 | Episode Reward: -1514.0685 | Running Time: 3.2920 Episode: 58/100 | Worker: 2 | Episode Reward: -1411.2714 | Running Time: 3.1554 Episode: 59/100 | Worker: 0 | Episode Reward: -1602.3725 | Running Time: 3.2737 Episode: 60/100 | Worker: 1 | Episode Reward: -1579.8769 | Running Time: 3.3140 Episode: 61/100 | Worker: 3 | Episode Reward: -1360.7916 | Running Time: 3.2856 Episode: 62/100 | Worker: 2 | Episode Reward: -1490.7107 | Running Time: 3.2861 Episode: 63/100 | Worker: 0 | Episode Reward: -1775.7557 | Running Time: 3.2644 Episode: 64/100 | Worker: 1 | Episode Reward: -1491.0894 | Running Time: 3.2828 Episode: 65/100 | Worker: 0 | Episode Reward: -1428.8124 | Running Time: 3.1239 Episode: 66/100 | Worker: 2 | Episode Reward: -1493.7703 | Running Time: 3.2680 Episode: 67/100 | Worker: 3 | Episode Reward: -1658.3558 | Running Time: 3.2853 Episode: 68/100 | Worker: 1 | Episode Reward: -1605.9077 | Running Time: 3.2911 Episode: 69/100 | Worker: 2 | Episode Reward: -1374.3309 | Running Time: 3.3644 Episode: 70/100 | Worker: 0 | Episode Reward: -1283.5023 | Running Time: 3.3819 Episode: 71/100 | Worker: 3 | Episode Reward: -1346.1850 | Running Time: 3.3860 Episode: 72/100 | Worker: 1 | Episode Reward: -1222.1988 | Running Time: 3.3724 Episode: 73/100 | Worker: 2 | Episode Reward: -1199.1266 | Running Time: 3.2739 Episode: 74/100 | Worker: 0 | Episode Reward: -1207.3161 | Running Time: 3.2670 Episode: 75/100 | Worker: 3 | Episode Reward: -1302.0207 | Running Time: 3.2562 Episode: 76/100 | Worker: 1 | Episode Reward: -1233.3584 | Running Time: 3.2892 Episode: 77/100 | Worker: 2 | Episode Reward: -964.8099 | Running Time: 3.2339 Episode: 78/100 | Worker: 0 | Episode Reward: -1208.2836 | Running Time: 3.2602 Episode: 79/100 | Worker: 3 | Episode Reward: -1149.2154 | Running Time: 3.2579 Episode: 80/100 | Worker: 1 | Episode Reward: -1219.3229 | Running Time: 3.2321 Episode: 81/100 | Worker: 2 | Episode Reward: -1097.7572 | Running Time: 3.2995 Episode: 82/100 | Worker: 3 | Episode Reward: -940.7949 | Running Time: 3.2981 Episode: 83/100 | Worker: 0 | Episode Reward: -1395.6272 | Running Time: 3.3076 Episode: 84/100 | Worker: 1 | Episode Reward: -1092.5180 | Running Time: 3.2936 Episode: 85/100 | Worker: 2 | Episode Reward: -1369.8868 | Running Time: 3.2517 Episode: 86/100 | Worker: 0 | Episode Reward: -1380.5247 | Running Time: 3.2390 Episode: 87/100 | Worker: 3 | Episode Reward: -1413.2114 | Running Time: 3.2740 Episode: 88/100 | Worker: 1 | Episode Reward: -1403.9904 | Running Time: 3.2643 Episode: 89/100 | Worker: 2 | Episode Reward: -1098.8470 | Running Time: 3.3078 Episode: 90/100 | Worker: 0 | Episode Reward: -983.4387 | Running Time: 3.3224 Episode: 91/100 | Worker: 3 | Episode Reward: -1056.6701 | Running Time: 3.3059 Episode: 92/100 | Worker: 1 | Episode Reward: -1357.6828 | Running Time: 3.2980 Episode: 93/100 | Worker: 2 | Episode Reward: -1082.3377 | Running Time: 3.3248 Episode: 94/100 | Worker: 3 | Episode Reward: -1052.0146 | Running Time: 3.3291 Episode: 95/100 | Worker: 0 | Episode Reward: -1373.0590 | Running Time: 3.3660 Episode: 96/100 | Worker: 1 | Episode Reward: -1044.4578 | Running Time: 3.3311 Episode: 97/100 | Worker: 2 | Episode Reward: -1179.2926 | Running Time: 3.3593 Episode: 98/100 | Worker: 3 | Episode Reward: -1039.1825 | Running Time: 3.3540 Episode: 99/100 | Worker: 0 | Episode Reward: -1193.3356 | Running Time: 3.3599 Episode: 100/100 | Worker: 1 | Episode Reward: -1378.5094 | Running Time: 3.2025 Episode: 101/100 | Worker: 2 | Episode Reward: -30.6317 | Running Time: 0.1128 Episode: 102/100 | Worker: 0 | Episode Reward: -141.0568 | Running Time: 0.2976 Episode: 103/100 | Worker: 3 | Episode Reward: -166.4818 | Running Time: 0.3256 Episode: 104/100 | Worker: 1 | Episode Reward: -123.2953 | Running Time: 0.2683 [TL] [*] Saving TL weights into model_Pendulum/dppo_actor.hdf5 [TL] [*] Saved [TL] [*] Saving TL weights into model_Pendulum/dppo_actor_old.hdf5 [TL] [*] Saved [TL] [*] Saving TL weights into model_Pendulum/dppo_critic.hdf5 [TL] [*] Saved 6. 模型推理Notebook暂时不支持Pendulum可视化,请将下面代码下载到本地,可查看可视化效果。from matplotlib import animation GLOBAL_PPO.load_ckpt() env = gym.make(env_name) s = env.reset() def display_frames_as_gif(frames): patch = plt.imshow(frames[0]) plt.axis('off') def animate(i): patch.set_data(frames[i]) anim = animation.FuncAnimation(plt.gcf(), animate, frames=len(frames), interval=5) anim.save('./DPPO_Pendulum.gif', writer='imagemagick', fps=30) total_reward = 0 frames = [] while True: env.render() frames.append(env.render(mode='rgb_array')) s, r, done, info = env.step(GLOBAL_PPO.choose_action(s)) if done: print('It is over, the window will be closed after 1 seconds.') time.sleep(1) break env.close() print('Total Reward : %.2f' % total_reward) display_frames_as_gif(frames) 7. 模型推理效果如下视频是训练1000 Episode模型的推理效果8. 作业请你调整步骤2中的训练参数,重新训练一个模型,使它在游戏中获得更好的表现。</lambda></function></lambda></function>
-
** 文字生成图片Stable Diffusion 操作文档**打开浏览器,点击输入链接 点击Run in ModelArts,点击“确定”点击左上角“No Kernel”,选择”Pytorch-1.8”,点击“Select”等待右上角几秒倒计时为0s 。点击“[限时免费]GPU: 1*P100|CPU: 8核 64GB”,点击“切换规格”注意:需要使用至少8核 16G以上,优先推荐使用[限时免费]GPU: 1*P100|CPU: 8核 64GB的规格,如果切换失败: 如果有领取优惠券,可以使用[付费]GPU: 1*P100|CPU: 8核 64GB规格 等待倒计时为零,资源切换完成,点击“确定” 点击界面图标,确保已经切换成功。接下来就是运行代码了,点击左边“运行”。 左边出现运行的时间,显示NVIDIA-SMI 440.33.01表示我们切换GPU成功。 直接点击步骤 一 “运行”按钮注意:出现时间表示已经运行成功。 点击步骤二 箭头处“运行”。注意:出现warning可忽略即可,显示 ”Model Package already exists!”,表示代码运行成功,不用再重复加载。点击“运行”,加载模型,这一步不会显示结果。注意:等待左边出现时间,表示运行成功。13. 点击步骤三“运行”,测试是否正常出图。 替换修改提示词,点击红色框,点击选中所有英文替换英文提示词 注意:下面图片红色框位置不要动,只修改prompt描述词横线的地方,运行出的图片如果是黑色表示这张图片出现不规范不适合浏览,再运行一次或者修改描述词再运行即可。15. 填写“作品名称”和“作者名” 点击左上角图标,确保是图片的目录,点击图片“AI_paint_output.png”,点击右键“DownLoad”下载 17. 运行步骤四,点击“运行”生成多张图片,等待左边出现运行完的时间,点击“文件夹图标”显示出左边任务栏,生成的图片在左边任务栏,从img_0_0.png~ img_2_2.png共9张图片,想要下载都可以点击图片选中,右键“Download”下载
码上开花_Lancer
发表于2022-12-02 17:36:38
2022-12-02 17:36:38
最后回复
yd_222917306
2023-05-14 15:49:38
10863 9 -
The power of AI
-
前言ModelArts 是面向开发者的一站式 AI 开发平台,为机器学习与深度学习提供海量数据预处理及交互式智能标注、大规模分布式训练、自动化模型生成,及端-边-云模型按需部署能力,帮助用户快速创建和部署模型,管理全周期 AI 工作流。背景我们将学习深度学习中的OCR(Optical Character Recognition)光学字符识别技术。OCR作为计算机视觉中较早使用深度学习技术的领域,有很多优秀的模型出现,所以通过此案例我们来学习深度学习下的OCR技术。普遍的深度学习下的OCR技术将文字识别过程分为:文本区域检测以及字符识别。本案例中介绍的模型CRNN就是一种字符识别模型,它将文字图片中的文字识别出来。CRNN模型是目前较为流行的图文识别模型,可识别较长的文本序列, 它利用BLSTM和CTC部件学习字符图像中的上下文关系, 从而有效提升文本识别准确率,使得模型更加鲁棒。 CRNN是一种卷积循环神经网络结构,用于解决基于图像的序列识别问题,特别是场景文字识别问题。 文章认为文字识别是对序列的预测方法,所以采用了对序列预测的RNN网络。通过CNN将图片的特征提取出来后采用RNN对序列进行预测,最后通过一个CTC的翻译层得到最终结果。说白了就是CNN+RNN+CTC的结构。CRNN 全称为 Convolutional Recurrent Neural Network,主要用于端到端地对不定长的文本序列进行识别,不用先对单个文字进行切割,而是将文本识别转化为时序依赖的序列学习问题,就是基于图像的序列识别。基础环境准备在使用 ModelArts 进行 AI 开发前,需先完成以下基础操作哦(如有已完成部分,请忽略),主要分为4步(注册–>实名认证–>服务授权–>领代金券):使用手机号注册华为云账号:点击注册点此去完成实名认证,账号类型选"个人",个人认证类型推荐使用"扫码认证"。点此进入 ModelArts 控制台数据管理页面,上方会提示访问授权,点击【服务授权】按钮,按下图顺序操作:进入 ModelArts 控制台首页,如下图,点击页面上的"彩蛋",领取新手福利代金券!后续步骤可能会产生资源消耗费用,请务必领取。以上操作,也提供了详细的视频教程,点此查看:ModelArts环境配置基于CRNN模型进行字符序列检测本案例使用框架:TensorFlow-1.13.1本案例使用硬件规格:8 vCPU + 64 GiB + 1 x Tesla V100-PCIE-32GB1.数据和代码下载运行下面代码,进行数据和代码的下载和解压from modelarts.session import Sessionsess = Session()if sess.region_name == 'cn-north-1': bucket_path="modelarts-labs/notebook/DL_ocr_crnn_sequence_recognition/crnn.tar"elif sess.region_name == 'cn-north-4': bucket_path="modelarts-labs-bj4/notebook/DL_ocr_crnn_sequence_recognition/crnn.tar"else: print("请更换地区到北京一或北京四")sess.download_data(bucket_path=bucket_path, path="./crnn.tar")2.解压文件!tar -xf crnn.tar!pip install torch==1.3.0!pip install torchvision==0.4.1!pip install keras==2.1.6!pip install keras_applications==1.0.5!pip install opencv-python==4.1.0.25运行后如下:from tensorflow import ConfigProtofrom tensorflow import InteractiveSession config = ConfigProto()config.gpu_options.allow_growth = Truesession = InteractiveSession(config=config)3.首先引用相关的库和文件import numpy as npimport data.dataset as datasetimport keys as keysimport torchfrom keras.layers import Flatten, BatchNormalization, Permute, TimeDistributed, Dense, Bidirectional, GRUfrom keras.layers import Input, Conv2D, MaxPooling2D, ZeroPadding2D,Lambdafrom keras.models import Modelfrom keras.optimizers import SGDfrom keras import backend as K函数ctc_lambda_func 将完成对ctc损失函数的计算。def ctc_lambda_func(args): y_pred, labels, input_length, label_length = args y_pred = y_pred[:, 2:, :] return K.ctc_batch_cost(labels, y_pred, input_length, label_length)定义文字类型数characters = keys.alphabet[:]nclass=len(characters)+14.构建网络input = Input(shape=(32, None, 1), name='the_input')# CNN卷积层部分m = Conv2D(64, kernel_size=(3, 3), activation='relu', padding='same', name='conv1')(input)m = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), name='pool1')(m)m = Conv2D(128, kernel_size=(3, 3), activation='relu', padding='same', name='conv2')(m)m = MaxPooling2D(pool_size=(2, 2), strides=(2, 2), name='pool2')(m)m = Conv2D(256, kernel_size=(3, 3), activation='relu', padding='same', name='conv3')(m)m = Conv2D(256, kernel_size=(3, 3), activation='relu', padding='same', name='conv4')(m)m = ZeroPadding2D(padding=(0, 1))(m)m = MaxPooling2D(pool_size=(2, 2), strides=(2, 1), padding='valid', name='pool3')(m)m = Conv2D(512, kernel_size=(3, 3), activation='relu', padding='same', name='conv5')(m)m = BatchNormalization(axis=1)(m)m = Conv2D(512, kernel_size=(3, 3), activation='relu', padding='same', name='conv6')(m)m = BatchNormalization(axis=1)(m)m = ZeroPadding2D(padding=(0, 1))(m)m = MaxPooling2D(pool_size=(2, 2), strides=(2, 1), padding='valid', name='pool4')(m)m = Conv2D(512, kernel_size=(2, 2), activation='relu', padding='valid', name='conv7')(m)m = Permute((2, 1, 3), name='permute')(m)m = TimeDistributed(Flatten(), name='timedistrib')(m)# RNN循环层部分m = Bidirectional(GRU(256, return_sequences=True), name='blstm1')(m)m = Dense(256, name='blstm1_out', activation='linear')(m)m = Bidirectional(GRU(256, return_sequences=True), name='blstm2')(m)y_pred = Dense(nclass, name='blstm2_out', activation='softmax')(m)basemodel = Model(inputs=input, outputs=y_pred)# 转录层部分labels = Input(name='the_labels', shape=[None, ], dtype='float32')input_length = Input(name='input_length', shape=[1], dtype='int64')label_length = Input(name='label_length', shape=[1], dtype='int64')loss_out = Lambda(ctc_lambda_func, output_shape=(1,), name='ctc')([y_pred, labels, input_length, label_length])# 模型输入model = Model(inputs=[input, labels, input_length, label_length], outputs=[loss_out])# 模型优化器sgd = SGD(lr=0.0001, decay=1e-6, momentum=0.9, nesterov=True, clipnorm=5)model.compile(loss={'ctc': lambda y_true, y_pred: y_pred}, optimizer=sgd)5.model用于训练,加入了ctc损失计算部分model.summary()basemodel用于推理部分,包含了CNN+RNN层部分basemodel.summary()函数one_hot对标签进行处理def one_hot(text, length, characters=characters): label = np.zeros(length) for i, char in enumerate(text): index = characters.find(char) if index == -1: index = characters.find(u' ') label[i] = index return label加载数据trainroot = './data/'# 读取lmdbtrain_dataset = dataset.lmdbDataset(root=trainroot, target_transform=one_hot)test_dataset = dataset.lmdbDataset( root=trainroot, transform=dataset.resizeNormalize((256, 32)), target_transform=one_hot)# 加载数据train_loader = torch.utils.data.DataLoader( train_dataset, batch_size=1, shuffle=True, sampler=None, num_workers=4, collate_fn=dataset.alignCollate( imgH=32, imgW=256,))test_loader = torch.utils.data.DataLoader( test_dataset, shuffle=True, batch_size=1, num_workers=4)数据生成器def gen(loader, flag='train'): while True: i = 0 n = len(loader) for X, Y in loader: X = X.numpy() X = X.reshape((-1, 32, 256, 1)) if flag == 'test': Y = Y.numpy() Y = np.array(Y) Length = int(256 / 4) - 1 batchs = X.shape[0] if i > n - 1: i = 0 break yield [ X, Y, np.ones(batchs) * int(Length), np.ones(batchs) * int(len(Y)) ], np.ones(batchs)def gen(loader, flag='train'): while True: i = 0 n = len(loader) for X, Y in loader: X = X.numpy() X = X.reshape((-1, 32, 256, 1)) if flag == 'test': Y = Y.numpy() Y = np.array(Y) Length = int(256 / 4) - 1 batchs = X.shape[0] if i > n - 1: i = 0 break yield [ X, Y, np.ones(batchs) * int(Length), np.ones(batchs) * int(len(Y)) ], np.ones(batchs)加载预训练模型modelPath = './model_crnn.h5' model.load_weights(modelPath)model.fit_generator( gen(train_loader, flag='train'), steps_per_epoch=100, epochs=2, validation_data=gen(test_loader, flag='test'), validation_steps=10)model.save_weights('./CRNN_results.h5')6.测试部分basemodel.load_weights('./CRNN_results.h5') # 加载刚训练好的模型7.加载待识别的原图将左边notebook的图片文件路径加载进去from PIL import Imagefrom CRNN_model import decodeimg = Image.open('./img_01.png')print('待识别的原图:')img8.定义CRNN字符识别函数def crnn_ocr(img): """ CRNN字符识别函数 :param img: 需要进行字符识别的图片 :return: ocr_result: 图片的字符识别结果,数据类型为字符串 """ img = img.convert('L') # 图片灰度化 scale = img.size[1] * 1.0 / 32 # 图片尺寸调整,把图片高度调整为32 w = img.size[0] / scale w = int(w) img = img.resize((w, 32)) img = np.array(img).astype(np.float32) / 255.0 X = img.reshape((32, w, 1)) X = np.array([X]) y_pred = basemodel.predict(X) # 预测 ocr_result = decode(y_pred) # 处理预测结果 return ocr_result9.调用函数,获得字符识别结果ocr_result = crnn_ocr(img)print('字符识别结果:', ocr_result)总结CNRR容易训练,模型容易收敛,且鲁棒性较高。易于部署,模型结构简单,inference速度快且支持变长输入,适合文本长度变化较大的生产环境。它将文字图片中的文字识别出来,全过程整体体验不错,在notebook的支持下能完美运行,对新人很友好,免费体验规格,大家动手试试吧!到此,全文结束。感恩能与大家在华为云遇见!希望能与大家一起在华为云社区共同成长。
-
问题现象Notebook中,运行训练代码出现如下错误。cudaCheckError() failed : no kernel image is available for execution on the device原因分析因为编译的时候需要设置setup.py中编译的参数arch和code和电脑的显卡匹配。解决方法对于Tesla V100的显卡,GPU算力为-gencode arch=compute_70,code=[sm_70,compute_70],设置setup.py中的编译参数即可解决。
-
在线部署AI应用时,'There should be one user defined service derived from ModelService.'Exception: There should be one user defined service derived from ModelService.出现该异常。ai应用状态正常,请问如何解决?
-
9月21日,【华为开发者大赛暨HCSD校园沙龙】走进宁波大学科学技术学院。本次活动由华为技术有限公司主办,宁波大学科学技术学院、华为云开发者联盟产品部以及华为云宁波沃土工场联合承办。2022华为开发者大赛联合高校圈层活动品牌HCSD(Huawei Cloud Student Developers)推出【华为开发者大赛暨HCSD校园沙龙】活动,旨在协权威专家走进校园,与高校开发者共话行业技术、勇攀技术高峰,共建高校生态,助力院校人才培养。活动现场助力产教融合,架起高校学子和“云服务”连接桥梁活动伊始,宁波大学科学技术学院副院长朱世华在活动当天的发言中表示华为作为我国民营高科技企业,在引领产业发展和人才培养方面均提供了很好的资源与平台,希望同学们好好利用本次机会,积极参与活动。本次沙龙活动促进了校企合作实质性开展。宁波大学科学技术学院副院长 朱世华“如今数字产业在迅速发展,而构建产业生态,人才是关键;数字人才,高校学生是核心;未来,高校人才是数字产业的践行者。”华为云中国区副总裁陈玙说,华为公司将继续紧密携手宁波大学科学技术学院,从人才培养、科学研究、师资建设到产学研合作,形成体系化及常态化的校企合作,助力高校数字人才的培养。华为云中国区副总裁 陈玙深化人才培养,华为开发者大赛与校园沙龙并驾齐驱华为云赋能云生态总监程国春从产业发展与人才培养双向驱动的角度出发,分享了基于开发者旅程,高校生态助力人才培养的发展战略。华为云持续参与教育部项目,通过贡献丰富的资源、活动和赛事全面助力产教融合,旨在持续深化校企合作,打造人才培养闭环,助力数字人才输出,共建优良高校生态。他表示,华为云将持续为院校提供优质的资源,共同打造值得信赖、共创共享共赢的生态体系,共谱人才培养新乐章。华为云赋能云生态总监 程国春接着,华为大赛运营专家郑彭元详细介绍了华为开发者大赛,“开发者是数字时代勇于创新的中坚力量,每一个开发者都了不起。”本次大赛以Spark Infinity 创想无限为主题。面向开发者开放华为云能力,为广大开发者提供了海量API接口。也期待各位开发者通过大赛这个顶级竞技舞台,可以发挥想象力进行创新创造,结合实际场景,解决行业痛点,共建云生态。华为大赛运营专家 郑彭元2022年,华为云推出了高校圈层活动品牌HCSD(Huawei Cloud Student Developers),即华为云学生开发者,由校园大使为代表的校园开发者组织,旨在通过树立榜样培养更多优秀的?校开发者。本次活动现场,信息工程学院院长钟才明和华为浙江宁波云业务部总经理吕想为优秀的校园大使卢羽、杨永祺同学进行颁发证书。AI实操升级,行业专家现场支招如今,以云和AI为核心的全场景智慧时代正在加速到来,各行各业将通过新技术进行数字化转型,重构体验、优化流程、使能创新,打造面向未来的核心竞争力。那么高校开发者如何抓住时代机遇,学好AI呢?华为云EI开发者生态工程师岩风针对人工智能定位、人工智能技术发展历程、人工智能入门难点及AI工程师面临的问题和挑战、华为云一站式AI开发管理平台ModelArts、AI Gallery社区、华为云开发者认证简介和华为云开发者认证权益等方面问题,进行了深入浅出的讲解。还带领学生体验了华为云开发者认证中的热门实验,学习了解如何识别OCR服务,既提升了同学们的运用能力,也带领同学们通过实践体验开发乐趣,增强同学们考取华为云开发者认证的信心。华为云EI开发者生态工程师 岩风活动现场,同学们踊跃参与,氛围十分活跃,不少同学抓住机会与专家进行互动问答,切磋交流。转眼活动进入尾声,还为同学们开放了华为云开发者认证考试券名额,旨在通过开发者认证体系,帮助同学们学习并考取认证,真正在云上作业,懂开发会开发,进而助力数字产业人才生态发展。师生现场互动指导实操
-
9⽉14⽇,【华为开发者大赛暨HCSD校园沙龙】走进无锡南洋职业技术学院。本次活动由华为技术有限公司主办,华为云开发者联盟产品部、华为无锡软件开发云创新中心联合承办,无锡南洋职业技术学院·智能装备与信息工程学院协办。2022华为开发者大赛联合高校圈层活动品牌HCSD(Huawei Cloud Student Developers)推出【华为开发者大赛暨HCSD校园沙龙】活动,旨在协华为专家走进校园,与高校开发者共话行业技术、勇攀技术高峰,共建高校生态,助力院校人才培养。活动现场合照活动伊始,无锡市计算机学会副理事长兼秘书长致辞表示,华为在人才培养方面提供了很好的资源与平台,希望同学们好好利用本次机会,积极参与活动。本次沙龙活动促进了校企合作实质性深层次开展。无锡市计算机学会副理事长兼秘书长随后,华为大赛运营专家刘雅婕详细介绍了华为开发者大赛江苏赛区,本届大赛江苏赛区以“Spark Infinity 创想无限”为主题。此次在无锡正式启动的2022华为开发者大赛江苏赛区是中国赛区的一个重要组成部分。面向开发者开放华为云能力,为广大开发者提供了海量API接口。也期待各位开发者通过大赛这个顶级竞技舞台,可以发挥想象力进行创新创造,结合实际场景,解决行业痛点,共建云生态。参赛者可在华为云官网报名,作品征集将持续至2022年10月8日,开发者可以充分发挥想象力进行创新创造,解决实际问题。华为开发者大赛运营专家 刘雅婕如今,以云和AI为核心的全场景智慧时代正在加速到来,各行各业将通过新技术进行数字化转型,重构体验、优化流程、使能创新,打造面向未来的核心竞争力。那么高校开发者如何抓住时代机遇学好AI呢?华为云EI开发者生态工程师青姚老师人工智能定位、人工智能技术发展历程、人工智能入门难点及AI工程师面临的问题和挑战、华为云一站式AI开发管理平台ModelArts、AI Gallery社区、华为云开发者认证简介和华为云开发者认证权益等方面问题,进行了深入浅出的讲解。还带领学生体验了华为云开发者认证中的热门实验,学习了解如何使用文字识别OCR服务,既提升了同学们的运用能力,也带领同学们通过实践体验开发乐趣,增强同学们考取华为云开发者认证的信心。华为云EI开发者生态工程师 青姚活动现场,同学们踊跃参与,氛围十分活跃,不少同学抓住机会与专家进行互动问答,切磋交流。转眼活动进入尾声,还为同学们开放了华为云开发者认证考试券名额,旨在通过开发者认证体系,帮助同学们学习并考取认证,真正在云上作业,懂开发会开发,进而助力数字产业人才生态发展。活动现场学生实操
-
10⽉18⽇下午,华为云【HCSD校园沙龙】走进江西应用科技学院。本次活动由华为技术有限公司主办,华为云开发者联盟产品部、华为(江西)VR/AR软件中心、华为(江西)智能网联汽车产业创新中心承办,江西应用科技学院软件与区块链学院协办。华为云推出【HCSD校园沙龙】活动,旨在携行业专家走进校园,与高校开发者共话行业技术、勇攀技术高峰,共建高校生态,助力院校人才培养。活动伊始,江西应用科技学院软件与区块链学院院长卞秀运对举办此次活动的目的作出说明,并向在场同学介绍此次莅临活动的嘉宾和专家。随后江西应用科技学院副校长李建萍发表致辞,他表示江西应用科技学院一直“坚持产教融合,深度开展校企合作”的办学特色,与华为公司一直保持着深度合作,探索Harmony OS人才培养模式,先后多名老师参加了华为的专项学习和培训,致力于将学校建设成为应用型技术技能型人才培养、前沿技术研发、人才创新创业的新高地。同时他也相信通过本次HCSD校园沙龙活动,同学们对华为云将会有更深刻的了解,对于如何成为一名优秀的开发者有更明确的方向和目标。深化人才培养,助力产教融合华为开发者创新中心运营总监高荀则从产业发展与人才培养双向驱动的角度出发,展开题为《华为开放能力 助力新时代人才培养》的主旨演讲。在演讲中,高荀向在场学生介绍了华为开放能力,并分享华为开发者创新应用中心的运营模式等相关内容。期间,她重点提到华为云人才培养战略,将多维度互动体验式教学,助力院校培养高水平应用型学生。同时,华为还将基于对产业行业的政策、市场、技术调研和发展趋势的预判,融合华为商用解决方案产品与技术,结合高校专业原有人才培养方案与教学计划,共同制定适合高校教学,适应行业需求,具备产教融合特色的人才培养方案,教学计划与课程体系,帮助高校明确与细化专业培养、目标及学生毕业能力要求。AI实操升级,行业专家现场支招如今,以云和AI为核心的全场景智慧时代正在加速到来,各行各业将通过新技术进行数字化转型,重构体验、优化流程、使能创新,打造面向未来的核心竞争力。那么高校开发者如何抓住时代机遇,学好AI呢?华为云EI开发者生态工程师岩风针对人工智能定位、人工智能技术发展历程、人工智能入门难点及AI工程师面临的问题和挑战、华为云一站式AI开发管理平台ModelArts、AI Gallery社区、华为云开发者认证简介和华为云开发者认证权益等方面问题,进行了深入浅出的讲解。还带领学生体验了华为云开发者认证中的热门实验<图像动漫化>,学习了解如何将图像变成动漫效果,既提升了同学们的运用能力,也带领同学们通过实践体验开发乐趣,增强同学们考取华为云开发者认证的信心。活动现场,同学们也踊跃参与到讨论中,氛围十分活跃,不少同学抓住此次机会与专家进行互动问答、切磋交流。转眼活动进入尾声,华为还为江西应用科技学院同学们开放了华为云开发者认证专属考试券名额,旨在通过开发者认证体系,帮助同学们学习并考取认证,真正在云上作业,懂开发会开发,进而助力数字产业人才生态发展。一直以来,华为云创新中心都积极与高校以及产教融合机构紧密合作,持续孵化江西开发者人才生态圈。华为云创新中心也将基于华为云生态体系,面向江西各大高校构筑开发者生态,输出华为云技术能力,为校园开发者提供丰富的开发工具、搭建学习、交流、分享、实践的平台,营造本地产业生态,建立伙伴互利共赢的合作机制。面向未来,华为云将继续扎根江西省,深入不同领域业务场景,加速全栈创新,为开发者带来实际技术提供支持,与伙伴一同发展,共同加快产业数字化进程,提升核心竞争力,助力江西培育更加适宜的“数字生态”,播撒更多优良的“技术种子”,打造数字经济发展新高地。
-
实验目标通过本案例的学习和课后作业的练习:了解DQN算法的基本概念了解如何基于DQN训练一个小游戏了解强化学习训练推理游戏的整体流程你也可以将本案例相关的 ipynb 学习笔记分享到 AI Gallery Notebook 版块获得成长值,分享方法请查看此文档。案例内容介绍《2048》是一款单人在线和移动端游戏,由19岁的意大利人Gabriele Cirulli于2014年3月开发。游戏任务是在一个网格上滑动小方块来进行组合,直到形成一个带有有数字2048的方块。该游戏可以上下左右移动方块。如果两个带有相同数字的方块在移动中碰撞,则它们会合并为一个方块,且所带数字变为两者之和。每次移动时,会有一个值为2或者4的新方块出现,所出现的数字都是2的幂。当值为2048的方块出现时,游戏即胜利,该游戏因此得名。(源自维基百科) DQN是强化学习的经典算法之一,最早由DeepMind于2013年发表的论文“Playing Atari with Deep Reinforcement Learning”中提出,属于value based的model free方法,在多种游戏环境中表现稳定且良好。 在本案例中,我们将展示如何基于simple dqn算法,训练一个2048的小游戏。整体流程:安装基础依赖->创建2048环境->构建DQN算法->训练->推理->可视化效果DQN算法的基本结构Deep Q-learning(DQN)是Q-learing和神经网络的结合,利用经验回放来进行强化学习的训练,结构如下:神经网络部分神经网络用来逼近值函数,一般采用全连接层表达特征输入,采用卷积层表达图像输入。其损失函数表达为γ经验折扣率,γ取0,表示只考虑当下,γ取1,表示只考虑未来。经验回放经验回放是指:模型与环境交互得到的(s,a,r,s')会存入一个replay buffer,然后每次从中随机采样出一批样本进行学习。采用该策略的优点如下:减少样本之间的相关性,以近似符合独立同分布的假设;同时增大样本的利用率。探索策略一般采用贪婪探索策略,即agent以ε的概率进行随机探索,其他时间则采取模型计算得到的动作。DQN的整体结构可以简单的表示为:DQN论文Nature DQN论文Nature DQN在DQN的基础上,采用两个结构一样的网络,一个当前Q网络用来选择动作,更新模型参数,另一个目标Q网络用于计算目标Q值。这样可以减少目标Q值和当前的Q值相关性。2048游戏环境简介2048环境来源于GitHub开源项目,继承于gym基本环境类。玩家可以上下左右移动方块,如果方块数字相同,则合并且所带数字变成两者之和。当值为2048的方块出现时,则获得胜利。结束标志只要出现非法移动,即移动的方向的数字无法合并,则该局结束。相比较于传统的可试验多次更加严格,难度提高。奖励函数奖励值为当前方块和累加最大合成数字最大能合成4096注意事项本案例运行环境为 Pytorch-1.0.0,支持 GPU和CPU 运行,请查看《ModelAtrs JupyterLab 硬件规格使用指南》了解切换硬件规格的方法;如果您是第一次使用 JupyterLab,请查看《ModelAtrs JupyterLab使用指导》了解使用方法;如果您在使用 JupyterLab 过程中碰到报错,请参考《ModelAtrs JupyterLab常见问题解决办法》尝试解决问题。实验步骤1. 程序初始化第1步:安装基础依赖!pip install gym第2步:导入相关的库import sysimport timeimport loggingimport argparseimport itertoolsfrom six import StringIOfrom random import sample, randintimport gymfrom gym import spacesfrom gym.utils import seedingimport numpy as npimport torchimport torch.nn as nnfrom PIL import Image, ImageDraw, ImageFontfrom IPython import displayimport matplotlibimport matplotlib.pyplot as plt2. 训练参数初始化¶本案例设置的 epochs = 3000,可以达到较好的训练效果,GPU下训练耗时约10分钟。CPU下训练较慢,建议调小 epochs 的值,如50,以便快速跑通代码。parser = argparse.ArgumentParser()parser.add_argument("--learning_rate", type=float, default=0.001) # 学习率parser.add_argument("--gamma", type=float, default=0.99) # 经验折扣率parser.add_argument("--epochs", type=int, default=50) # 迭代多少局数parser.add_argument("--buffer_size", type=int, default=10000) # replaybuffer大小parser.add_argument("--batch_size", type=int, default=128) # batchsize大小parser.add_argument("--pre_train_model", type=str, default=None) # 是否加载预训练模型parser.add_argument("--use_nature_dqn", type=bool, default=True) # 是否采用nature dqnparser.add_argument("--target_update_freq", type=int, default=250) # 如果采用nature dqn,target模型更新频率parser.add_argument("--epsilon", type=float, default=0.9) # 探索epsilon取值args, _ = parser.parse_known_args()3. 创建环境2048游戏环境继承于gym.Env,主要几个部分:init函数 定义动作空间、状态空间和游戏基本设置step函数 与环境交互,获取动作并执行,返回状态、奖励、是否结束和补充信息reset函数 一局结束后,重置环境render函数 绘图,可视化环境def pairwise(iterable): "s -> (s0,s1), (s1,s2), (s2, s3), ..." a, b = itertools.tee(iterable) next(b, None) return zip(a, b)class IllegalMove(Exception): passdef stack(flat, layers=16): """Convert an [4, 4] representation into [4, 4, layers] with one layers for each value.""" # representation is what each layer represents representation = 2 ** (np.arange(layers, dtype=int) + 1) # layered is the flat board repeated layers times layered = np.repeat(flat[:, :, np.newaxis], layers, axis=-1) # Now set the values in the board to 1 or zero depending whether they match representation. # Representation is broadcast across a number of axes layered = np.where(layered == representation, 1, 0) return layeredclass Game2048Env(gym.Env): metadata = {'render.modes': ['ansi', 'human', 'rgb_array']} def __init__(self): # Definitions for game. Board must be square. self.size = 4 self.w = self.size self.h = self.size self.squares = self.size * self.size # Maintain own idea of game score, separate from rewards self.score = 0 # Members for gym implementation self.action_space = spaces.Discrete(4) # Suppose that the maximum tile is as if you have powers of 2 across the board. layers = self.squares self.observation_space = spaces.Box(0, 1, (self.w, self.h, layers), dtype=np.int) self.set_illegal_move_reward(-100) self.set_max_tile(None) # Size of square for rendering self.grid_size = 70 # Initialise seed self.seed() # Reset ready for a game self.reset() def seed(self, seed=None): self.np_random, seed = seeding.np_random(seed) return [seed] def set_illegal_move_reward(self, reward): """Define the reward/penalty for performing an illegal move. Also need to update the reward range for this.""" # Guess that the maximum reward is also 2**squares though you'll probably never get that. # (assume that illegal move reward is the lowest value that can be returned self.illegal_move_reward = reward self.reward_range = (self.illegal_move_reward, float(2 ** self.squares)) def set_max_tile(self, max_tile): """Define the maximum tile that will end the game (e.g. 2048). None means no limit. This does not affect the state returned.""" assert max_tile is None or isinstance(max_tile, int) self.max_tile = max_tile # Implement gym interface def step(self, action): """Perform one step of the game. This involves moving and adding a new tile.""" logging.debug("Action {}".format(action)) score = 0 done = None info = { 'illegal_move': False, } try: score = float(self.move(action)) self.score += score assert score <= 2 ** (self.w * self.h) self.add_tile() done = self.isend() reward = float(score) except IllegalMove: logging.debug("Illegal move") info['illegal_move'] = True done = True reward = self.illegal_move_reward # print("Am I done? {}".format(done)) info['highest'] = self.highest() # Return observation (board state), reward, done and info dict return stack(self.Matrix), reward, done, info def reset(self): self.Matrix = np.zeros((self.h, self.w), np.int) self.score = 0 logging.debug("Adding tiles") self.add_tile() self.add_tile() return stack(self.Matrix) def render(self, mode='human'): if mode == 'rgb_array': black = (0, 0, 0) grey = (200, 200, 200) white = (255, 255, 255) tile_colour_map = { 2: (255, 255, 255), 4: (255, 248, 220), 8: (255, 222, 173), 16: (244, 164, 96), 32: (205, 92, 92), 64: (240, 255, 255), 128: (240, 255, 240), 256: (193, 255, 193), 512: (154, 255, 154), 1024: (84, 139, 84), 2048: (139, 69, 19), 4096: (178, 34, 34), } grid_size = self.grid_size # Render with Pillow pil_board = Image.new("RGB", (grid_size * 4, grid_size * 4)) draw = ImageDraw.Draw(pil_board) draw.rectangle([0, 0, 4 * grid_size, 4 * grid_size], grey) fnt = ImageFont.truetype('/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf', 30) for y in range(4): for x in range(4): o = self.get(y, x) if o: draw.rectangle([x * grid_size, y * grid_size, (x + 1) * grid_size, (y + 1) * grid_size], tile_colour_map[o]) (text_x_size, text_y_size) = draw.textsize(str(o), font=fnt) draw.text((x * grid_size + (grid_size - text_x_size) // 2, y * grid_size + (grid_size - text_y_size) // 2), str(o), font=fnt, fill=black) assert text_x_size < grid_size assert text_y_size < grid_size return np.asarray(pil_board) outfile = StringIO() if mode == 'ansi' else sys.stdout s = 'Score: {}\n'.format(self.score) s += 'Highest: {}\n'.format(self.highest()) npa = np.array(self.Matrix) grid = npa.reshape((self.size, self.size)) s += "{}\n".format(grid) outfile.write(s) return outfile # Implement 2048 game def add_tile(self): """Add a tile, probably a 2 but maybe a 4""" possible_tiles = np.array([2, 4]) tile_probabilities = np.array([0.9, 0.1]) val = self.np_random.choice(possible_tiles, 1, p=tile_probabilities)[0] empties = self.empties() assert empties.shape[0] empty_idx = self.np_random.choice(empties.shape[0]) empty = empties[empty_idx] logging.debug("Adding %s at %s", val, (empty[0], empty[1])) self.set(empty[0], empty[1], val) def get(self, x, y): """Return the value of one square.""" return self.Matrix[x, y] def set(self, x, y, val): """Set the value of one square.""" self.Matrix[x, y] = val def empties(self): """Return a 2d numpy array with the location of empty squares.""" return np.argwhere(self.Matrix == 0) def highest(self): """Report the highest tile on the board.""" return np.max(self.Matrix) def move(self, direction, trial=False): """Perform one move of the game. Shift things to one side then, combine. directions 0, 1, 2, 3 are up, right, down, left. Returns the score that [would have] got.""" if not trial: if direction == 0: logging.debug("Up") elif direction == 1: logging.debug("Right") elif direction == 2: logging.debug("Down") elif direction == 3: logging.debug("Left") changed = False move_score = 0 dir_div_two = int(direction / 2) dir_mod_two = int(direction % 2) shift_direction = dir_mod_two ^ dir_div_two # 0 for towards up left, 1 for towards bottom right # Construct a range for extracting row/column into a list rx = list(range(self.w)) ry = list(range(self.h)) if dir_mod_two == 0: # Up or down, split into columns for y in range(self.h): old = [self.get(x, y) for x in rx] (new, ms) = self.shift(old, shift_direction) move_score += ms if old != new: changed = True if not trial: for x in rx: self.set(x, y, new[x]) else: # Left or right, split into rows for x in range(self.w): old = [self.get(x, y) for y in ry] (new, ms) = self.shift(old, shift_direction) move_score += ms if old != new: changed = True if not trial: for y in ry: self.set(x, y, new[y]) if changed != True: raise IllegalMove return move_score def combine(self, shifted_row): """Combine same tiles when moving to one side. This function always shifts towards the left. Also count the score of combined tiles.""" move_score = 0 combined_row = [0] * self.size skip = False output_index = 0 for p in pairwise(shifted_row): if skip: skip = False continue combined_row[output_index] = p[0] if p[0] == p[1]: combined_row[output_index] += p[1] move_score += p[0] + p[1] # Skip the next thing in the list. skip = True output_index += 1 if shifted_row and not skip: combined_row[output_index] = shifted_row[-1] return (combined_row, move_score) def shift(self, row, direction): """Shift one row left (direction == 0) or right (direction == 1), combining if required.""" length = len(row) assert length == self.size assert direction == 0 or direction == 1 # Shift all non-zero digits up shifted_row = [i for i in row if i != 0] # Reverse list to handle shifting to the right if direction: shifted_row.reverse() (combined_row, move_score) = self.combine(shifted_row) # Reverse list to handle shifting to the right if direction: combined_row.reverse() assert len(combined_row) == self.size return (combined_row, move_score) def isend(self): """Has the game ended. Game ends if there is a tile equal to the limit or there are no legal moves. If there are empty spaces then there must be legal moves.""" if self.max_tile is not None and self.highest() == self.max_tile: return True for direction in range(4): try: self.move(direction, trial=True) # Not the end if we can do any move return False except IllegalMove: pass return True def get_board(self): """Retrieve the whole board, useful for testing.""" return self.Matrix def set_board(self, new_board): """Retrieve the whole board, useful for testing.""" self.Matrix = new_board4. 定义DQN算法DQN算法分了两部分构造,拟合函数部分-神经网络算法逻辑本身Replay Buffer部分神经网络结构包含三层卷积网络和一层全连接网络,输出维度为动作空间维度。 神经网络部分可自由设计,以训练出更好的效果。class Net(nn.Module): #obs是状态空间输入,available_actions_count为动作输出维度 def __init__(self, obs, available_actions_count): super(Net, self).__init__() self.conv1 = nn.Conv2d(obs, 128, kernel_size=2, stride=1) self.conv2 = nn.Conv2d(128, 64, kernel_size=2, stride=1) self.conv3 = nn.Conv2d(64, 16, kernel_size=2, stride=1) self.fc1 = nn.Linear(16, available_actions_count) self.relu = nn.ReLU(inplace=True) def forward(self, x): x = x.permute(0, 3, 1, 2) x = self.relu(self.conv1(x)) x = self.relu(self.conv2(x)) x = self.relu(self.conv3(x)) x = self.fc1(x.view(x.shape[0], -1)) return xDQN核心逻辑部分class DQN: def __init__(self, args, obs_dim, action_dim): # 是否加载预训练模型 if args.pre_train_model: print("Loading model from: ", args.pre_train_model) self.behaviour_model = torch.load(args.pre_train_model).to(device) # 如果采用Nature DQN,则需要额外定义target_network self.target_model = torch.load(args.pre_train_model).to(device) else: self.behaviour_model = Net(obs_dim, action_dim).to(device) self.target_model = Net(obs_dim, action_dim).to(device) self.optimizer = torch.optim.Adam(self.behaviour_model.parameters(), args.learning_rate) self.criterion = nn.MSELoss() # 动作维度 self.action_dim = action_dim # 统计学习步数 self.learn_step_counter = 0 self.args = args def learn(self, buffer): # 当replaybuffer中存储的数据大于batchsize时,从中随机采样一个batch的数据学习 if buffer.size >= self.args.batch_size: # 更新target_model的参数 if self.learn_step_counter % args.target_update_freq == 0: self.target_model.load_state_dict(self.behaviour_model.state_dict()) self.learn_step_counter += 1 # 从replaybuffer中随机采样一个五元组(当前观测值,动作,下一个观测值,是否一局结束,奖励值) s1, a, s2, done, r = buffer.get_sample(self.args.batch_size) s1 = torch.FloatTensor(s1).to(device) s2 = torch.FloatTensor(s2).to(device) r = torch.FloatTensor(r).to(device) a = torch.LongTensor(a).to(device) if args.use_nature_dqn: q = self.target_model(s2).detach() else: q = self.behaviour_model(s2) # 每个动作的q值=r+gamma*(1-0或1)*q_max target_q = r + torch.FloatTensor(args.gamma * (1 - done)).to(device) * q.max(1)[0] target_q = target_q.view(args.batch_size, 1) eval_q = self.behaviour_model(s1).gather(1, torch.reshape(a, shape=(a.size()[0], -1))) # 计算损失函数 loss = self.criterion(eval_q, target_q) self.optimizer.zero_grad() loss.backward() self.optimizer.step() def get_action(self, state, explore=True): # 判断是否探索,如果探索,则采用贪婪探索策略决定行为 if explore: if np.random.uniform() >= args.epsilon: action = randint(0, self.action_dim - 1) else: # Choose the best action according to the network. q = self.behaviour_model(torch.FloatTensor(state).to(device)) m, index = torch.max(q, 1) action = index.data.cpu().numpy()[0] else: q = self.behaviour_model(torch.FloatTensor(state).to(device)) m, index = torch.max(q, 1) action = index.data.cpu().numpy()[0] return actionreplay buffer数据存储部分class ReplayBuffer: def __init__(self, buffer_size, obs_space): self.s1 = np.zeros(obs_space, dtype=np.float32) self.s2 = np.zeros(obs_space, dtype=np.float32) self.a = np.zeros(buffer_size, dtype=np.int32) self.r = np.zeros(buffer_size, dtype=np.float32) self.done = np.zeros(buffer_size, dtype=np.float32) # replaybuffer大小 self.buffer_size = buffer_size self.size = 0 self.pos = 0 # 不断将数据存储入buffer def add_transition(self, s1, action, s2, done, reward): self.s1[self.pos] = s1 self.a[self.pos] = action if not done: self.s2[self.pos] = s2 self.done[self.pos] = done self.r[self.pos] = reward self.pos = (self.pos + 1) % self.buffer_size self.size = min(self.size + 1, self.buffer_size) # 随机采样一个batchsize def get_sample(self, sample_size): i = sample(range(0, self.size), sample_size) return self.s1[i], self.a[i], self.s2[i], self.done[i], self.r[i]5. 训练模型初始化环境和算法# 初始化环境env = Game2048Env()device = torch.device("cuda" if torch.cuda.is_available() else "cpu")# 初始化dqndqn = DQN(args, obs_dim=env.observation_space.shape[2], action_dim=env.action_space.n)# 初始化replay buffermemory = ReplayBuffer(buffer_size=args.buffer_size, obs_space=(args.buffer_size, env.observation_space.shape[0], env.observation_space.shape[1], env.observation_space.shape[2]))开始训练print('\ntraining...')begin_t = time.time()max_reward = 0for i_episode in range(args.epochs): # 每局开始,重置环境 s = env.reset() # 累计奖励值 ep_r = 0 while True: # 计算动作 a = dqn.get_action(np.expand_dims(s, axis=0)) # 执行动作 s_, r, done, info = env.step(a) # 存储信息 memory.add_transition(s, a, s_, done, r) ep_r += r # 学习优化过程 dqn.learn(memory) if done: print('Ep: ', i_episode, '| Ep_r: ', round(ep_r, 2)) if ep_r > max_reward: max_reward = ep_r print("current_max_reward {}".format(max_reward)) # 保存模型 torch.save(dqn.behaviour_model, "2048.pt") break s = s_print("finish! time cost is {}s".format(time.time() - begin_t))6. 使用模型推理游戏¶#加载模型model = torch.load("2048.pt").to(device)model.eval()s = env.reset()img = plt.imshow(env.render(mode='rgb_array'))while True: plt.axis("off") img.set_data(env.render(mode='rgb_array')) display.display(plt.gcf()) display.clear_output(wait=True) s = torch.FloatTensor(np.expand_dims(s, axis=0)).to(device) a = torch.argmax(model(s), dim=1).cpu().numpy()[0] # take action s_, r, done, info = env.step(a) time.sleep(0.1) if done: break s = s_env.close()plt.close()7. 作业请你调整步骤2中的训练参数,重新训练一个模型,使它在游戏中获得更好的表现。
-
Stable Diffusion文字生成图像Stable Diffusion 是由 CompVis、Stability AI 和 LAION 共同开发的一个文本转图像模型,它通过 LAION-5B 子集大量的 512x512 图文模型进行训练,我们只要简单的输入一段文本,Stable Diffusion 就可以迅速将其转换为图像,同样我们也可以置入图片或视频,配合文本对其进行处理。这个活动简单又好玩Ta-Da!机会来啦!AI Gallery 正式上线了 AI 作画中最火的模型 “Stable Diffusion”,通过 ModelArts Notebook能力,即可实现【一键 AI 作画】,自动生成作品海报,人人都是大艺术家!https://developer.huaweicloud.com/develop/aigallery/notebook/detail?id=03aab198-dc21-4974-ab33-352e9f56939c&ticket=ST-193587-MgJGfa1fxl6GFZ77F7vFjRod-sso切换GPU免费规格Prompt框架:[图片的风格]+[内容主题]+[细节描述]+[绘画风格或者艺术家风格]图片的风格按照这个框架,简单构造一个图片风格,比如:可以是一幅美丽的画,或者是一幅美丽的山水画。内容的主体+详细描述主体可以是各种各样的选择。推荐一些可以激发大家灵感的事物,如山水,人物,建筑房屋,动物等等,尽可能细致刻画这些实体事物。如果是简单的输入“大海”两个字,模型不知道我们想要的风景是什么样子的,一段作品的相对准确的描述,比如:梦幻的大海,白沙滩岸边铺满了粉色的玫瑰花,月光轻柔的人洒在海面上,绿色发光的海浪,我们要去尽量用详细的语言,用细节清楚的描述我们幻想的场景。比如:云中的Skyrim风格的山,有一个照明螺栓击中尖端导致雪崩,山区风景。
上滑加载中
推荐直播
-
华为云码道Agent集成与鸿蒙实战2026/08/11 周二 19:00-21:00
王一男-华为云码道产品规划专家;李炎-华为云码道产品专家;彭江敏-华为云鸿蒙端云一体化开发专家
本次直播带你解读华为云码道7月份产品新特性、新功能。更有专家演示码道Agent Space × 钉钉机器集成实战,从0到1打通消息通道;码道鸿蒙端云一体化实战,快速搭建员工签到系统。
回顾中 -
华为云开发者AI素养直播课·第五期2026/09/04 周五 16:00-18:00
林华鼎-华为云AI开发者运营负责人;蒋春阳-华为云AI开发者案例开发专家
本期直播内容: AI工具体验营 · 第5-8课连讲。Agent-Team 多智能体协作完成毕业设计实践
回顾中 -
华为云开发者AI素养ClassRoom·第六期2026/09/08 周二 19:00-20:00
樊渊-2026华为软件挑战赛冠军
高手来了:看软挑高手解析二维排样问题—从工业难题到算法突破
回顾中
热门标签