• [问题求助] 使用FHD6.1 备份功能备份hive大表失败
    我在使用FDH6.5.1 的备份功能 备份 hive的一张大表(orc)时报错,备份小表可成功,使用的介质是NFS共享的磁盘,请华为的同事帮忙看看,多谢。报错如下:Error: java.io.IOException: File copy failed:截图如下:
  • [基础组件] 【MRS产品】【Hive元数据功能】如何使用JDBC的方式连接Hive元数据?
    【功能模块】【操作步骤&问题现象】1、用JAVA编写JDBC连接hive元数据,但是无法登录【截图信息】连接信息如下报错信息如下我尝试修改/srv/BigData/dbdata_service/data/pg_hba.conf,然后重启DBService之后,这个文件的变动又被改回去了【日志信息】(可选,上传日志内容或者附件)
  • [性能调优] 【FusionInsight 产品】【Hive】Hive 查询 view 性能慢,是否建议使用 Tez 引擎
    【功能模块】FusionInsight C70 集群,使用默认的 Hive 配置(MR引擎)客户有许多hive表,然后在上面创建视图,对外使用的时候通过视图来访问,这样底层改变的话,不影响上层应用。【操作步骤&问题现象】因为使用视图,所以发现一些查询条件无法被利用到做分区过滤,导致 hive 启动大量的 task (可能上万);而其实很多文件/目录下没有合适的数据,所以许多任务执行时间很短;但是启动这么多 task 的资源浪费是比较严重的,导致许多的调度开销和任务等待。受环境管控,无法提供日志或截图。【问题】FI 是否支持和建议使用 Tez 引擎?什么情况下建议开启?除了使用 Tez 引擎,是否有其它优化手段?是否 view 支持分区列声明和查询条件下压?希望能得到指导,感谢!
  • [问题求助] 【ROMA Connect】【接入MRS Hive数据源】MRS客户端配置文件找不到version文件
    【功能模块】华为云Stack8.0.3应用与数据集成平台(ROMA Connect)数据源管理-接入数据源【操作步骤&问题现象】1、数据源管理-接入数据源—按照用户指南操作下载客户端配置文件解压文件2、在MRS_Services_ClientConfig_ConfigFiles中无法找到“Version”文件【截图信息】【日志信息】(可选,上传日志内容或者附件)
  • [问题求助] 【MRS】通过jdbc读取hbase映射的hive表报错
    适配hbase过程中,通过jdbc读取hbase映射的hive表报错报错信息在附件异常信息.txt中,网上查阅资料说需要修改集群的配置但是尝试后并无效果,阻塞了hbase的适配。
  • [运维管理] Hive创建表报错
    报错语句及报错信息如图:
  • [其他问题] 【Gausssdb A产品】对于hive加密存储的数据,能否直接将加密的数据迁移至GaussDB中
    专家好:      目前hive数据表通过aes_encrypt()函数加密后进行存储。目前需要将hive中的数据迁移至GaussDB,通过测试,GaussDB中的加解密函数为gs_encrypt_aes128()和gs_decrypt_aes128(),咨询一下,对于hive加密存储的数据,能否直接将加密的数据迁移至GaussDB中,在GaussDB中通过解密函数解密后进行业务操作,请帮忙解答,谢谢!
  • [openEuler] 基于openEuler-20.03-LTS-SP1安装部署Apache Hive-3.1.1
    部署说明本文以root权限执行以下操作,如非root权限用户可加sudo后或切换root权限用户执行命令。一、基础环境属性说明OSopenEuler-20.03-LTS-SP1CPU架构x86_64/aarch64规格4C/8G/40G二、确认架构环境以X86架构为例,执行如下命令查看操作系统及系统架构:cat /etc/*-release arch三、安装Hive-3.1.1详细步骤①yum安装jdk-1.8.0,并配置环境变量yum install java-1.8.0-openjdk* vim /etc/profile export JAVA_HOME=/usr/lib/jvm/java export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib/rt.jar export PATH=$PATH:$JAVA_HOME/bin # 环境变量生效 source /etc/profile校验java是否安装完成及其版本java -version②部署Hadoop3.3.1注意hive3.x和hadoop3.x适配#下载hadoop-3.1.1源码 wget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz解压至/usr/local目录tar zxf hadoop-3.3.1.tar.gz -C /usr/local配置环境变量vim /etc/profile在文件中添加以下内容HADOOP_HOME=/usr/local/hadoop-3.3.1 PATH=$HADOOP_HOME/bin:$PATH export HADOOP_HOME PATH使环境变量立即生效source /etc/profile检查Hadoop是否安装成功hadoop version③部署mariadbyum安装mariadb,启动服务并设置开机自启动yum install mariadb* systemctl start mariadb systemctl enable mariadb查看mariadb服务状态systemctl status mariadb登陆mysql配置密码并授权mysql -uroot -p # 首次登陆无密码,直接回车登陆 set password=password('你的密码'); grant all on *.* to root@'%' identified by '你的密码'; flush privileges;配置my.cnfvim /etc/my.cnf添加以下内容[mysqld] init_connect='SET collation_connection = utf8_unicode_ci' init_connect='SET NAMES utf8' character-set-server=utf8 collation-server=utf8_unicode_ci skip-character-set-client-handshake重启mariadbsystemctl restart mariadb④安装Hive-3.1.1下载apache hive-3.1.1安装包wget https://archive.apache.org/dist/hive/hive-3.1.1/apache-hive-3.1.1-bin.tar.gz解压至/usr/local目录tar zxf apache-hive-3.1.1-bin.tar.gz -C /usr/local/配置环境变量vim /etc/profile添加以下内容# hive export HIVE_HOME=/usr/local/apache-hive-3.1.1-bin export PATH=$HIVE_HOME/bin:$PATH使环境变量立即生效source /etc/profile检查hive是否安装成功hive --version复制配置文件模板cd /usr/local/apache-hive-3.1.1-bin/conf cp hive-env.sh.template hive-env.sh vim hive-env.sh添加以下内容export JAVA_HOME=/usr/lib/jvm/java export HADOOP_HOME=/usr/local/hadoop-3.3.1 export HIVE_CONF_DIR=/usr/local/apache-hive-3.1.1-bin/conf复制hive-site.xml模板文件cp hive-default.xml.template hive-site.xml sed -i 's/for&#/for/g' hive-site.xml vim hive-site.xml修改为下列的value的值,注意格式<name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.mariadb.jdbc.Driver</value> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <name>javax.jdo.option.ConnectionPassword</name> <value>数据库密码</value> <name>hive.exec.local.scratchdir</name> <value>/tmp/hive-3.1.1</value> <name>hive.downloaded.resources.dir</name> <value>/tmp/${hive.session.id}_resources</value> <name>hive.querylog.location</name> <value>/tmp/hive-3.1.1</value>下载JDBC驱动,并放置/usr/local/apache-hive-3.1.1-bin/lib目录下cd /usr/local/apache-hive-3.1.1-bin/lib wget https://downloads.mariadb.com/Connectors/java/connector-java-2.3.0/mariadb-java-client-2.3.0.jar创建hive数据目录并配置文件夹权限/usr/local/hadoop-3.3.1/bin/hadoop fs -mkdir /tmp /usr/local/hadoop-3.3.1/bin/hadoop fs -mkdir -p /user/hive/warehouse /usr/local/hadoop-3.3.1/bin/hadoop fs -chmod g+w /tmp /usr/local/hadoop-3.3.1/bin/hadoop fs -chmod g+w /user/hive/warehouse创建Hive日志目录mkdir -p /usr/local/apache-hive-3.1.1-bin/log/ touch /usr/local/apache-hive-3.1.1-bin/log/hiveserver.log touch /usr/local/apache-hive-3.1.1-bin/log/hiveserver.err初始化Hiveschematool -dbType mysql -initSchema启动Hive-3.1.1至此,在openEuler-20.03-LTS-SP1上面部署hive-3.1.1已完成。
  • [openEuler] 基于openEuler 20.03-LTS-SP1安装部署Hive 3.0.0
    一、查看系统信息二、安装Java注意:请勿安装jdk1.7及之前的版本,否则hive启动时会报错。jdk下载链接:https://www.oracle.com/cn/java/technologies/javase/javase-jdk8-downloads.html下载jdk软件包,解压至/usr/java目录tar zxf jdk-8u301-linux-x64.tar.gz -C /usr/java/配置环境变量vim /etc/porfile文件中添加以下内容export JAVA_HOME=/usr/java/jdk1.8.0_301 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar执行以下命令使环境变量立即生效source /etc/profile检查Java是否安装成功java -version三、安装Hadoop下载Hadoopwget https://archive.apache.org/dist/hadoop/common/hadoop-3.3.1/hadoop-3.3.1.tar.gz解压至/usr/local目录,并在目录下创建软链接tar zxf hadoop-3.3.1.tar.gz -C /usr/local cd /usr/local ln -sv hadoop-3.3.1 hadoop配置环境变量vim /etc/profile在文件中添加以下内容HADOOP_HOME=/usr/local/hadoop PATH=$HADOOP_HOME/bin:$PATH export HADOOP_HOME PATH使环境变量立即生效source /etc/profile检查Hadoop是否安装成功hadoop version四、安装hive首先安装并启动mariadbyum install mariadb* systemctl start mariadb.service2、登陆mysql配置密码并授权mysql -uroot -p # 首次登陆无密码,直接回车登陆 set password=password('你的密码'); grant all on *.* to root@'%' identified by '你的密码'; flush privileges;输入quit退出配置my.cnfvim /etc/my.cnf添加以下内容[mysqld] init_connect='SET collation_connection = utf8_unicode_ci' init_connect='SET NAMES utf8' character-set-server=utf8 collation-server=utf8_unicode_ci skip-character-set-client-handshake重启mariadbsystemctl restart mariadb.service下载apache hive 3.0.0安装包wget https://archive.apache.org/dist/hive/hive-3.0.0/apache-hive-3.0.0-bin.tar.gz解压至/usr/local目录并创建软链接tar zxf apache-hive-3.0.0-bin.tar.gz -C /usr/local/ cd /usr/local ln -sv apache-hive-3.0.0-bin hive配置环境变量vim /etc/profile添加以下内容# hive export HIVE_HOME=/usr/local/hive export PATH=$HIVE_HOME/bin:$PATH使环境变量立即生效source /etc/profile检查hive是否安装成功hive --version复制配置文件模板cd /usr/local/hive/conf cp hive-env.sh.template hive-env.sh vim hive-env.sh添加以下内容export JAVA_HOME=/usr/java/jdk1.8.0_301 export HADOOP_HOME=/usr/local/hadoop export HIVE_CONF_DIR=/usr/local/hive/conf复制hive-site.xml模板文件cp hive-default.xml.template hive-site.xml sed -i 's/for&#/for/g' hive-site.xml vim hive-site.xml找到以下对应name标签,修改value的值<name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.mariadb.jdbc.Driver</value> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <name>javax.jdo.option.ConnectionPassword</name> <value>数据库密码</value> <name>hive.exec.local.scratchdir</name> <value>/tmp/hive</value> <name>hive.downloaded.resources.dir</name> <value>/tmp/${hive.session.id}_resources</value> <name>hive.querylog.location</name> <value>/tmp/hive</value>下载JDBC驱动,并放置/usr/local/hive/lib目录下cd /usr/local/hive/lib wget https://downloads.mariadb.com/Connectors/java/connector-java-2.3.0/mariadb-java-client-2.3.0.jar创建hive数据目录并配置文件夹权限/usr/local/hadoop/bin/hadoop fs -mkdir /tmp /usr/local/hadoop/bin/hadoop fs -mkdir -p /user/hive/warehouse /usr/local/hadoop/bin/hadoop fs -chmod g+w /tmp /usr/local/hadoop/bin/hadoop fs -chmod g+w /user/hive/warehouse创建Hive日志目录mkdir -p /usr/local/hive/log/ touch /usr/local/hive/log/hiveserver.log touch /usr/local/hive/log/hiveserver.err初始化Hiveschematool -dbType mysql -initSchema启动hive至此,hive3.0.0安装完成。
  • [openEuler] 基于openEuler 20.03-LTS-SP1安装部署Hive 2.3.1
    一、查看系统信息二、安装Java注意:请勿安装jdk1.7及之前的版本,否则hive启动时会报错。jdk下载链接:https://www.oracle.com/cn/java/technologies/javase/javase-jdk8-downloads.html下载jdk软件包,解压至/usr/java目录tar zxf jdk-8u301-linux-x64.tar.gz -C /usr/java/配置环境变量vim /etc/porfile文件中添加以下内容export JAVA_HOME=/usr/java/jdk1.8.0_301 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar执行以下命令使环境变量立即生效source /etc/profile检查Java是否安装成功java -version三、安装Hadoop下载Hadoopwget http://archive.apache.org/dist/hadoop/core/hadoop-2.9.2/hadoop-2.9.2.tar.gz解压至/usr/local目录,并在目录下创建软链接tar zxf hadoop-2.9.2.tar.gz -C /usr/local cd /usr/local ln -sv hadoop-2.9.2 hadoop配置环境变量vim /etc/profile在文件中添加以下内容HADOOP_HOME=/usr/local/hadoop PATH=$HADOOP_HOME/bin:$PATH export HADOOP_HOME PATH使环境变量立即生效source /etc/profile检查Hadoop是否安装成功hadoop version四、安装hive安装并启动mariadbyum install mariadb* systemctl start mariadb.service登陆mysql配置密码并授权mysql -uroot -p # 首次登陆无密码,直接回车登陆 set password=password('你的密码'); grant all on *.* to root@'%' identified by '你的密码'; flush privileges;配置my.cnfvim /etc/my.cnf添加以下内容[mysqld] init_connect='SET collation_connection = utf8_unicode_ci' init_connect='SET NAMES utf8' character-set-server=utf8 collation-server=utf8_unicode_ci skip-character-set-client-handshake重启mariadbsystemctl restart mariadb.service下载apache hive 2.3.1安装包wget https://archive.apache.org/dist/hive/hive-2.3.1/apache-hive-2.3.1-bin.tar.gz解压至/usr/local目录并创建软链接tar zxf apache-hive-2.3.1-bin.tar.gz -C /usr/local/ cd /usr/local ln -sv apache-hive-2.3.1-bin hive配置环境变量vim /etc/profile添加以下内容# hive export HIVE_HOME=/usr/local/hive export PATH=$HIVE_HOME/bin:$PATH使环境变量立即生效source /etc/profile检查hive是否安装成功hive --version复制配置文件模板cd /usr/local/hive/conf cp hive-env.sh.template hive-env.sh vim hive-env.sh添加以下内容export JAVA_HOME=/usr/java/jdk1.8.0_301 export HADOOP_HOME=/usr/local/hadoop export HIVE_CONF_DIR=/usr/local/hive/conf复制hive-site.xml模板文件cp hive-default.xml.template hive-site.xml sed -i 's/for&#/for/g' hive-site.xml vim hive-site.xml找到以下对应name标签,修改value的值<name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.mariadb.jdbc.Driver</value> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <name>javax.jdo.option.ConnectionPassword</name> <value>数据库密码</value> <name>hive.exec.local.scratchdir</name> <value>/tmp/hive</value> <name>hive.downloaded.resources.dir</name> <value>/tmp/${hive.session.id}_resources</value> <name>hive.querylog.location</name> <value>/tmp/hive</value>下载JDBC驱动,并放置/usr/local/hive/lib目录下cd /usr/local/hive/lib wget https://downloads.mariadb.com/Connectors/java/connector-java-2.3.0/mariadb-java-client-2.3.0.jar创建hive数据目录并配置文件夹权限/usr/local/hadoop/bin/hadoop fs -mkdir /tmp /usr/local/hadoop/bin/hadoop fs -mkdir -p /user/hive/warehouse /usr/local/hadoop/bin/hadoop fs -chmod g+w /tmp /usr/local/hadoop/bin/hadoop fs -chmod g+w /user/hive/warehouse创建Hive日志目录mkdir -p /usr/local/hive/log/ touch /usr/local/hive/log/hiveserver.log touch /usr/local/hive/log/hiveserver.err初始化Hiveschematool -dbType mysql -initSchema启动hive至此,hive2.3.1安装完成。
  • [openEuler] openEuler【20.03 LTS sp1】部署Apache Hive2.0.X操作指南
    一.环境信息二.安装jdkyum install java-1.8.0-openjdk*配置环境变量vim /etc/profile export JAVA_HOME=/usr/lib/jvm/java export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib/rt.jar export PATH=$PATH:$JAVA_HOME/bin环境变量生效source /etc/profile查看jdk版本java -version三.安装部署hadoop注意hive2.x和hadoop2.x适配下载wget http://archive.apache.org/dist/hadoop/core/hadoop-2.6.5/hadoop-2.6.5.tar.gz解压至/usr/local目录下tar -xzvf hadoop-2.6.5.tar.gz -C /usr/local配置环境变量vim /etc/profile export HADOOP_HOME=/usr/local/hadoop export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH环境变量生效source /etc/profile创建软连接cd /usr/local/ ln -s hadoop-2.6.5/ hadoop查看hadoop版本hadoop version四.安装部署apache hive首先安装并启动mariadbyum install mariadb* systemctl start mariadb.service查看启动状态systemctl status mariadb.service登陆mysql配置密码并授权,密码不可过于简单mysql -uroot -p # 首次登陆无密码,直接回车登陆 set password=password('qwer1234!@#$'); grant all on *.* to root@'%' identified by 'qwer1234!@#$'; flush privileges;编辑配置my.cnfvim /etc/my.cnf [mysqld] init_connect='SET collation_connection = utf8_unicode_ci' init_connect='SET NAMES utf8' character-set-server=utf8 collation-server=utf8_unicode_ci skip-character-set-client-handshake重启mariadbsystemctl restart mariadb获取hive 2.0.1版本的软件安装包wget https://archive.apache.org/dist/hive/hive-2.0.1/apache-hive-2.0.1-bin.tar.gz解压至/usr/local目录下tar -xzvf apache-hive-2.0.1-bin.tar.gz -C /usr/local/创建软连接cd /usr/local ln -s apache-hive-2.0.1-bin hive配置环境变量vim /etc/profile export HIVE_HOME=/usr/local/hive export PATH=$HIVE_HOME/bin:$PATH环境变量生效source /etc/profile查看hive版本hive --version修改配置文件cd /usr/local/hive/conf cp hive-env.sh.template hive-env.sh vim hive-env.sh export JAVA_HOME=/usr/lib/jvm/java export HADOOP_HOME=/usr/local/hadoop export HIVE_CONF_DIR=/usr/local/hive/conf复制hive-site.xml这样才会生效cp hive-default.xml.template hive-site.xml sed -i 's/for&#/for/g' hive-site.xml vim hive-site.xml添加如下内容,注意格式 <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>qwer1234!@#$</value> </property> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.mariadb.jdbc.Driver</value> </property> <property> <name>hive.metastore.schema.verification</name> <value>false</value> </property>如果后面启动有报错,需要把其他内容删除,只保留这一部分。下载JDBC驱动,并放置/usr/local/hive/lib目录下cd /usr/local/hive/lib wget https://downloads.mariadb.com/Connectors/java/connector-java-2.3.0/mariadb-java-client-2.3.0.jar创建Hive数据目录/usr/local/hadoop/bin/hadoop fs -mkdir /tmp /usr/local/hadoop/bin/hadoop fs -mkdir -p /user/hive/warehouse /usr/local/hadoop/bin/hadoop fs -chmod g+w /tmp /usr/local/hadoop/bin/hadoop fs -chmod g+w /user/hive/warehouse创建Hive日志目录mkdir -p /usr/local/hive/log/ touch /usr/local/hive/log/hiveserver.log touch /usr/local/hive/log/hiveserver.err初始化Hiveschematool -dbType mysql -initSchema启动hive至此Apache Hive2.0.X部署完成。五.Hive基础操作1.查看数据库show databases;2.创建数据库create database if not exists euler;3.创建测试表create table test( name string, friends array<string>, children map<string, int>, address struct<street:string, city:string> ) row format delimited fields terminated by ',' collection items terminated by '_' map keys terminated by ':' lines terminated by '\n';字段解释:row format delimited fields terminated by ‘,’ – 列分隔符collection items terminated by ‘_’ --MAP STRUCT 和 ARRAY 的分隔符(数据分割符号)map keys terminated by ‘:’ – MAP中的key与value的分隔符lines terminated by ‘\n’; – 行分隔符4.创建本地测试文件test.txtongsong,bingbing_lili,xiao song:18_xiaoxiao song:19,hui long guan_beijing yangyang,caicai_susu,xiao yang:18_xiaoxiao yang:19,chao yang_beijing5.导入文本数据到测试表 load data local inpath "/root/test.txt"into table test;6.访问三种集合列里的数据,以下分别是ARRAY,MAP,STRUCT的访问方式select friends[1],children['xiao song'],address.city from test where name="ongsong";
  • [赋能学习] HetuEngine学习3-创建dws数据源并和hive做跨仓融合分析
    根据上图本课程学习链路4,如何配置dws外部数据源,并通过链路1,通过客户端直接访问HetuEngine做到链路2和链路4,hive以及dws的跨仓融合分析1. 在HetuEngine Console页面增加dws数据源,并且配置参考产品文档《组件操作指南》->《HetuEngine》->《通过HetuEngine HSConsole配置数据源》-> 《配置JDBC数据源》2. 登陆dws数据库使用如下命令创建新的库以及测试表CREATE USER developuser WITH PASSWORD "Bigdata@123";GRANT ALL PRIVILEGES TO developuser;CREATE DATABASE test;CREATE TABLE test.jizhan (  rowid bigint(20) ,  province varchar(20) DEFAULT '',  jizhanid bigint(20) DEFAULT NULL,  date varchar(20) DEFAULT NULL,  PRIMARY KEY (`rowid`));insert into jizhan (rowid, province, jizhanid, date) values('12345','湖南',1,'2021-3-29 17:23:55');insert into jizhan (rowid, province, jizhanid, date) values('22345','湖北',2,'2021-3-28 16:23:55');insert into jizhan (rowid, province, jizhanid, date) values('32345','河南',1,'2021-3-27 15:23:55');insert into jizhan (rowid, province, jizhanid, date) values('42345','河北',1,'2021-3-26 14:23:55');insert into jizhan (rowid, province, jizhanid, date) values('52345','北京',1,'2021-3-25 13:23:55');2. 登陆hetu客户端hetu-cliselect des.test.test6;select hive.default.people;select *  from  hive.default.people h1 inner join dws.public.jizhan h2 on h1.rowid=h2.rowid;视频
  • [赋能学习] HetuEngine学习2-创建hive样例数据并且和hbase做跨源融合分析
    根据上图本课程学习通过链路1,使用客户端访问HetuEgine做链路2,链路3的hive, hbase数据源跨源融合分析1. 准备hive样例数据登录客户端 source /opt/client/bigdata_env创建hdfs路径 hdfs dfs -mkdir -p /tmp/people上传数据文件hdfs dfs -put people.txt /tmp/people/people.txt文件内容12345|jack|m|33|22345|gary|m|44|32345|eddi|m|55|42345|lina|f|66|52345|zhou|f|77|使用beeline登陆hive客户端创建表CREATE EXTERNAL TABLE `people`(                                                  `rowid` string,                                                             `name` string,                                                             `sex` string,                                                                  `age` int)                                                    ROW FORMAT SERDE                                                                     'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'                             WITH SERDEPROPERTIES (                                                               'field.delim'='|',                                                                 'serialization.format'='|')                                                      STORED AS INPUTFORMAT                                                                'org.apache.hadoop.mapred.TextInputFormat'                                       OUTPUTFORMAT                                                                         'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'                     LOCATION                                                                             'hdfs://hacluster/tmp/people'     TBLPROPERTIES (                                                                      'STATS_GENERATED_VIA_STATS_TASK'='workaround for potential lack of HIVE-12730',    'bucketing_version'='2',                                                           'serialization.null.format'='',                                                    'transient_lastDdlTime'='1616490878');2. 登陆hetu客户端hetu-cliselect hbase.default.test6;select hive.default.people;select *  from  hive.default.people h1 inner join hbase.default.test6 h2 on h1.rowid=h2.rowid;视频
  • [openEuler] 基于openEuler 20.03-LTS-SP1安装部署Hive 2.1.1及Hive升级至2.2.0
    介绍       hive是基于Hadoop的一个数据仓库工具,用来进行数据提取、转化、加载,这是一种可以存储、查询和分析存储在Hadoop中的大规模数据的机制。hive数据仓库工具能将结构化的数据文件映射为一张数据库表,并提供SQL查询功能,能将SQL语句转变成MapReduce任务来执行。Hive的优点是学习成本低,可以通过类似SQL语句实现快速MapReduce统计,使MapReduce变得更加简单,而不必开发专门的MapReduce应用程序。hive十分适合对数据仓库进行统计分析。安装部署说明本文档是基于openEuler20.03-LTS-SP1系统进行安装部署Hive 2.1.1,及Hive版本升级(以升级至2.2.0为例)。Hive的安装部署需要提前安装JDK、ZooKeeper以及Hadoop,为了适应兼容Hive 2.1.x以及2.2.x的安装使用,此次JDK版本选择默认的1.8.0,ZooKeeper版本选择3.7.0,Hadoop版本选择2.9.0,采用单节点进行部署测试。环境配置属性说明OSopenEuler-20.03-LTS-SP1CPU架构x86_64规格4C/8G/40G安装部署1、确认环境架构cat /etc/os-release uname -i2、安装JDKyum install java-1.8.0-openjdk*配置环境变量vim /etc/profile export JAVA_HOME=/usr/lib/jvm/java export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib/rt.jar export PATH=$PATH:$JAVA_HOME/bin # 环境变量生效 source /etc/profile查看jdk是否安装成功java -version3、安装Zookeeper下载zookeeper 3.7.0版本安装包wget https://downloads.apache.org/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz解压至/usr/local目录下tar -xzvf apache-zookeeper-3.7.0-bin.tar.gz -C /usr/local/建立链接,便于版本更换cd /usr/local/ ln -s apache-zookeeper-3.7.0-bin/ zookeeper配置环境变量vim /etc/profile export ZOOKEEPER_HOME=/usr/local/zookeeper export PATH=$ZOOKEEPER_HOME/bin:$PATH # 环境变量生效 source /etc/profile 修改zookeeper的配置文件cd /usr/local/zookeeper/conf cp zoo_sample.cfg zoo.cfg vim zoo.cfg # 修改数据存放路径 dataDir=/usr/local/zookeeper/tmp创建数据目录mkdir /usr/local/zookeeper/tmp启动zookeepercd /usr/local/zookeeper/bin ./zkServer.sh start4、安装Hadoop(原先安装hadoop 3.3.1版本发现与hive 2.1.1不兼容,故尝试安装2.9.0版本)获取hadoop软件安装包wget http://archive.apache.org/dist/hadoop/core/hadoop-2.9.0/hadoop-2.9.0.tar.gz解压至/usr/local目录下tar -xzvf hadoop-2.9.0.tar.gz -C /usr/local建立链接,方便版本升级cd /usr/local/ ln -s hadoop-2.9.0/ hadoop配置环境变量vim /etc/profile export HADOOP_HOME=/usr/local/hadoop export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH # 生效环境变量 source /etc/profile查看hadoop版本hadoop version5、安装HiveHive需将元数据保存在数据库中,则在安装Hive前,需先安装数据库软件,并在hive-site.xml中配置数据库相关信息。本次以MariaDB为例。安装并启动mariadbyum install mariadb* systemctl start mariadb.service登陆mysql配置密码并授权mysql -uroot -p # 首次登陆无密码,直接回车登陆 set password=password('xxxxxxx'); grant all on *.* to root@'%' identified by 'xxxxxxxx'; flush privileges;编辑配置my.cnfvim /etc/my.cnf [mysqld] init_connect='SET collation_connection = utf8_unicode_ci' init_connect='SET NAMES utf8' character-set-server=utf8 collation-server=utf8_unicode_ci skip-character-set-client-handshake重启mariadbsystemctl restart mariadb获取hive 2.1.1版本的软件安装包wget https://archive.apache.org/dist/hive/hive-2.1.1/apache-hive-2.1.1-bin.tar.gz解压至/usr/local目录底下tar -xzvf apache-hive-2.1.1-bin.tar.gz -C /usr/local/建立链接,方便后续版本升级cd /usr/local ln -s apache-hive-2.1.1-bin hive配置环境变量vim /etc/profile export HIVE_HOME=/usr/local/hive export PATH=$HIVE_HOME/bin:$PATH #生效环境变量 source /etc/profile查看hive版本hive --version修改配置文件cd /usr/local/hive/conf cp hive-env.sh.template hive-env.sh vim hive-env.sh export JAVA_HOME=/usr/lib/jvm/java export HADOOP_HOME=/usr/local/hadoop export HIVE_CONF_DIR=/usr/local/hive/conf cp hive-default.xml.template hive-site.xml sed -i 's/for&#/for/g' hive-site.xml vim hive-site.xml <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.mariadb.jdbc.Driver</value> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <name>javax.jdo.option.ConnectionPassword</name> <value>xxxxxxxxxxx</value> <name>hive.exec.local.scratchdir</name> <value>/tmp/hive</value> <name>hive.downloaded.resources.dir</name> <value>/tmp/${hive.session.id}_resources</value> <name>hive.querylog.location</name> <value>/tmp/hive</value>下载JDBC驱动,并放置/usr/local/hive/lib目录下cd /usr/local/hive/lib wget https://downloads.mariadb.com/Connectors/java/connector-java-2.3.0/mariadb-java-client-2.3.0.jar创建Hive数据目录/usr/local/hadoop/bin/hadoop fs -mkdir /tmp /usr/local/hadoop/bin/hadoop fs -mkdir -p /user/hive/warehouse /usr/local/hadoop/bin/hadoop fs -chmod g+w /tmp /usr/local/hadoop/bin/hadoop fs -chmod g+w /user/hive/warehouse创建Hive日志目录mkdir -p /usr/local/hive/log/ touch /usr/local/hive/log/hiveserver.log touch /usr/local/hive/log/hiveserver.err初始化Hiveschematool -dbType mysql -initSchema至此Hive 2.1.1安装部署完成。6、升级Hive至2.2.0安装部署完Hive 2.1.1后,如果后续用户需要升级Hive更高版本,可以替换/usr/local/hive的软链接,并按上述修改hive配置文件即可。环境变量等无需再修改。以Hive升级至2.2.0版本为例,已安装的jdk、zookeeper、hadoop、mariadb无需修改。下载hive 2.2.0版本软件安装包wget https://archive.apache.org/dist/hive/hive-2.2.0/apache-hive-2.2.0-bin.tar.gz解压至/usr/local/目录下tar -xzvf apache-hive-2.2.0-bin.tar.gz -C /usr/local/删除原软链接,建立新软链接cd /usr/local/ rm -rf hive ln -s apache-hive-2.2.0-bin hive然后按照hive修改配置文件步骤操作cd /usr/local/hive/conf cp hive-env.sh.template hive-env.sh vim hive-env.sh export JAVA_HOME=/usr/lib/jvm/java export HADOOP_HOME=/usr/local/hadoop export HIVE_CONF_DIR=/usr/local/hive/conf cp hive-default.xml.template hive-site.xml sed -i 's/for&#/for/g' hive-site.xml vim hive-site.xml <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true</value> <name>javax.jdo.option.ConnectionDriverName</name> <value>org.mariadb.jdbc.Driver</value> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <name>javax.jdo.option.ConnectionPassword</name> <value>xxxxxxxxxxx</value> <name>hive.exec.local.scratchdir</name> <value>/tmp/hive</value> <name>hive.downloaded.resources.dir</name> <value>/tmp/${hive.session.id}_resources</value> <name>hive.querylog.location</name> <value>/tmp/hive</value>初始化前进行数据库备份,再迁移至新库mysql dump -uroot -pxxxxxxx hive > /tmp/my_backup.sql drop database hive; create database hive; mysql -uroot -pxxxxxxxx hive < /tmp/my_backup.sql然后进行hive初始化schematool -dbType mysql -initSchema至此hive 2.2.0升级完毕
  • [运维管理] 【FI产品】Hive查询报错
    Hive执行命令报错,截图如下: