hadoop3.3源码编译集群部署

服务启动速览：

启动hadoop：
start-dfs.sh
start-yarn.sh
启动 hadoop history-server 【可选，默认没有启动】
mr-jobhistory-daemon.sh start historyserver
或 mapred --daemon start
mapred historyserver start

yarn-daemon.sh start timelineserver
代替命令：yarn --daemon start timelineserver

hadoop dfsadmin -safemode leave

若干节点上都启动 zookeeper
zkServer.sh start

启动hive  【或 hive --service hiveserver2 &】https://blog.csdn.net/leanaoo/article/details/83351240
nohup hiveserver2 > /app/logs/hiveserver2.log 2>&1 &
nohup hive --service metastore> /app/logs/metastore.log 2>&1 &

启动hbase
start-hbase.sh
启动 thrift server https://www.jianshu.com/p/93e36d008313
hbase-daemon.sh start thrift
或 hbase thrift start &
thrift默认端口是9090，启动成功后可以查看端口是否起来。
lsof -i:9090

启动livy
livy-server start

启动：
oozied.sh start
http://spark:11000

启动hue
hue runserver

vmware 虚拟机看不到共享目录，挂载共享目录：https://www.cnblogs.com/nanqiang/p/9837828.html
vmhgfs-fuse .host:/ /mnt/hgfs

///////////////////////////////////////////////////////////
VMware WorkStation Pro 15.5
Centos 7.2
jdk-8u261-linux-x64.tar.gz
scala-2.12.12.tgz
hadoop3.3.0
Spark3.0.0
apache-hive-3.1.2-src.tar.gz
Anaconda3-2019.10-Linux-x86_64.sh
///////////////////////////////////////////////////////////

////////////////////////////////////////////////////////
安装vmtools
////////////////////////////////////////////////////////
环境设置：
centos 中设置两个网卡，一个nat（自动dhcp,vmnet8），一个host only（vmnet1,固定ip地址（本例192.168.11.2），与主机通信用）

查看SELinux状态：
1、/usr/sbin/sestatus -v ##如果SELinux status参数为enabled即为开启状态
SELinux status: enabled
2、getenforce ##也可以用这个命令检查

关闭SELinux：
1、临时关闭（不用重启机器）：
setenforce 0 ##设置SELinux 成为permissive模式
##setenforce 1 设置SELinux 成为enforcing模式
2、修改配置文件需要重启机器：
vim /etc/selinux/config
将SELINUX=enforcing改为SELINUX=disabled
重启机器即可

//执行setup命令，关闭***，iptables、ip6tables服务

# 查看***状态
firewall-cmd --state

systemctl stop firewalld
systemctl disable firewalld

////////////////////////////////////////////////////////
设置yum本地源 https://www.cnblogs.com/halberd-lee/p/12793537.html
////////////////////////////////////////////////////////
mkdir /iso
mount /mnt/hgfs/os/CentOS-8.2.2004-x86_64-dvd1.iso /iso

vim /etc/yum.repos.d/CentOS-Media.repo
[InstallMedia]
name=CentOS Linux 8
baseurl=file:///iso/BaseOS
gpgcheck=0
enabled=1

[AppStream]
name=AppStream
baseurl=file:///iso/AppStream
enabled=1
gpgcheck=0

////////////////////////////////////////////////////////
更改yum源为阿里服务器
////////////////////////////////////////////////////////
mkdir /etc/yum.repos.d/bak
mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak
wget -O /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-8.repo

yum clean all
yum makecache

yum 更新
yum -y install epel-release

cp /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.bak
sed -e "s/^mirrorlist/#mirrorlist/g" -e "s/#baseurl/baseurl/g" -e "s/download\.fedoraproject\.org\/pub/mirrors.neusoft.edu.cn/g" -i /etc/yum.repos.d/epel.repo

yum install -y libzstd libzstd-devel zstd nasm yasm gcc clang icc VC make autoconf automake
yum -y install texlive-latex* texlive-fonts* libaio* svn ncurses-devel gcc* lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel
yum -y install make gcc gcc-c++ gcc-g77 flex bison file libtool libtool-libs autoconf kernel-devel libjpeg libjpeg-devel libpng libpng-devel libpng10 libpng10-devel gd gd-devel freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel glib2 glib2-devel bzip2 bzip2-devel libevent libevent-devel ncurses ncurses-devel curl curl-devel e2fsprogs e2fsprogs-devel krb5 krb5-devel libidn libidn-devel openssl openssl-devel gettext gettext-devel ncurses-devel gmp-devel pspell-devel unzip libcap lsof build-essential cmake 1g-dev pkg-config libssl-dev lzo-devel fuse fuse-devel zlib1g-dev libprotobuf-dev protobuf-compiler snappy libbz2-dev libjansson-dev libfuse-dev

yum -y install libgfortran* svn ncurses-devel gcc* lzo-devel zlib-devel autoconf automake libtool cmake openssl-devel make gcc gcc-c++ gcc-g77 flex bison file libtool libtool-libs autoconf kernel-devel libjpeg libjpeg-devel libpng libpng-devel libpng10 libpng10-devel gd gd-devel freetype freetype-devel libxml2 libxml2-devel zlib zlib-devel glib2 glib2-devel bzip2 bzip2-devel libevent libevent-devel ncurses ncurses-devel curl curl-devel e2fsprogs e2fsprogs-devel krb5 krb5-devel libidn libidn-devel openssl openssl-devel gettext gettext-devel ncurses-devel gmp-devel pspell-devel unzip libcap lsof build-essential cmake 1g-dev pkg-config libssl-dev lzo-devel fuse fuse-devel zlib1g-dev libprotobuf-dev protobuf-compiler snappy libbz2-dev libjansson-dev libfuse-dev python-devel libffi-devel libxml2 libxslt-devel libxml2-devel *ldap-dev* zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel R pandoc nc
yum -y install libpng* pspell* libssl* zlib* libprotobuf* libbz2* libjansson* libfuse*

yum install perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker asciidoc xmlto
yum -y groupinstall "Development Tools"

////////////////////////////////////////////////////////
建立hadoop用户组和用户的方式：
groupadd hadoop
useradd hadoop -g hadoop
passwd hadoop

////////////////////////////////////////////////
复制虚拟机，
更新hostname，检查主机名
vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=spark

更新hosts，
sudo vim /etc/hosts
192.168.11.2 spark

重启网络
sudo service network restart

已编译hadoop的普通用户登录设置如下:
//////////////////////////////////////////////////////
ssh免密码
ssh-keygen -t rsa -P ""
cp .ssh/id_rsa.pub .ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
合并4机器id_rsa.pub内容到同一个authorized_keys文件中，并复制到每个机器的~/.ssh/下面
///////////////////////////////////////////////////////

///////////////////////////////////////////////////////////
安装 anaconda3
///////////////////////////////////////////////////////////

在终端输入python进入Python解释器界面，输入如下内容：
import sys
print sys.maxunicode
如果结果<=65535 则 vim configure 改 have_ucs4_tcl=no 为 have_ucs4_tcl=yes

///////////////////////////////////////////////////////////
安装maven
///////////////////////////////////////////////////////////

cd /app

tar -zxvf apache-maven-3.6.3-bin.tar.gz
ln -s /app/apache-maven-3.3.9 maven

vim /etc/profile

export MAVEN_HOME=/app/maven
export PATH=$PATH:$MAVEN_HOME/bin

source /etc/profile
mvn -version

/////////////////////////////////////////////////////////
配置maven镜像：
修改maven根目录下的conf文件夹中的setting.xml文件，内容如下：
vim /app/maven/conf/settings.xml

<mirrors>
<mirror>
<id>alimaven</id>
<name>aliyun maven</name>
      <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
<mirrorOf>central</mirrorOf>
</mirror>
<mirror>
<id>mvnrepository</id>
<name>mvnrepository</name>
      <url>http://central.maven.org/maven2/</url>
<mirrorOf>central</mirrorOf>
</mirror>
<mirror>
<id>repo2</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
      <url>http://repo2.maven.org/maven2/</url>
</mirror>
<mirror>
<id>ui</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
     <url>http://uk.maven.org/maven2/</url>
</mirror>
<mirror>
<id>ibiblio</id>
<mirrorOf>central</mirrorOf>
<name>Human Readable Name for this Mirror.</name>
     <url>http://mirrors.ibiblio.org/pub/mirrors/maven2/</url>
</mirror>
<mirror>
<id>jboss-public-repository-group</id>
<mirrorOf>central</mirrorOf>
<name>JBoss Public Repository Group</name>
     <url>http://repository.jboss.org/nexus/content/groups/public</url>
</mirror>
</mirrors>

设置proxy代理，可选
<proxies>
<proxy>
<id>my-proxy</id>
<active>true</active>
<protocol>http</protocol>
<host>192.168.174.1</host>
<port>1080</port>

</proxy>
</proxies>

设置资源库位置
<localRepository>/app/maven/repository</localRepository>
之后就能享受如飞的maven下载速度。

///////////////////////////////////

///////////////////////////////////
安装ant
tar -zxvf apache-ant-1.10.8-bin.tar.gz
ln -s /app/apache-ant-1.9.7 ant

vim /etc/profile
export ANT_HOME=/app/ant
export PATH=$PATH:$ANT_HOME/bin

source /etc/profile
ant -version

///////////////////////////////////////////////////////////
更新git
///////////////////////////////////////////////////////////

yum remove git
yum install perl-ExtUtils-CBuilder perl-ExtUtils-MakeMaker asciidoc xmlto
wget https://github.com/git/git/archive/v2.16.1.tar.gz
tar zxvf git-2.16.1.tar.gz
cd git-2.16.1
make configure
./configure --prefix=/usr/local/git --with-iconv=/usr/local/libiconv
make all doc
make install install-doc install-html

vim /etc/profile
添加一行
export PATH=/usr/local/git/bin:$PATH

source ~/.bashrc
git --version

配置http代理
git config --global http.proxy socks5://192.168.193.1:10808
***h代理
vim ~/.ssh/config
Host github.com
User git
Port 22
Hostname github.com
ProxyCommand nc -x 192.168.11.1:10808 %h %p

///////////////////////////////////////////////////
配置jdk，
cd /app
tar -zxvf jdk-8u261-linux-x64.tar.gz
ln -s jdk1.8.0_261 jdk

更新环境变量
vim /etc/profile
#set java environmen
export JAVA_HOME=/app/jdk
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

source ~/.bashrc

///////////////////////////////////////////////////////////
配置环境变量
///////////////////////////////////////////////////////////

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/tekken/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/home/tekken/anaconda3/etc/profile.d/conda.sh" ]; then
. "/home/tekken/anaconda3/etc/profile.d/conda.sh"
else
export PATH="/home/tekken/anaconda3/bin:$PATH"
fi
fi
unset __conda_setup
# <<< conda initialize <<<

export JAVA_HOME=/app/jdk
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH

export MAVEN_HOME=/app/maven
export PATH=$PATH:$MAVEN_HOME/bin

export ANT_HOME=/app/ant
export PATH=$PATH:$ANT_HOME/bin

export SCALA_HOME=/app/scala
export PATH=$PATH:$SCALA_HOME/bin

export RAPIDMINER_LIBS=/app/rapidminer_libs

export HDFS_NAMENODE_USER=tekken
export HDFS_DATANODE_USER=tekken
export HDFS_SECONDARYNAMENODE_USER=tekken
export YARN_RESOURCEMANAGER_USER=tekken
export YARN_NODEMANAGER_USER=tekken
export HADOOP_HOME=/app/hadoop
export HADOOP_CLASSPATH=$HADOOP_HOME/lib:$HADOOP_HOME/share/hadoop/yarn/lib:$RAPIDMINER_LIBS/*.jar
export HADOOP_USER_CLASSPATH_FIRST=true
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

export HIVE_HOME=/app/hive
export PATH=$PATH:$HIVE_HOME/bin

export ECLIPSE_HOME=/app/eclipse
export PATH=$ECLIPSE_HOME:$PATH

export MYSQL_HOME=/app/mysql
export PATH=$PATH:$MYSQL_HOME/bin

export SPARK_HOME=/app/spark
export PATH=$PATH:$SPARK_HOME/bin

export LIVY_HOME=/app/livy
export PATH=$PATH:$LIVY_HOME/bin

export MAHOUT_HOME=/app/mahout
export MAHOUT_CONF_DIR=$MAHOUT_HOME/conf
export PATH=$PATH:$MAHOUT_HOME/bin:$MAHOUT_CONF_DIR

export ELASTICSEARCH_HOME=/app/elasticsearch
export PATH=$PATH:$ELASTICSEARCH_HOME/bin

export KIBANA_HOME=/app/kibana
export PATH=$PATH:$KIBANA_HOME/bin

export LOGSTASH_HOME=/app/logstash
export PATH=$PATH:$LOGSTASH_HOME/bin

export GRADLE_HOME=/app/gradle
export PATH=$PATH:$GRADLE_HOME/bin

export ZOOKEEPER_HOME=/app/zookeeper
export PATH=$PATH:$ZOOKEEPER_HOME/bin

export KAFKA_HOME=/app/kafka
export PATH=$PATH:$KAFKA_HOME/bin

export HBASE_HOME=/app/hbase
export PATH=$PATH:$HBASE_HOME/bin
export HBASE_MANAGES_ZK=false

export HUE_HOME=/app/hue
export PATH=$PATH:$HUE_HOME/build/env/bin

export IDEA_HOME=/app/idea
export PATH=$PATH:$IDEA_HOME/bin

export CATALINA_HOME=/app/tomcat
export PATH=$PATH:$CATALINA_HOME/bin

#export PYTHON_HOME=/usr/local/python-2.7.14
#export PATH=$PATH:$PYTHON_HOME/bin

export PATH=/usr/local/bin:/usr/local/git/bin:/app/pycharm-2020.2.3/bin:$PATH

///////////////////////////////////////////////////////////
hadoop 3.3 源码编译
///////////////////////////////////////////////////////////

系统需求，源码包内：BUIDING.txt

Build instructions for Hadoop

----------------------------------------------------------------------------------
Requirements:

* Unix System
* JDK 1.8
* Maven 3.3 or later
* Protocol Buffers 3.7.1 (if compiling native code)
* CMake 3.1 or newer (if compiling native code)
* Zlib devel (if compiling native code)
* Cyrus SASL devel (if compiling native code)
* One of the compilers that support thread_local storage: GCC 4.8.1 or later, Visual Studio,
Clang (community version), Clang (version for iOS 9 and later) (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Doxygen ( if compiling libhdfspp and generating the documents )
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
* python (for releasedocs)
* bats (for shell code testing)
* Node.js / bower / Ember-cli (for YARN UI v2 building)

----------------------------------------------------------------------------------

环境准备

cmake 3.10.x centos7上编译安装：
tar -zxvf cmake-3.10.2.tar.gz
cd cmake-3.10.2
./bootstrap
gmake
gmake install
vim /etc/profile
export PATH=$PATH:/usr/local/bin

安装snappy：https://github.com/google/snappy/releases
注意，必须使用1.1.7版本，因为1.1.8版本是c++11标准，hadoop不支持。
tar -zxvf snappy-1.1.8.tar.gz
cd snappy-1.1.8
mkdir build
cd build
cmake ..
make
make install
ls -lh /usr/lib64|grep snappy

安装 Protocol Buffers 3.7.1
tar zxf /mnt/hgfs/Deep\ Learning/hadoop3.3/protobuf-java-3.7.1.tar.gz
cd protobuf-3.7.1/
./configure
make
make install
protoc --version

编译安装 intel 纠删码支持库
https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
&nbs***bsp;https://github.com/01org/isa-l)
yum install -y libzstd libzstd-devel zstd nasm yasm gcc clang icc VC make autoconf automake

yasm安装
curl -O -L http://www.tortall.net/projects/yasm/releases/yasm-1.3.0.tar.gz
./configure;make -j 8;make install

nasm安装
wget  https://www.nasm.us/pub/nasm/releasebuilds/2.15/nasm-2.15.tar.gz
tar xf nasm-2.14.02.tar.xz
./configure;make -j 8;make install

cd isa-l-master/
autoreconf -ivf
./autogen.sh
./configure --prefix=/usr --libdir=/usr/lib64

make

[root@spark isa-l-master]# make
Building isa-l.h
make --no-print-directory all-am
CC erasure_code/ec_base.lo
libtool: Version mismatch error. This is libtool 2.4.6, but the
libtool: definition of this LT_INIT comes from libtool 2.4.2.
libtool: You should recreate aclocal.m4 with macros from libtool 2.4.6
libtool: and run autoconf again.
make[1]: *** [erasure_code/ec_base.lo] 错误 63
make: *** [all] 错误 2

[ 根据错误提示，修改 xxxx.m4 文件中的版本号为提示版本号，然后执行 make]
make && make install

安装PMDK
https://www.jianshu.com/p/bba1cdf01647
https://github.com/pmem/pmdk
unzip pmdk-master.zip
yum install ndctl* daxctl*
make EXTRA_CFLAGS="-Wno-error"
make install prefix=/usr/lib64
cp /usr/lib64/lib/lib* /usr/lib64/

安装nodejs、bower
https://blog.csdn.net/longge9999/article/details/84721217
https://nodejs.org/en/download/
bower是前端的包管理工具，非常方便。由于是nodejs编写的所以先安装nodejs环境。
1.首先安装nodejs
（1）.下载nodejs：从nodejs官网下载nodejs的linux-64位版
（2）.解压nodejs的tar.gz包：tar -zxvf node-v0.12.4-linux-x64.tar.gz 。tar xvJf /mnt/hgfs/Deep\ Learning/hadoop3.3/node-v12.18.3-linux-x64.tar.xz
（3）.cd到nodejs的bin目录：查看node版本 ./node -v 查看npm版本 ./npm -v
（4）.设置全局
ln -s /usr/mingSoft/node-v0.12.4-linux-x64/bin/node /usr/local/bin/node
ln -s /usr/mingSoft/node-v0.12.4-linux-x64/bin/npm /usr/local/bin/npm
2.安装express：npm install express -gd
3.安装git：yum install git
4.全局安装bower：npm install bower -g
5.检测是否安装成功：bower --allow-root ,如果--allow-root不添加会提示root用户没有执行权限,除非用root以外的账户

find /app/maven/repository/ -name "*.lastUpdated" | xargs rm -rf
export MAVEN_OPTS="-Xms256m -Xmx4096m"

tar zxf /mnt/hgfs/Deep\ Learning/hadoop3.3/hadoop-3.3.0-src.tar.gz

[INFO] Downloading https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz to /app/maven/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
[INFO] Downloading https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz to /app/maven/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
[INFO] Downloading https://nodejs.org/dist/v8.17.0/node-v8.17.0-linux-x64.tar.gz to /app/maven/repository/com/github/eirslett/node/8.17.0/node-8.17.0-linux-x64.tar.gz
[INFO] Downloading https://github.com/yarnpkg/yarn/releases/download/v1.21.1/yarn-v1.21.1.tar.gz to /app/maven/repository/com/github/eirslett/yarn/1.21.1/yarn-1.21.1.tar.gz

git config --global http.https://github.com.proxy socks5://192.168.193.1:10808
之后去除git全局代理：git config --global --unset http.https://github.com.proxy

mvn clean package -Pdist,native,yarn-ui -DskipTests -Dtar -Drequire.snappy=true -Dbundle.snappy=true -Dsnappy.lib=/usr/lib64 -Drequire.zstd=true -Dbundle.zstd=true -Dzstd.lib=/usr/lib64 -Drequire.openssl=true -Dbundle.openssl=true -Dopenssl.lib=/usr/lib64 -Drequire.isal=true -Dbundle.isal=true -Disal.lib=/usr/lib64 -Drequire.pmdk=true -Dbundle.pmdk=true -Dpmdk.lib=/usr/lib64 -e -X
mvn clean package -Pdist,native,yarn-ui -DskipTests -Dtar -Drequire.snappy=true -Dbundle.snappy=true -Dsnappy.lib=/usr/lib64 -Drequire.zstd=true -Dbundle.zstd=true -Dzstd.lib=/usr/lib64 -Drequire.openssl=true -Dbundle.openssl=true -Dopenssl.lib=/usr/lib64 -Drequire.isal=true -Dbundle.isal=true -Disal.lib=/usr/lib64 -rf :hadoop-mapreduce-client-nativetask -e -X

编译结果位于
/app/hadoop-3.3.0-src/hadoop-dist/target/hadoop-3.3.0.tar.gz

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.3.0:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 2.170 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 45.283 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.489 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.264 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.258 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.856 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 4.247 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.464 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.682 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 1.572 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [01:04 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.970 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 3.335 s]
[INFO] Apache Hadoop Registry ............................. SUCCESS [ 3.297 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.133 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 18.688 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 58.261 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [01:02 min]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 5.770 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.641 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 23.675 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.115 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.129 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 15.479 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 23.790 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.105 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 8.737 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 25.840 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 2.935 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 5.091 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 3.182 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 18.162 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 1.452 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 5.029 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 2.712 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.926 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 0.112 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 4.956 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ 5.157 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ 0.103 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [ 3.060 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ 3.583 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 3.509 s]
[INFO] Apache Hadoop YARN TimelineService DocumentStore ... SUCCESS [ 3.835 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.103 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 2.032 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.512 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.258 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 5.262 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 6.562 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.178 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 6.048 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 3.960 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 4.745 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.725 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [ 0.109 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 2.793 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 1.528 s]
[INFO] Apache Hadoop YARN Application Catalog ............. SUCCESS [ 0.116 s]
[INFO] Apache Hadoop YARN Application Catalog Webapp ...... SUCCESS [ 16.586 s]
[INFO] Apache Hadoop YARN Application Catalog Docker Image SUCCESS [ 0.133 s]
[INFO] Apache Hadoop YARN Application MaWo ................ SUCCESS [ 0.109 s]
[INFO] Apache Hadoop YARN Application MaWo Core ........... SUCCESS [ 1.807 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.126 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 0.430 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [02:21 min]
[INFO] Apache Hadoop YARN CSI ............................. SUCCESS [ 23.686 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 11.127 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.677 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 13.806 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ 1.776 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 3.946 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 3.790 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 4.358 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 3.113 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 1.429 s]
[INFO] Apache Hadoop Dynamometer Workload Simulator ....... SUCCESS [ 3.162 s]
[INFO] Apache Hadoop Dynamometer Cluster Simulator ........ SUCCESS [ 2.851 s]
[INFO] Apache Hadoop Dynamometer Block Listing Generator .. SUCCESS [ 1.643 s]
[INFO] Apache Hadoop Dynamometer Dist ..................... SUCCESS [ 4.282 s]
[INFO] Apache Hadoop Dynamometer .......................... SUCCESS [ 0.110 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.448 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 2.082 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 5.101 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.604 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 3.878 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.628 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 2.716 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 3.059 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 9.529 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 1.965 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.644 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 3.838 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.652 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 2.922 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 1.820 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 2.156 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 7.653 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.106 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [01:06 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:00 min]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.319 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:53 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.179 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.277 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 22.059 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.098 s]
[INFO] Apache Hadoop Tencent COS Support .................. SUCCESS [ 4.143 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.413 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.107 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:47 min
[INFO] Finished at: 2020-09-11T15:41:47+08:00
[INFO] ------------------------------------------------------------------------

INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.2.2:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 0.797 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 0.767 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 1.364 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.231 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.261 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.137 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 2.640 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.495 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 11.353 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 1.865 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 56.598 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.870 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 2.726 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.121 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [02:19 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 34.550 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 4.456 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 3.934 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.171 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 12.324 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.105 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.111 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 9.353 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 21.628 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 2.869 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.108 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 6.688 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 22.736 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 11.219 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 3.405 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 2.892 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 13.352 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.890 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.390 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.896 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.879 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 0.107 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 2.727 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ 2.836 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ 0.103 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [ 2.706 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ 13.792 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 2.762 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.110 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.845 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.398 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.246 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 10.300 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 10.410 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.205 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 4.621 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 3.011 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 3.309 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.651 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [ 0.126 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 1.552 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 0.923 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 1.941 s]
[INFO] Yet Another Learning Platform ...................... SUCCESS [ 2.213 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.105 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [02:02 min]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 8.237 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.437 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 17.666 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ 1.530 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 2.946 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 3.536 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 4.558 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 2.984 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.552 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.691 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 3.373 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.599 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.702 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.619 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 3.631 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.336 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [01:28 min]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 1.274 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 18.753 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 1.654 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 1.466 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 2.806 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 2.389 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 8.323 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 7.516 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.091 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [01:08 min]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [ 57.557 s]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.221 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:52 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.182 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.208 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 22.206 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.097 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.445 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.098 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:15 min
[INFO] Finished at: 2021-03-16T15:14:43+08:00
[INFO] ------------------------------------------------------------------------

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.1.4:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 0.574 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 1.381 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.781 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 1.610 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.219 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.035 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 7.070 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.163 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 3.251 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 1.464 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 37.352 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.477 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 2.289 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.100 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 18.338 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 31.720 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 3.538 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 3.266 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 1.807 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 10.068 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.097 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.102 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 12.081 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 19.547 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 2.793 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.110 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 6.100 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 18.001 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.751 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 2.947 s]
[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [ 2.475 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 17.033 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.881 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 3.200 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.877 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.781 s]
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [ 0.109 s]
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [ 2.681 s]
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [ 2.745 s]
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [ 0.112 s]
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SUCCESS [ 2.517 s]
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [ 1.593 s]
[INFO] Apache Hadoop YARN Router .......................... SUCCESS [ 2.320 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.098 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.619 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.254 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.225 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 9.217 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 6.589 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 2.026 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 4.276 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 2.537 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 6.644 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.658 s]
[INFO] Apache Hadoop YARN Services ........................ SUCCESS [ 0.108 s]
[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [ 1.704 s]
[INFO] Apache Hadoop YARN Services API .................... SUCCESS [ 0.933 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.104 s]
[INFO] Apache Hadoop YARN UI .............................. SUCCESS [01:45 min]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 6.764 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.198 s]
[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [ 21.889 s]
[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [ 1.372 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 2.692 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 3.193 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 2.523 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 2.495 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.339 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.407 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 2.846 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 2.308 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.533 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.457 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 2.938 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 1.930 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [ 4.085 s]
[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [ 1.176 s]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 2.562 s]
[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [ 1.562 s]
[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [ 1.365 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 2.412 s]
[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [ 1.983 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 1.450 s]
[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [ 1.725 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 7.898 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.107 s]
[INFO] Apache Hadoop Client API ........................... SUCCESS [ 53.207 s]
[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [ 48.834 s]
[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [ 0.216 s]
[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [01:32 min]
[INFO] Apache Hadoop Client Packaging Invariants for Test . SUCCESS [ 0.196 s]
[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [ 0.211 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 15.416 s]
[INFO] Apache Hadoop Client Modules ....................... SUCCESS [ 0.106 s]
[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [ 0.450 s]
[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [ 0.104 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:11 min
[INFO] Finished at: 2020-11-23T15:46:35+08:00
[INFO] ------------------------------------------------------------------------

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 2.8.5:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [01:11 min]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 48.554 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 19.762 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 32.378 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.223 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 38.739 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 46.262 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [10:11 min]
[INFO] Apache Hadoop Auth ................................. SUCCESS [07:07 min]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 6.253 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [12:07 min]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 2.277 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 36.426 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.087 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [04:19 min]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:01 min]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 8.815 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 19.009 s]
[INFO] Apache Hadoop HDFS BookKeeper Journal .............. SUCCESS [01:05 min]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 1.639 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.086 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.084 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 6.699 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [07:28 min]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.089 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 3.236 s]
[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 8.896 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [ 1.585 s]
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [ 52.719 s]
[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 10.000 s]
[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [ 0.756 s]
[INFO] Apache Hadoop YARN Client .......................... SUCCESS [ 2.522 s]
[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [ 1.687 s]
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [ 1.563 s]
[INFO] Apache Hadoop YARN Applications .................... SUCCESS [ 0.089 s]
[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [ 1.501 s]
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [ 1.172 s]
[INFO] Apache Hadoop YARN Site ............................ SUCCESS [ 0.095 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 2.239 s]
[INFO] Apache Hadoop YARN Project ......................... SUCCESS [ 2.503 s]
[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [ 0.209 s]
[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 9.512 s]
[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 7.130 s]
[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [ 1.819 s]
[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [ 4.196 s]
[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [ 2.526 s]
[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [ 18.030 s]
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [ 1.027 s]
[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [ 2.391 s]
[INFO] Apache Hadoop MapReduce ............................ SUCCESS [ 1.783 s]
[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [ 13.272 s]
[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [ 11.606 s]
[INFO] Apache Hadoop Archives ............................. SUCCESS [ 1.131 s]
[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [ 1.200 s]
[INFO] Apache Hadoop Rumen ................................ SUCCESS [ 2.543 s]
[INFO] Apache Hadoop Gridmix .............................. SUCCESS [ 1.875 s]
[INFO] Apache Hadoop Data Join ............................ SUCCESS [ 1.290 s]
[INFO] Apache Hadoop Ant Tasks ............................ SUCCESS [ 0.987 s]
[INFO] Apache Hadoop Extras ............................... SUCCESS [ 1.404 s]
[INFO] Apache Hadoop Pipes ................................ SUCCESS [ 8.599 s]
[INFO] Apache Hadoop OpenStack support .................... SUCCESS [ 2.179 s]
[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [06:34 min]
[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 26.222 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 3.622 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 0.438 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 3.730 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 54.449 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 3.866 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.080 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 24.875 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:01 h
[INFO] Finished at: 2020-11-21T21:14:12+08:00
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "yarn-ui" could not be activated because it does not exist.

检查native库是否加载成功，如若有失败项，返回编译阶段查找问题。
hadoop checknative -a

hadoop 3.3.0 部署
mkdir -p /data/hadoop/dfs/name
mkdir -p /data/hadoop/dfs/data
mkdir -p /data/hadoop/dfs/tmp

cd /app/hadoop/etc/hadoop

vim core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://spark:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>file:///data/hadoop/dfs/tmp</value>
<description>Abase for other temporary directories.</description>
</property>

<property>
<name>hadoop.proxyuser.root.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.root.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.tekken.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.tekken.groups</name>
<value>*</value>
</property>
</configuration>

vim hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.http-address</name>
<value>spark:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>spark:50090</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///data/hadoop/dfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///data/hadoop/dfs/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
</property>

</configuration>

vim mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>spark:9001</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>spark:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>spark:19888</value>
</property>

<property>
<name>mapreduce.map.cpu.vcores</name>
<value>8</value>
</property>
<property>
<name>mapreduce.reduce.cpu.vcores</name>
<value>8</value>
</property>
<property>
<name>yarn.app.mapreduce.am.resource.cpu-vcores</name>
<value>8</value>
</property>
<property>

<name>mapreduce.application.classpath</name>
<value>/app/hadoop/lib:/app/hadoop/share/hadoop/yarn/lib:/app/rapidminer_libs/*.jar:/app/hadoop/etc/hadoop:/app/hadoop/share/hadoop/common/lib/*:/app/hadoop/share/hadoop/common/*:/app/hadoop/share/hadoop/hdfs:/app/hadoop/share/hadoop/hdfs/lib/*:/app/hadoop/share/hadoop/hdfs/*:/app/hadoop/share/hadoop/mapreduce/*:/app/hadoop/share/hadoop/yarn:/app/hadoop/share/hadoop/yarn/lib/*:/app/hadoop/share/hadoop/yarn/*</value>
</property>
</configuration>

开启yarn并行模式（默认是串行，一次只能运行一个任务）https://blog.csdn.net/u010770993/article/details/70312473
vim yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>spark</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://spark:19888/jobhistory/logs</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>spark:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>spark:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>spark:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.https.address</name>
<value>spark:8090</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>spark:8088</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<description>Classpath for typical applications.通过执行 hadoop classpath来确认hadoop的相关类路径</description>
<name>yarn.application.classpath</name>
<value>/app/hadoop/lib:/app/hadoop/share/hadoop/yarn/lib:/app/rapidminer_libs/*.jar:/app/hadoop/etc/hadoop:/app/hadoop/share/hadoop/common/lib/*:/app/hadoop/share/hadoop/common/*:/app/hadoop/share/hadoop/hdfs:/app/hadoop/share/hadoop/hdfs/lib/*:/app/hadoop/share/hadoop/hdfs/*:/app/hadoop/share/hadoop/mapreduce/*:/app/hadoop/share/hadoop/yarn:/app/hadoop/share/hadoop/yarn/lib/*:/app/hadoop/share/hadoop/yarn/*</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-vcores</name>
<value>8</value>
</property>
<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>8</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>12288</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>12288</value>
</property>


<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
</property>
<property>
<name>yarn.scheduler.fair.preemption</name>
<value>true</value>
</property>

<property>
<name>yarn.scheduler.fair.preemption.cluster-utilization-threshold</name>
<value>1.0</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.webapp.ui2.enable</name>
<value>true</value>
</property>
</configuration>

vim slaves
vim workers

//ubuntu 系统需要这样设置，解决报错：pdsh@gsta005: gsta005: rcmd: socket: Permission denied
sudo vim /etc/pdsh/rcmd_default
加入一行：
ssh

vim hadoop-env.sh
vim yarn-env.sh
将 export JAVA_HOME=${JAVA_HOME} 改为：
export JAVA_HOME=/app/jdk

hdfs namenode -format

start-dfs.sh
start-yarn.sh

启动history-server 【可选，默认没有启动】
mr-jobhistory-daemon.sh start historyserver
mapred --daemon start
mapred historyserver

timelineserver 启动有问题，新版本timelineserver 还没正式上线
yarn-daemon.sh start timelineserver
yarn timelineserver

hadoop dfsadmin -safemode leave

打开YARN-UI2界面
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YarnUI2.html

http://spark:8088/
http://spark:8088/ui2
http://spark:9870/dfshealth.html#tab-overview

////////////////////////////////////////////////////////
Hive3.1.2 源码编译部署
////////////////////////////////////////////////////////
find /app/maven/repository/ -name "*.lastUpdated" | xargs rm -rf
export MAVEN_OPTS="-Xms256m -Xmx4096m"

//hbase 版本必须是2.0.0，hadoop.version 最高3.2.x，高于此版本会报找不到类库错误
mvn clean package -Pdist -DskipTests -Dtar -Dhadoop.version=3.2.1 -Dspark.version=2.4.6 -Dhbase.version=2.0.0 -Dscala.binary.version=2.12 -Dscala.version=2.12.12 -Dprotobuf.version=3.7.1 -Dpig.version=0.16.0 -Djunit.version=4.12 -Dtez.version=0.9.1 -Dsnappy.version=1.1.7 -Dzookeeper.version=3.4.10 -e -X

mvn clean package -Pdist -DskipTests -Dtar -Dhadoop.version=3.1.4 -Dspark.version=2.4.7 -Dhbase.version=2.0.0 -Dscala.binary.version=2.11 -Dscala.version=2.11.12 -Dprotobuf.version=2.5.0 -Dpig.version=0.16.0 -Djunit.version=4.12 -Dtez.version=0.9.1 -Dsnappy.version=1.1.7 -Dzookeeper.version=3.4.10 -e -X

// spark 版本不支持3.x，只能使用2.4.x
mvn clean package -Pdist -DskipTests -Dtar -Dhadoop.version=3.2.2 -Dspark.version=2.4.7 -Dhbase.version=2.0.0 -Dscala.binary.version=2.12 -Dscala.version=2.12.13 -Dprotobuf.version=2.5.0 -Dpig.version=0.16.0 -Djunit.version=4.12 -Dtez.version=0.9.1 -Dsnappy.version=1.1.8.2 -Dzookeeper.version=3.4.14 -e -X

编译结果位于：/app/apache-hive-3.1.2-src/packaging/target/apache-hive-3.1.2-bin.tar.gz

INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Hive 3.1.2:
[INFO]
[INFO] Hive Upgrade Acid .................................. SUCCESS [ 4.440 s]
[INFO] Hive ............................................... SUCCESS [ 0.219 s]
[INFO] Hive Classifications ............................... SUCCESS [ 0.527 s]
[INFO] Hive Shims Common .................................. SUCCESS [ 1.482 s]
[INFO] Hive Shims 0.23 .................................... SUCCESS [ 2.202 s]
[INFO] Hive Shims Scheduler ............................... SUCCESS [ 1.366 s]
[INFO] Hive Shims ......................................... SUCCESS [ 0.940 s]
[INFO] Hive Common ........................................ SUCCESS [ 4.125 s]
[INFO] Hive Service RPC ................................... SUCCESS [ 2.097 s]
[INFO] Hive Serde ......................................... SUCCESS [ 3.442 s]
[INFO] Hive Standalone Metastore .......................... SUCCESS [ 20.806 s]
[INFO] Hive Metastore ..................................... SUCCESS [ 8.203 s]
[INFO] Hive Vector-Code-Gen Utilities ..................... SUCCESS [ 0.274 s]
[INFO] Hive Llap Common ................................... SUCCESS [ 2.900 s]
[INFO] Hive Llap Client ................................... SUCCESS [ 2.138 s]
[INFO] Hive Llap Tez ...................................... SUCCESS [ 2.264 s]
[INFO] Hive Spark Remote Client ........................... SUCCESS [ 2.272 s]
[INFO] Hive Query Language ................................ SUCCESS [ 32.556 s]
[INFO] Hive Llap Server ................................... SUCCESS [ 4.952 s]
[INFO] Hive Service ....................................... SUCCESS [ 4.756 s]
[INFO] Hive Accumulo Handler .............................. SUCCESS [ 3.713 s]
[INFO] Hive JDBC .......................................... SUCCESS [ 22.582 s]
[INFO] Hive Beeline ....................................... SUCCESS [ 3.393 s]
[INFO] Hive CLI ........................................... SUCCESS [ 2.940 s]
[INFO] Hive Contrib ....................................... SUCCESS [ 2.304 s]
[INFO] Hive Druid Handler ................................. SUCCESS [ 12.188 s]
[INFO] Hive HBase Handler ................................. SUCCESS [ 3.797 s]
[INFO] Hive JDBC Handler .................................. SUCCESS [ 2.185 s]
[INFO] Hive HCatalog ...................................... SUCCESS [ 0.417 s]
[INFO] Hive HCatalog Core ................................. SUCCESS [ 3.830 s]
[INFO] Hive HCatalog Pig Adapter .......................... SUCCESS [ 3.380 s]
[INFO] Hive HCatalog Server Extensions .................... SUCCESS [ 3.268 s]
[INFO] Hive HCatalog Webhcat Java Client .................. SUCCESS [ 3.401 s]
[INFO] Hive HCatalog Webhcat .............................. SUCCESS [ 9.786 s]
[INFO] Hive HCatalog Streaming ............................ SUCCESS [ 3.678 s]
[INFO] Hive HPL/SQL ....................................... SUCCESS [ 3.868 s]
[INFO] Hive Streaming ..................................... SUCCESS [ 3.103 s]
[INFO] Hive Llap External Client .......................... SUCCESS [ 3.214 s]
[INFO] Hive Shims Aggregator .............................. SUCCESS [ 0.099 s]
[INFO] Hive Kryo Registrator .............................. SUCCESS [ 2.517 s]
[INFO] Hive TestUtils ..................................... SUCCESS [ 0.193 s]
[INFO] Hive Packaging ..................................... SUCCESS [ 35.135 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:51 min
[INFO] Finished at: 2021-03-22T21:29:27+08:00
[INFO] ------------------------------------------------------------------------

安装 mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz
============================================================
cd /app
tar zxf /mnt/hgfs/Deep\ Learning/hadoop3.3/mysql-5.7.31-linux-glibc2.12-x86_64.tar.gz
ln -sf mysql-5.7.31-linux-glibc2.12-x86_64 mysql

root身份执行：
groupadd mysql
useradd -r -g mysql mysql
chown -R mysql:mysql /app/mysql/
passwd mysql

mkdir -p /data/mysql/data
chown -R mysql:mysql /data/mysql
chown -R mysql:mysql /app/mysql

yum install libaio*

cd /app/mysql
./bin/mysqld --initialize --user=mysql --basedir=/app/mysql --datadir=/data/mysql/data
//临时密码for root：z<FgyL;_49Ge
A temporary password is generated for root@localhost: 1qr#<it.9MBu

./bin/mysql_ssl_rsa_setup --datadir=/data/mysql/data

vim /etc/my.cnf

[mysqld]
basedir=/app/mysql
datadir=/data/mysql/data
socket=/app/mysql/mysql.sock
character-set-server=utf8mb4
lower_case_table_names=1

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Settings user and group are ignored when systemd is used.
# If you need to run mysqld under a different user or group,
# customize your systemd unit file for mariadb according to the
# instructions in http://fedoraproject.org/wiki/Systemd

[mysqld_safe]
log-error=/data/mysql/mysqld.log
pid-file=/app/mysql/mysqld.pid
#
# include all files from the config directory
#
#!includedir /etc/my.cnf.d

vim /app/mysql/support-files/mysql.server
修改文件中的几个变更值
basedir=/app/mysql
datadir=/data/mysql/data
mysqld_pid_file_path=/app/mysql/mysqld.pid

#若mysql的安装目录是/usr/local/mysql,则可省略此步:
cp /app/mysql/support-files/mysql.server /etc/init.d/mysqld
cd /etc/init.d
ln -s mysqld mysqld.service
vim /data/mysql/mysqld.log

chown -R mysql:mysql /data/mysql

添加自启动服务/    启动 / 查看状态
systemctl enable mysqld.service
systemctl start mysqld.service
systemctl status mysqld.service

如果启动失败，则手工启动尝试：
bin/mysqld_safe --user=mysql --datadir=/data/mysql/data &

ln -s /app/mysql/mysql.sock /tmp/mysql.sock

登录mysql及改密码与配置远程访问
/app/mysql/bin/mysql --user=root -p

使用之前的临时密码登陆，

mysql> set password=password('123456');
mysql> grant all privileges on *.* to root@'%' identified by '123456';
mysql> flush privileges;

用root用户登录到mysql
mysql -uroot -p -P 3306 --default-character-set=utf8mb4

//设置root用户可远程访问
mysql> grant all on *.* to 'root'@'%' identified by '123456' WITH GRANT OPTION;
mysql> flush privileges;

忘记root密码后，如何找回密码
cd $MYSQL_HOME
./bin/mysqld_safe --basedir=/data/mysql --datadir=/data/mysql/data --skip-grant-tables &
mysql -u root mysql
UPDATE user SET password=PASSWORD("new_password") WHERE user='root';
//5.7.10版本用此语句：
UPDATE user SET authentication_string=PASSWORD("123456") WHERE User='root';
FLUSH PRIVILEGES;

创建数据库实例 hive，作为hive的元数据库。默认编码latin1 , 决不能用utf8编码，否则在调用create语句中报错：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:For direct MetaStore DB connections, we don't support retries at the client level.)
mysql> create database hive DEFAULT CHARACTER SET latin1 COLLATE latin1_swedish_ci;

Hive设置MySQL元数据中文乱码编码问题以及解决
https://blog.csdn.net/cuichunchi/article/details/107683467

alter database hive character set latin1;

alter table COLUMNS_V2 modify column COMMENT varchar(256) character set utf8;
alter table TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
alter table PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;
alter table PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8;
alter table INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;

最后别忘记修改hive的jdbc连接，将字符集修改为utf8，如
javax.jdo.option.ConnectionURL
jdbc:mysql://192.168.0.128:3306/hive?characterEncoding=UTF-8

create table test (user_id string comment '工号',brand_id string comment '品牌',type string comment '类别',visit_datetime string comment '日期') row format delimited fields terminated by ',' stored as textfile;
insert into test values('1','2','3','4');
select * from test;
show create table test;

Hive3.1.2 源码编译部署
============================================================
tar zxf apache-hive-3.1.2-src/packaging/target/apache-hive-3.1.2-bin.tar.gz
ln -sf apache-hive-3.1.2-bin hive

修改hive中的xml配置
cd /app/hive/conf
cp hive-default.xml.template hive-site.xml
cp hive-log4j2.properties.template hive-log4j2.properties
cp hive-exec-log4j2.properties.template hive-exec-log4j2.properties

vim hive-site.xml
<property>
<name>datanucleus.connectionPoolingType</name>
<value>dbcp</value>
<description>
Expects one of [bonecp, dbcp, hikaricp, none].
Specify connection pool library for datanucleus
</description>
</property>


<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://spark:3306/hive?useSSL=false&createDatabaseIfNotExist=true&characterEncoding=latin1</value>
</property>

<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
</property>

<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>root</value>
</property>

<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>123456</value>
</property>

<property>
<name>hive.exec.local.scratchdir</name>
<value>/data/hive/tmp</value>
<description>Local scratch space for Hive jobs</description>
</property>

<property>
<name>hive.downloaded.resources.dir</name>
<value>/data/hive/tmp/${hive.session.id}_resources</value>
<description>Temporary local directory for added resources in the remote file system.</description>
</property>

<property>
<name>hive.server2.logging.operation.log.location</name>
<value>/data/hive/tmp/${hive.session.id}_operation_logs</value>
<description>Top level directory where operation logs are stored if logging functionality is enabled</description>
</property>

<property>
<name>hive.metastore.schema.verification</name>
<value>false</value>
<description>
Enforce metastore schema version consistency.
True: Verify that version information stored in is compatible with one from Hive jars. Also disable automatic
schema migration attempt. Users are required to manually migrate schema after Hive upgrade which ensures
proper metastore schema migration. (Default)
False: Warn if the version information stored in metastore doesn't match with one from in Hive jars.
</description>
</property>

查找  将其删掉。然后保存退出

在hive/lib目录中放置mysql驱动
cp /mnt/hgfs/hadoop3/mysql-connector-java-5.1.48-bin.jar /app/hive/lib/

初始化hive数据库
解决初始化hive数据库时候jar版本不一致，导致的找不到方法的报错
cp /app/hadoop/share/hadoop/common/lib/guava-27.0-jre.jar /app/hive/lib
mv /app/hive/lib/guava-19.0.jar /app/hive/lib/guava-19.0.jar.bak
/app/hive/bin/schematool -initSchema -dbType mysql

//hadoop2.7.x适用，hadoop3 不适用
更新hadoop/lib中的旧jar，用hive中用的jar文件复制粘贴到hadoop的lib中
rm /app/hadoop/lib/jline*.jar
rm /app/hadoop/share/hadoop/yarn/lib/jline*.jar
cp /app/hive/lib/jline-2.12.jar /app/hadoop/lib
cp /app/hive/lib/jline-2.12.jar /app/hadoop/share/hadoop/common/lib/

重启hadoop服务。
测试hive，输入hive进入hive的shell

配置hive web ui：现在新版本Hive不推荐用hwi了，官方推荐用Hue
https://www.cnblogs.com/qingyunzong/p/8715925.html

///////////////////////////////////////////////////////////////////////
安装 zookeeper3.6.2
http://www.cnblogs.com/z-sm/p/5691752.html
///////////////////////////////////////////////////////////////////////
tar zxf /mnt/hgfs/Deep\ Learning/hadoop3.3/apache-zookeeper-3.6.2-bin.tar.gz

ln -sf apache-zookeeper-3.6.2-bin zookeeper

vim /etc/profile
export ZOOKEEPER_HOME=/app/zookeeper
export PATH=$PATH:$ZOOKEEPER_HOME/bin

source /etc/profile

cd /app/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
mkdir -p /data/zookeeper/data
//在每个集群节点上创建myid文件，myid内容应和zoo.cfg中的server.N指定的值相同
echo 1 > /data/zookeeper/data/myid

vim zoo.cfg
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/data/zookeeper/data
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

server.1=spark:2888:3888
server.2=h02:2888:3888
server.3=h03:2888:3888

////////////////////////////
启动zookeeper
zkServer.sh start
停止
zkServer.sh stop

//////////////////////////////////////////////////////////////////
zookeeper shell 使用示意
zkCli.sh -server localhost:2181
1. 显示根目录下、文件： ls / 使用 ls 命令来查看当前 ZooKeeper 中所包含的内容
2. 显示根目录下、文件： ls2 / 查看当前节点数据并能看到更新次数等数据
3. 创建文件并设置初始内容：create /zk "test" 创建一个新的 znode节点“ zk ”以及与它关联的字符串
4. 获取文件内容： get /zk 确认 znode 是否包含我们所创建的字符串
5. 修改文件内容： set /zk "zkbak" 对 zk 所关联的字符串进行设置
6. 删除文件： delete /zk 将刚才创建的 znode 删除
7. 删除目录（即使非空）：rmr 目录如可用于删除Kafka的某个Consumer Group
8. 退出客户端： quit
9. 帮助命令： help

zookeeper 命令行四字命令
先安装nc包
yum install nc

可通过四字命令更方便地获取服务端信息，四字命令的用法为 echo 四字命令|nc localhost 2181 ，常用的四字命令如下：

conf：输出Zookeeper服务器配置的详细信息
cons：输出所有连接到服务器的客户端的完全的连接/会话的详细信息。包括“接收/发送”的包数量、会话ID、操作延迟、最后的操作执行等
dump：输出未经处理的会话和临时节点
envi：输出关于服务器运行环境的详细信息
reqs：输出未经处理的请求
ruok：测试服务是否处于正确状态。若是则会返回“imok”，否则不做任何反应
stat：输出关于性能和连接的客户端的列表。（通过此命令也可查看节点是leader还是follower）
wchs：输出服务器watch的详细信息
wchc：通过session列出服务器watch的详细信息，它的输出是一个与watch相关的会话的列表
wchp：通过路径列出服务器watch的详细信息，它输出一个与session相关的路径
mntr：输出一些Zookeeper运行时信息，通过对这些返回结果的解析可以达到监控效果

/////////////////////////////////////////////////////////////////////////////
java 客户端连接zookeeper
package cn.edu.buaa.act.test.TestZookeeper;

import java.io.IOException;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs.Ids;
import org.apache.zookeeper.ZooKeeper;

/**
* 此类包含两个主要的 ZooKeeper 函数，分别为 createZKInstance （）和 ZKOperations （）。<br>
* <br>
*
* (1)createZKInstance （）函数负责对 ZooKeeper 实例 zk 进行初始化。 ZooKeeper 类有两个构造函数，我们这里使用
* “ ZooKeeper （ String connectString, ， int sessionTimeout, ， Watcher watcher
* ）”对其进行初始化。因此，我们需要提供初始化所需的，连接字符串信息，会话超时时间，以及一个 watcher 实例。<br>
* <br>
*
* (2)ZKOperations （）函数是我们所定义的对节点的一系列操作。它包括：创建 ZooKeeper
* 节点、查看节点、修改节点数据、查看修改后节点数据、删除节点、查看节点是否存在。另外，需要注意的是：在创建节点的时候，需要提供节点的名称、数据、
* 权限以及节点类型。此外，使用 exists 函数时，如果节点不存在将返回一个 null 值。
*/
public class ZookeeperTest {

// 会话超时时间，设置为与系统默认时间一致

private static final int SESSION_TIMEOUT = 30000;

// 创建 ZooKeeper 实例

ZooKeeper zk;

// 创建 Watcher 实例

Watcher wh = new Watcher() {

public void process(org.apache.zookeeper.WatchedEvent event) {
System.out.println("event=" + event.toString());
}

};

// 初始化 ZooKeeper 实例

private void createZKInstance() throws IOException {
zk = new ZooKeeper("192.168.6.131:2181,192.168.6.132:2181,192.168.6.133:2181", ZookeeperTest.SESSION_TIMEOUT,
this.wh);
}

private void ZKOperations() throws IOException, InterruptedException, KeeperException {

System.out.println("\n 创建 ZooKeeper 节点 (znode ： zoo2, 数据： myData2 ，权限： OPEN_ACL_UNSAFE ，节点类型： Persistent");
zk.create("/zoo2", "myData2".getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);

System.out.println("\n 查看是否创建成功： ");
System.out.println(new String(zk.getData("/zoo2", false, null)));

System.out.println("\n 修改节点数据 ");
zk.setData("/zoo2", "shenlan211314".getBytes(), -1);

System.out.println("\n 查看是否修改成功： ");
System.out.println(new String(zk.getData("/zoo2", false, null)));

System.out.println("\n 删除节点 ");
zk.delete("/zoo2", -1);

System.out.println("\n 查看节点是否被删除： ");
System.out.println(" 节点状态： [" + zk.exists("/zoo2", false) + "]");

}

private void ZKClose() throws InterruptedException {
zk.close();
}

public static void main(String[] args) throws IOException, InterruptedException, KeeperException {
ZookeeperTest dm = new ZookeeperTest();
dm.createZKInstance();
dm.ZKOperations();
dm.ZKClose();
}

}

hbase 2.3.2 源码编译
============================================================
tar zxf /mnt/hgfs/Deep\ Learning/hadoop3.3/hbase-2.3.2-src.tar.gz
cd hbase-2.3.2/
修改pom.xml 增加阿里资源库镜像
<repositories>
<repository>
<id>ali</id>
<name>ali</name>
      <url>http://maven.aliyun.com/repository/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>

mvn clean package -Phadoop-2.0 -Dhadoop-tow.version=2.8.5 -DskipTests assembly:single -e -X
mvn clean package -Phadoop-3.0 -Dhadoop-three.version=3.2.1 -DskipTests assembly:single -e -X
mvn clean package -Phadoop-3.0 -Dhadoop-three.version=3.3.0 -DskipTests assembly:single -e -X
mvn clean package -Phadoop-3.0 -Dhadoop-three.version=3.1.4 -DskipTests assembly:single -e -X

mvn clean package -Phadoop-3.0 -Dhadoop-three.version=3.2.2 -DskipTests assembly:single -e -X

编译结果位于：hbase-2.3.2-src/hbase-assembly/target/hbase-2.3.2-bin.tar.gz

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache HBase 2.4.2:
[INFO]
[INFO] Apache HBase ....................................... SUCCESS [ 6.888 s]
[INFO] Apache HBase - Checkstyle .......................... SUCCESS [ 0.534 s]
[INFO] Apache HBase - Annotations ......................... SUCCESS [ 0.630 s]
[INFO] Apache HBase - Build Configuration ................. SUCCESS [ 0.117 s]
[INFO] Apache HBase - Logging ............................. SUCCESS [ 1.015 s]
[INFO] Apache HBase - Shaded Protocol ..................... SUCCESS [ 25.505 s]
[INFO] Apache HBase - Common .............................. SUCCESS [ 18.260 s]
[INFO] Apache HBase - Metrics API ......................... SUCCESS [ 1.361 s]
[INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS [ 1.653 s]
[INFO] Apache HBase - Metrics Implementation .............. SUCCESS [ 1.935 s]
[INFO] Apache HBase - Hadoop Two Compatibility ............ SUCCESS [ 5.235 s]
[INFO] Apache HBase - Protocol ............................ SUCCESS [ 5.082 s]
[INFO] Apache HBase - Client .............................. SUCCESS [ 7.752 s]
[INFO] Apache HBase - Zookeeper ........................... SUCCESS [ 1.638 s]
[INFO] Apache HBase - Replication ......................... SUCCESS [ 1.433 s]
[INFO] Apache HBase - Resource Bundle ..................... SUCCESS [ 0.255 s]
[INFO] Apache HBase - HTTP ................................ SUCCESS [ 16.636 s]
[INFO] Apache HBase - Asynchronous FileSystem ............. SUCCESS [ 4.396 s]
[INFO] Apache HBase - Procedure ........................... SUCCESS [ 1.883 s]
[INFO] Apache HBase - Server .............................. SUCCESS [ 17.357 s]
[INFO] Apache HBase - MapReduce ........................... SUCCESS [ 3.419 s]
[INFO] Apache HBase - Testing Util ........................ SUCCESS [ 2.077 s]
[INFO] Apache HBase - Thrift .............................. SUCCESS [ 5.794 s]
[INFO] Apache HBase - RSGroup ............................. SUCCESS [ 3.095 s]
[INFO] Apache HBase - Shell ............................... SUCCESS [ 27.354 s]
[INFO] Apache HBase - Coprocessor Endpoint ................ SUCCESS [ 3.391 s]
[INFO] Apache HBase - Integration Tests ................... SUCCESS [ 4.036 s]
[INFO] Apache HBase - Rest ................................ SUCCESS [ 8.188 s]
[INFO] Apache HBase - Examples ............................ SUCCESS [ 3.299 s]
[INFO] Apache HBase - Shaded .............................. SUCCESS [ 0.180 s]
[INFO] Apache HBase - Shaded - Client (with Hadoop bundled) SUCCESS [ 22.635 s]
[INFO] Apache HBase - Shaded - Client ..................... SUCCESS [ 8.077 s]
[INFO] Apache HBase - Shaded - MapReduce .................. SUCCESS [ 19.217 s]
[INFO] Apache HBase - External Block Cache ................ SUCCESS [ 3.200 s]
[INFO] Apache HBase - HBTop ............................... SUCCESS [ 1.390 s]
[INFO] Apache HBase - Assembly ............................ SUCCESS [01:43 min]
[INFO] Apache HBase - Shaded - Testing Util ............... SUCCESS [ 39.594 s]
[INFO] Apache HBase - Shaded - Testing Util Tester ........ SUCCESS [ 2.090 s]
[INFO] Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 1.322 s]
[INFO] Apache HBase Shaded Packaging Invariants (with Hadoop bundled) SUCCESS [ 1.070 s]
[INFO] Apache HBase - Archetypes .......................... SUCCESS [ 0.099 s]
[INFO] Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 2.229 s]
[INFO] Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 2.412 s]
[INFO] Apache HBase - Archetype builder ................... SUCCESS [ 0.440 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 06:33 min
[INFO] Finished at: 2021-03-22T21:47:07+08:00
[INFO] ------------------------------------------------------------------------

[INFO] Assemblies have been skipped per configuration of the skipAssembly parameter.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache HBase 2.3.2:
[INFO]
[INFO] Apache HBase ....................................... SUCCESS [ 4.620 s]
[INFO] Apache HBase - Checkstyle .......................... SUCCESS [ 1.013 s]
[INFO] Apache HBase - Annotations ......................... SUCCESS [ 1.335 s]
[INFO] Apache HBase - Build Configuration ................. SUCCESS [ 0.126 s]
[INFO] Apache HBase - Logging ............................. SUCCESS [ 0.743 s]
[INFO] Apache HBase - Shaded Protocol ..................... SUCCESS [ 18.026 s]
[INFO] Apache HBase - Common .............................. SUCCESS [ 10.263 s]
[INFO] Apache HBase - Metrics API ......................... SUCCESS [ 1.173 s]
[INFO] Apache HBase - Hadoop Compatibility ................ SUCCESS [ 1.959 s]
[INFO] Apache HBase - Metrics Implementation .............. SUCCESS [ 1.541 s]
[INFO] Apache HBase - Hadoop Two Compatibility ............ SUCCESS [ 4.396 s]
[INFO] Apache HBase - Protocol ............................ SUCCESS [ 5.327 s]
[INFO] Apache HBase - Client .............................. SUCCESS [ 8.098 s]
[INFO] Apache HBase - Zookeeper ........................... SUCCESS [ 1.814 s]
[INFO] Apache HBase - Replication ......................... SUCCESS [ 1.445 s]
[INFO] Apache HBase - Resource Bundle ..................... SUCCESS [ 0.214 s]
[INFO] Apache HBase - HTTP ................................ SUCCESS [ 6.234 s]
[INFO] Apache HBase - Asynchronous FileSystem ............. SUCCESS [ 3.101 s]
[INFO] Apache HBase - Procedure ........................... SUCCESS [ 1.879 s]
[INFO] Apache HBase - Server .............................. SUCCESS [ 19.325 s]
[INFO] Apache HBase - MapReduce ........................... SUCCESS [ 4.099 s]
[INFO] Apache HBase - Testing Util ........................ SUCCESS [ 2.386 s]
[INFO] Apache HBase - Thrift .............................. SUCCESS [ 7.234 s]
[INFO] Apache HBase - RSGroup ............................. SUCCESS [ 3.260 s]
[INFO] Apache HBase - Shell ............................... SUCCESS [ 4.175 s]
[INFO] Apache HBase - Coprocessor Endpoint ................ SUCCESS [ 4.593 s]
[INFO] Apache HBase - Integration Tests ................... SUCCESS [ 5.862 s]
[INFO] Apache HBase - Rest ................................ SUCCESS [ 4.248 s]
[INFO] Apache HBase - Examples ............................ SUCCESS [ 3.514 s]
[INFO] Apache HBase - Shaded .............................. SUCCESS [ 0.170 s]
[INFO] Apache HBase - Shaded - Client (with Hadoop bundled) SUCCESS [ 15.094 s]
[INFO] Apache HBase - Shaded - Client ..................... SUCCESS [ 8.383 s]
[INFO] Apache HBase - Shaded - MapReduce .................. SUCCESS [ 11.982 s]
[INFO] Apache HBase - External Block Cache ................ SUCCESS [ 1.780 s]
[INFO] Apache HBase - HBTop ............................... SUCCESS [ 1.321 s]
[INFO] Apache HBase - Assembly ............................ SUCCESS [01:21 min]
[INFO] Apache HBase - Shaded - Testing Util ............... SUCCESS [ 34.686 s]
[INFO] Apache HBase - Shaded - Testing Util Tester ........ SUCCESS [ 2.253 s]
[INFO] Apache HBase Shaded Packaging Invariants ........... SUCCESS [ 1.449 s]
[INFO] Apache HBase Shaded Packaging Invariants (with Hadoop bundled) SUCCESS [ 0.728 s]
[INFO] Apache HBase - Archetypes .......................... SUCCESS [ 0.077 s]
[INFO] Apache HBase - Exemplar for hbase-client archetype . SUCCESS [ 2.586 s]
[INFO] Apache HBase - Exemplar for hbase-shaded-client archetype SUCCESS [ 2.471 s]
[INFO] Apache HBase - Archetype builder ................... SUCCESS [ 0.625 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:00 min
[INFO] Finished at: 2020-10-22T13:04:34+08:00
[INFO] ------------------------------------------------------------------------

hbase 2.3.2 部署

tar zxf hbase-2.3.2-src/hbase-assembly/target/hbase-2.3.2-bin.tar.gz
ln -sf hbase-2.3.2 hbase
cd hbase/conf

cp /app/hadoop/etc/hadoop/hdfs-site.xml /app/hbase/conf/
cp /app/hadoop/etc/hadoop/core-site.xml /app/hbase/conf/

vim /app/hbase/conf/hbase-site.xml
<configuration>

<property>
<name>hbase.rootdir</name>
<value>hdfs://spark:9000/hbase</value>
</property>
<property>
<name>hbase.master.info.bindAddress</name>
<value>spark</value>
</property>
<property>
<name>hbase.master.info.port</name>
<value>16010</value>
</property>

<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>

<property>
<name>hbase.zookeeper.quorum</name>
<value>spark</value>
</property>

<property>
<name>hbase.tmp.dir</name>
<value>/data/hbase/tmp</value>
</property>
<property>
<name>hbase.unsafe.stream.capability.enforce</name>
<value>false</value>
</property>

</configuration>

vim /app/hbase/conf/hbase-env.sh
export JAVA_HOME="/app/jdk"

vim /app/hbase/conf/regionservers
spark

启动所有的hbase
分别启动zk
    zkServer.sh start
启动hadoop集群
    start-all.sh
启动hbase，在主节点上运行：
    start-hbase.sh
通过浏览器访问hbase管理页面
http://spark:16010/master-status
为保证集群的可靠性，要启动多个HMaster
hbase-daemon.sh start master

查看hbase版本
hbase version
hbase shell操作
进入hbase shell
hbase shell

查看当前用户
hbase(main)> whoami

查看有哪些表
hbase(main)> list

创建表
# 语法：create <table>, {NAME => <family>, VERSIONS => <VERSIONS>}
# 例如：创建表t1，有两个family name：f1，f2，且版本数均为2
hbase(main)> create 't1',{NAME => 'f1', VERSIONS => 2},{NAME => 'f2', VERSIONS => 2}

删除表
分两步：首先disable，然后drop
例如：删除表t1

hbase(main)> disable 't1'
hbase(main)> drop 't1'

查看表的结构
# 语法：describe <table>
# 例如：查看表t1的结构
hbase(main)> describe 't1'

修改表结构
修改表结构必须先disable
# 语法：alter 't1', {NAME => 'f1'}, {NAME => 'f2', METHOD => 'delete'}
# 例如：修改表test1的cf的TTL为180天
hbase(main)> disable 'test1'
hbase(main)> alter 'test1',{NAME=>'body',TTL=>'15552000'},{NAME=>'meta', TTL=>'15552000'}
hbase(main)> enable 'test1'

分配权限
# 语法 : grant <user> <permissions> <table> <column family> <column qualifier> 参数后面用逗号分隔
# 权限用五个字母表示： "RWXCA".
# READ('R'), WRITE('W'), EXEC('X'), CREATE('C'), ADMIN('A')
# 例如，给用户‘test'分配对表t1有读写的权限，
hbase(main)> grant 'test','RW','t1'

查看权限
# 语法：user_permission <table>
# 例如，查看表t1的权限列表
hbase(main)> user_permission 't1'

收回权限
# 与分配权限类似，语法：revoke <user> <table> <column family> <column qualifier>
# 例如，收回test用户在表t1上的权限
hbase(main)> revoke 'test','t1'

添加数据
# 语法：put <table>,<rowkey>,<family:column>,<value>,<timestamp>
# 例如：给表t1的添加一行记录：rowkey是rowkey001，family name：f1，column name：col1，value：value01，timestamp：系统默认
hbase(main)> put 't1','rowkey001','f1:col1','value01'

查询某行记录
# 语法：get <table>,<rowkey>,[<family:column>,....]
# 例如：查询表t1，rowkey001中的f1下的col1的值
hbase(main)> get 't1','rowkey001', 'f1:col1'
# 或者：
hbase(main)> get 't1','rowkey001', {COLUMN=>'f1:col1'}
# 查询表t1，rowke002中的f1下的所有列值
hbase(main)> get 't1','rowkey001'

扫描表
# 语法：scan <table>, {COLUMNS => [ <family:column>,.... ], LIMIT => num}
# 另外，还可以添加STARTROW、TIMERANGE和FITLER等高级功能
# 例如：扫描表t1的前5条数据
hbase(main)> scan 't1',{LIMIT=>5}

查询表中的数据行数
# 语法：count <table>, {INTERVAL => intervalNum, CACHE => cacheNum}
# INTERVAL设置多少行显示一次及对应的rowkey，默认1000；CACHE每次去取的缓存区大小，默认是10，调整该参数可提高查询速度
# 例如，查询表t1中的行数，每100条显示一次，缓存区为500
hbase(main)> count 't1', {INTERVAL => 100, CACHE => 500}

删除行中的某个列值
# 语法：delete <table>, <rowkey>, <family:column> , <timestamp>,必须指定列名
# 例如：删除表t1，rowkey001中的f1:col1的数据
hbase(main)> delete 't1','rowkey001','f1:col1'
注：将删除该行f1:col1列所有版本的数据

删除行
# 语法：deleteall <table>, <rowkey>, <family:column> , <timestamp>，可以不指定列名，删除整行数据
# 例如：删除表t1，rowk001的数据
hbase(main)> deleteall 't1','rowkey001'

删除表中的所有数据
# 语法： truncate <table>
# 其具体过程是：disable table -> drop table -> create table
# 例如：删除表t1的所有数据
hbase(main)> truncate 't1'

移动region
# 语法：move 'encodeRegionName', 'ServerName'
# encodeRegionName指的regioName后面的编码，ServerName指的是master-status的Region Servers列表
# 示例
hbase(main)>move '4343995a58be8e5bbc739af1e91cd72d', 'db-41.xxx.xxx.org,60020,139****516739'

开启/关闭region
# 语法：balance_switch true|false
hbase(main)> balance_switch

手动split
# 语法：split 'regionName', 'splitKey'
手动触发major compaction

#语法：
#Compact all regions in a table:
#hbase> major_compact 't1'
#Compact an entire region:
#hbase> major_compact 'r1'
#Compact a single column family within a region:
#hbase> major_compact 'r1', 'c1'
#Compact a single column family within a table:
#hbase> major_compact 't1', 'c1'

修改hdfs配置
hdfs配置位置：/etc/hadoop/conf
# 同步hdfs配置
cat /home/hadoop/slaves|xargs -i -t scp /etc/hadoop/conf/hdfs-site.xml hadoop@{}:/etc/hadoop/conf/hdfs-site.xml
#关闭：
cat /home/hadoop/slaves|xargs -i -t ssh hadoop@{} "sudo /home/hadoop/cdh4/hadoop-2.0.0-cdh4.2.1/sbin/hadoop-daemon.sh --config /etc/hadoop/conf stop datanode"
#启动：
cat /home/hadoop/slaves|xargs -i -t ssh hadoop@{} "sudo /home/hadoop/cdh4/hadoop-2.0.0-cdh4.2.1/sbin/hadoop-daemon.sh --config /etc/hadoop/conf start datanode"

修改hbase配置
hbase配置位置：

# 同步hbase配置
cat /home/hadoop/hbase/conf/regionservers|xargs -i -t scp /home/hadoop/hbase/conf/hbase-site.xml hadoop@{}:/home/hadoop/hbase/conf/hbase-site.xml

# graceful重启
cd ~/hbase
bin/graceful_stop.sh --restart --reload --debug inspurXXX.xxx.xxx.org

Spark3.0.1 源码编译安装
cd /app
tar zxf /mnt/hgfs/Deep\ Learning/hadoop3.3/spark-3.0.1.tgz
mv spark-3.0.1 spark-3.0.1-src
cd spark-3.0.1-src

修改pom.xml 增加阿里资源库镜像
<repository>
<id>ali</id>
<name>ali</name>
      <url>http://maven.aliyun.com/repository/public/</url>
<releases>
<enabled>true</enabled>
</releases>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>

下载oracle jdbc jar，手动添加到maven，这个包只能从oracle官网下：https://www.cnblogs.com/651434092qq/p/12010985.html
mvn install:install-file -DgroupId=com.oracle -DartifactId=ojdbc6 -Dversion=11.2.0.1.0 -Dpackaging=jar -Dfile=ojdbc6.jar

*****************************************************************************
vim core/src/main/scala/org/apache/spark/internal/config/package.scala: private[spark] val LISTENER_BUS_EVENT_QUEUE_CAPACITY = 的默认值，从1w增加到100w，解决报错：Dropping event from queue
*****************************************************************************

下载hive1.2.2源码：https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-1.2.2/apache-hive-1.2.2-bin.tar.gz
提取 apache-hive-1.2.2-src/shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java 到 /app/spark-2.4.7-src/sql/hive-thriftserver/src/main/java/org/apache/hadoop/hive/shims/
cd /opt/tgz
tar zxf /mnt/hgfs/hadoop3/apache-hive-1.2.2-src.tar.gz
mkdir -p /opt/tgz/spark-2.4.0-src/sql/hive-thriftserver/src/main/java/org/apache/hadoop/hive/shims/
cp /opt/tgz/apache-hive-1.2.2-src/shims/common/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java /opt/tgz/spark-2.4.0-src/sql/hive-thriftserver/src/main/java/org/apache/hadoop/hive/shims/

修改源文件，解决spark编译R组件时，抛出的异常信息：Unrecognized Hadoop major version number: 3.1.4
vim /app/spark-2.4.7-src/sql/hive-thriftserver/src/main/java/org/apache/hadoop/hive/shims/ShimLoader.java
注释掉：throw new IllegalArgumentException("Unrecognized Hadoop major version number: " + vers);
添加：return HADOOP23VERSIONNAME;

find /app/maven/repository/ -name "*.lastUpdated" | xargs rm -rf
export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=1024M -XX:ReservedCodeCacheSize=1024m"

mvn clean package -DskipTests -Pspark-ganglia-lgpl -Pkinesis-asl -Pdocker-integration-tests -Pyarn -Pmesos -Pkubernetes -Phadoop-3.2 -Dhadoop.version=3.2.1 -Phive-2.3 -Phive-thriftserver -Phadoop-cloud -Pscala-2.12 -Dscala.version=2.12.12 -Dscala.binary.version=2.12 -Dzookeeper.version=3.4.14 -Dcurator.version=4.0.0 -Dhive.version=2.3.7 -Dhive.version.short=2.3 -Dparquet.version=1.10.1 -Dsnappy.version=1.1.7.5 -e -X

尝试提高hive版本：失败失败失败
mvn clean package -DskipTests -Pspark-ganglia-lgpl -Pkinesis-asl -Pdocker-integration-tests -Pyarn -Pmesos -Pkubernetes -Phadoop-3.2 -Dhadoop.version=3.3.0 -Phive-2.3 -Phive-thriftserver -Phadoop-cloud -Pscala-2.12 -Dscala.version=2.12.12 -Dscala.binary.version=2.12 -Dzookeeper.version=3.4.14 -Dcurator.version=4.0.0 -Dhive.version=3.1.2 -Dhive.version.short=3.1 -Dhive23.version=3.1.2 -Dparquet.version=1.10.1 -Dsnappy.version=1.1.7.5 -e -X -rf :spark-sql_2.12

spark-3.1.1 编译
修改pom.xml第2527行为：<version>1.7.1</version> 注：为能找到对应版本的scala
<compilerPlugin>
<groupId>com.github.ghik</groupId>
<artifactId>silencer-plugin_${scala.version}</artifactId>
<version>1.7.1</version>
</compilerPlugin>

mvn clean package -DskipTests -Pspark-ganglia-lgpl -Pkinesis-asl -Pdocker-integration-tests -Pyarn -Pmesos -Pkubernetes -Phadoop-3.2 -Dhadoop.version=3.2.2 -Phive-2.3 -Phive-thriftserver -Phadoop-cloud -Pscala-2.12 -Dscala.version=2.12.13 -Dscala.binary.version=2.12 -Dzookeeper.version=3.4.14 -Dcurator.version=4.0.0 -Dhive.version=2.3.7 -Dhive.version.short=2.3 -Dparquet.version=1.10.1 -Dsnappy.version=1.1.8.2 -e -X

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.1.1:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 2.172 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 4.482 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 4.770 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 1.338 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 2.535 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 1.374 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 5.691 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 1.587 s]
[INFO] Spark Project Core ................................. SUCCESS [01:37 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 20.003 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 26.456 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 48.988 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:59 min]
[INFO] Spark Project SQL .................................. SUCCESS [02:59 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:14 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 6.415 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:28 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 21.442 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 8.676 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 45.861 s]
[INFO] Spark Project Mesos ................................ SUCCESS [ 41.535 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [ 47.041 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 45.401 s]
[INFO] Spark Ganglia Integration .......................... SUCCESS [ 7.929 s]
[INFO] Spark Project Hadoop Cloud Integration ............. SUCCESS [ 14.728 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 3.378 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 27.894 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 40.072 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [01:04 min]
[INFO] Spark Kinesis Integration .......................... SUCCESS [ 39.486 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 58.174 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 8.733 s]
[INFO] Spark Avro ......................................... SUCCESS [02:00 min]
[INFO] Spark Project Kinesis Assembly ..................... SUCCESS [ 11.400 s]
[INFO] Spark Project Docker Integration Tests ............. SUCCESS [ 53.694 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 23:26 min
[INFO] Finished at: 2021-03-17T14:38:27+08:00
[INFO] ------------------------------------------------------------------------

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 2.4.7:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 2.049 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 3.396 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 4.576 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 2.305 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 4.019 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 2.360 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 6.546 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 3.679 s]
[INFO] Spark Project Core ................................. SUCCESS [01:51 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 18.470 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 28.845 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 52.508 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:41 min]
[INFO] Spark Project SQL .................................. SUCCESS [02:53 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:00 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 4.613 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:16 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 24.116 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 5.047 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 34.315 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 28.167 s]
[INFO] Spark Ganglia Integration .......................... SUCCESS [ 5.114 s]
[INFO] Spark Project Cloud Integration through Hadoop Libraries SUCCESS [ 3.050 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 3.342 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 26.117 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 44.272 s]
[INFO] Spark Kinesis Integration .......................... SUCCESS [ 25.876 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 27.120 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 4.011 s]
[INFO] Spark Avro ......................................... SUCCESS [ 25.568 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [ 11.860 s]
[INFO] Spark Project External Flume ....................... SUCCESS [ 25.069 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [ 1.999 s]
[INFO] Spark Project Kinesis Assembly ..................... SUCCESS [ 7.634 s]
[INFO] Spark Project Docker Integration Tests ............. SUCCESS [ 24.186 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 17:24 min
[INFO] Finished at: 2020-11-22T02:57:00+08:00
[INFO] ------------------------------------------------------------------------

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 3.0.1:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [ 2.103 s]
[INFO] Spark Project Tags ................................. SUCCESS [ 4.359 s]
[INFO] Spark Project Sketch ............................... SUCCESS [ 4.464 s]
[INFO] Spark Project Local DB ............................. SUCCESS [ 1.253 s]
[INFO] Spark Project Networking ........................... SUCCESS [ 2.512 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 1.065 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 5.686 s]
[INFO] Spark Project Launcher ............................. SUCCESS [ 1.623 s]
[INFO] Spark Project Core ................................. SUCCESS [01:27 min]
[INFO] Spark Project ML Local Library ..................... SUCCESS [ 19.539 s]
[INFO] Spark Project GraphX ............................... SUCCESS [ 25.161 s]
[INFO] Spark Project Streaming ............................ SUCCESS [ 43.714 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [01:45 min]
[INFO] Spark Project SQL .................................. SUCCESS [02:45 min]
[INFO] Spark Project ML Library ........................... SUCCESS [02:11 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 6.863 s]
[INFO] Spark Project Hive ................................. SUCCESS [01:25 min]
[INFO] Spark Project REPL ................................. SUCCESS [ 21.531 s]
[INFO] Spark Project YARN Shuffle Service ................. SUCCESS [ 7.448 s]
[INFO] Spark Project YARN ................................. SUCCESS [ 47.056 s]
[INFO] Spark Project Mesos ................................ SUCCESS [ 43.106 s]
[INFO] Spark Project Kubernetes ........................... SUCCESS [ 42.560 s]
[INFO] Spark Project Hive Thrift Server ................... SUCCESS [ 47.606 s]
[INFO] Spark Ganglia Integration .......................... SUCCESS [ 6.879 s]
[INFO] Spark Project Hadoop Cloud Integration ............. SUCCESS [ 15.315 s]
[INFO] Spark Project Assembly ............................. SUCCESS [ 2.924 s]
[INFO] Kafka 0.10+ Token Provider for Streaming ........... SUCCESS [ 23.821 s]
[INFO] Spark Integration for Kafka 0.10 ................... SUCCESS [ 33.652 s]
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SUCCESS [ 53.042 s]
[INFO] Spark Kinesis Integration .......................... SUCCESS [ 34.093 s]
[INFO] Spark Project Examples ............................. SUCCESS [ 39.712 s]
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SUCCESS [ 7.335 s]
[INFO] Spark Avro ......................................... SUCCESS [01:23 min]
[INFO] Spark Project Kinesis Assembly ..................... SUCCESS [ 11.745 s]
[INFO] Spark Project Docker Integration Tests ............. SUCCESS [ 40.359 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20:56 min
[INFO] Finished at: 2020-10-21T15:52:33+08:00
[INFO] ------------------------------------------------------------------------

yum install r-base pandoc pandoc-citeproc curl
R -e "install.packages(c('pdflatex','curl', 'gh', 'httr', 'usethis', 'xml2','knitr', 'rmarkdown', 'devtools', 'testthat', 'e1071', 'survival', 'roxygen2'), repos='http://cran.us.r-project.org')"
R CMD INSTALL /mnt/hgfs/Deep\ Learning/hadoop3.3/latexpdf_0.1.6.tar.gz

R/install-dev.sh

vim R/pkg/vignettes/sparkr-vignettes.Rmd

替换第80行：sparkR.session(master = "local[1]", sparkConfig = sparkSessionConfig, enableHiveSupport = FALSE) 为：
sparkR.session(master = "local[1]", enableHiveSupport = FALSE)
如果后面打包命令至此还是报错，直接把该文件内容删掉，创建一个空文件略过其中的脚本

./dev/make-distribution.sh --pip --r --tgz --mvn mvn -Psparkr -Pspark-ganglia-lgpl -Pkinesis-asl -Pdocker-integration-tests -Pyarn -Pmesos -Pkubernetes -Phadoop-3.2 -Dhadoop.version=3.2.2 -Phive-2.3 -Phive-thriftserver -Phadoop-cloud -Pscala-2.12 -Dscala.version=2.12.13 -Dscala.binary.version=2.12 -Dzookeeper.version=3.4.14 -Dcurator.version=4.0.0 -Dhive.version=2.3.7 -Dhive.version.short=2.3 -Dparquet.version=1.10.1 -Dsnappy.version=1.1.8.2 -e -X

编译结果位于：当前目录下：./spark-3.0.1-bin-3.2.1.tgz

配置spark
cd /app
tar zxf spark-3.0.1-src/spark-3.0.1-bin-3.2.1.tgz

ln -sf spark-3.0.1-bin-3.2.1 spark

cp /app/hive/conf/hive-site.xml /app/spark/conf/
cp /app/hive/conf/hive-log4j2.properties /app/spark/conf/
cp /app/spark/conf/log4j.properties.template /app/spark/conf/log4j.properties
cp /app/spark/conf/spark-env.sh.template /app/spark/conf/spark-env.sh
cp /app/hive/lib/mysql-connector-java-5.1.48-bin.jar /app/spark/jars
cp /app/hive/lib/hive-hbase-handler-3.1.2.jar /app/spark/jars/
cp /app/hbase/lib/hbase*.jar /app/spark/jars/
cp /app/hbase/lib/metrics-core-3.2.6.jar /app/spark/jars/

cp /app/hadoop/share/hadoop/common/lib/guava-27.0-jre.jar /app/spark/jars/
mv /app/spark/jars/guava-14.0.1.jar /app/spark/jars/guava-14.0.1.jar.bak

vim /app/spark/conf/spark-env.sh
export HIVE_HOME=/app/hive

cd /app/spark/conf/
cp slaves.template slaves
vim slaves

cp workers.template workers
vim workers

vim log4j.properties

hue 编译安装
参考：
https://blog.csdn.net/Colar_/article/details/108535554
https://docs.gethue.com/administrator/installation/dependencies/
https://blog.csdn.net/qq_36525906/article/details/101528128?utm_medium=distribute.pc_relevant_download.none-task-blog-blogcommendfrombaidu-2.nonecase&depth_1-utm_source=distribute.pc_relevant_download.none-task-blog-blogcommendfrombaidu-2.nonecas
https://blog.csdn.net/qq_44418841/article/details/108079870

yum install libtidy ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel -y
yum -y install gcc gcc-c++ libstdc++-devel
yum -y install ant asciidoc cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-plain gcc gcc-c++ krb5-devel libffi-devel libxml2-devel libxslt-devel make mysql mysql-devel openldap-devel python-devel sqlite-devel gmp-devel rsync
yum install -y centos-release-SCL scl-utils nodejs

# If you are using Python 3.5+, set PYTHON_VER before the build, like

export PYTHON_VER=python3.8

sudo ln -sf /app/anaconda3/include/python3.8 /usr/include/python3.8
sudo ln -sf /app/anaconda3/bin/python3.8 /usr/bin/python3.8

mkdir /data/hue
chown -R tekken:tekken /data/hue

ln -sf /data/hue

切换到中文版本
vim desktop/core/src/desktop/settings.py
LANGUAGE_CODE = 'zh_CN'

LANGUAGES = [
('de', _('German')),
('en-us', _('English')),
('es', _('Spanish')),
('fr', _('French')),
('ja', _('Japanese')),
('ko', _('Korean')),
('pt', _('Portuguese')),
('pt_BR', _('Brazilian Portuguese')),
# 注意这里原版是zh-CN，修改为zh_CN
('zh_CN', _('Simplified Chinese')),
]

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

pandas 1.0.5 requires python-dateutil>=2.6.1, but you'll have python-dateutil 2.4.2 which is incompatible.
pandas 1.0.5 requires pytz>=2017.2, but you'll have pytz 2015.2 which is incompatible.
gevent 20.6.2 requires greenlet>=0.4.16; platform_python_implementation == "CPython", but you'll have greenlet 0.4.15 which is incompatible.
distributed 2.20.0 requires msgpack>=0.6.0, but you'll have msgpack 0.5.6 which is incompatible.
anaconda-client 1.7.2 requires python-dateutil>=2.6.1, but you'll have python-dateutil 2.4.2 which is incompatible.
daphne 2.5.0 requires asgiref~=3.2, but you'll have asgiref 2.3.2 which is incompatible.
django-timezone-field 4.0 requires django>=2.2, but you'll have django 1.11.28 which is incompatible.
spyder 4.1.4 requires pyqt5<5.13; python_version >= "3", which is not installed.
spyder 4.1.4 requires pyqtwebengine<5.13; python_version >= "3", which is not installed.
jupyter-console 6.1.0 requires prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0, but you'll have prompt-toolkit 1.0.18 which is incompatible.
flask 1.1.2 requires Werkzeug>=0.15, but you'll have werkzeug 0.14.1 which is incompatible.

make clean

make locales
make apps

执行以下修改：
https://blog.csdn.net/Colar_/article/details/108535554

find ./ -name "runserver.py"

修改以下py中 127.0.0.1 为 0.0.0.0
vim ./desktop/core/ext-py/Django-1.11.29/django/core/management/commands/runserver.py
vim ./build/env/lib/python3.8/site-packages/django/core/management/commands/runserver.py

mysql> create database hue DEFAULT CHARACTER SET utf8 COLLATE utf8_general_ci;

vim ./desktop/conf/pseudo-distributed.ini

[[database]]
# Database engine is typically one of:
# postgresql_psycopg2, mysql, sqlite3 or oracle.
#
# Note that for sqlite3, 'name', below is a path to the filename. For other backends, it is the database name.
# Note for Oracle, options={"threaded":true} must be set in order to avoid crashes.
# Note for Oracle, you can use the Oracle Service Name by setting "host=" and "port=" and then "name=<host>:<port>/<service_name>".
# Note for MariaDB use the 'mysql' engine.
engine=mysql
host=spark
port=3306
user=root
password=spark123
name=hue

配置中文环境：
vim ./desktop/core/src/desktop/settings.py
LANGUAGE_CODE = 'zh_CN'

LANGUAGES = [
('de', _('German')),
('en-us', _('English')),
('es', _('Spanish')),
('fr', _('French')),
('ja', _('Japanese')),
('ko', _('Korean')),
('pt', _('Portuguese')),
('pt_BR', _('Brazilian Portuguese')),
('zh_CN', _('Simplified Chinese')),
]

重新编译
make app

配置启动用户参数：
vim ./desktop/conf/pseudo-distributed.ini

# Webserver runs as this user
server_user=tekken
server_group=tekken

./build/env/bin/hue syncdb
./build/env/bin/hue migrate

./build/env/bin/hue runserver

apache livy 编译
https://blog.csdn.net/qq_45437546/article/details/104725315

find /app/maven/repository/ -name "*.lastUpdated" | xargs rm -rf
export MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=1024M -XX:ReservedCodeCacheSize=1024m"

官网下载的源码编译：只支持scala 2.11
mvn -DskipTests clean package -Pthriftserver,spark-2.4  -Dspark.scala-2.11.version=2.4.7 -Dspark.bin.download.url=https://downloads.apache.org/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz -Dspark.bin.name=spark-2.4.7-bin-hadoop2.7
/app/apache-livy-0.7.0-incubating/assembly/target/apache-livy-0.7.0-incubating-bin.zip

使用github上最新仓库，以支持scala2.12 和spark3.0
https://github.com/apache/incubator-livy

unzip /mnt/hgfs/Deep\ Learning/hadoop3.3/incubator-livy-master.zip

cd incubator-livy-master

在 dependency 中注释掉2.11相关项
vim coverage/pom.xml
vim assembly/pom.xml

mvn -DskipTests clean package -Pthriftserver,spark-3.0  -Dscala.version=2.12.12 -Dscala.binary.version=2.12 -Dscala-2.12.version=2.12.12 -Dspark.version=3.0.1 -Dspark.scala-2.12.version=3.0.1 -Dspark.bin.download.url=https://mirror.bit.edu.cn/apache/spark/spark-3.0.1/spark-3.0.1-bin-hadoop3.2.tgz -Dspark.bin.name=spark-3.0.1-bin-hadoop3.2

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Livy Project Parent POM 0.8.0-incubating-SNAPSHOT:
[INFO]
[INFO] Livy Project Parent POM ............................ SUCCESS [ 1.645 s]
[INFO] livy-api ........................................... SUCCESS [ 3.727 s]
[INFO] livy-client-common ................................. SUCCESS [ 0.726 s]
[INFO] livy-test-lib ...................................... SUCCESS [ 3.627 s]
[INFO] multi-scala-project-root ........................... SUCCESS [ 0.105 s]
[INFO] livy-core-parent ................................... SUCCESS [ 0.150 s]
[INFO] livy-core_2.12 ..................................... SUCCESS [ 7.691 s]
[INFO] livy-rsc ........................................... SUCCESS [ 2.722 s]
[INFO] livy-repl-parent ................................... SUCCESS [ 1.663 s]
[INFO] livy-repl_2.12 ..................................... SUCCESS [ 20.211 s]
[INFO] livy-server ........................................ SUCCESS [ 30.653 s]
[INFO] livy-thriftserver-session .......................... SUCCESS [ 1.152 s]
[INFO] livy-thriftserver .................................. SUCCESS [ 30.671 s]
[INFO] livy-assembly ...................................... SUCCESS [ 1.496 s]
[INFO] livy-client-http ................................... SUCCESS [ 8.776 s]
[INFO] livy-scala-api-parent .............................. SUCCESS [ 0.426 s]
[INFO] livy-scala-api_2.12 ................................ SUCCESS [ 11.903 s]
[INFO] livy-integration-test .............................. SUCCESS [ 30.876 s]
[INFO] livy-coverage-report ............................... SUCCESS [ 1.629 s]
[INFO] livy-examples ...................................... SUCCESS [ 16.100 s]
[INFO] livy-python-api .................................... SUCCESS [ 3.113 s]
[INFO] livy-beeline ....................................... SUCCESS [ 1.983 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:01 min
[INFO] Finished at: 2020-10-29T17:57:51+08:00
[INFO] ------------------------------------------------------------------------

编译结果位于：
/app/incubator-livy-master/apache-livy-0.8.0-incubating-SNAPSHOT-bin

修改配置文件：

          **①在livy-env中添加**
            export SPARK_HOME=Spark安装目录
            export HADOOP_CONF_DIR=Hadoop配置目录
          **②livy.conf中可以进行一些配置**
            # 配置Livy会话所使用的Spark集群运行模式
            livy.spark.master = yarn
            # 配置Livy会话所使用的Spark集群部署模式
            livy.spark.deploy-mode = cluster
            //默认使用hiveContext
            livy.repl.enable-hive-context = true
            //开启用户代理
            livy.impersonation.enabled = true
            //设置session空闲过期时间
            livy.server.session.timeout = 1h

解决scala 2.12环境下，找不到相关jar包错误
cd /app/livy/repl_2.11-jars
ln -sf repl_2.11-jars repl_2.12-jars
cp jars/livy-* /app/spark/jars

启动前提：启动zookeeper集群，hadoop集群
livy-server start
访问：http://spark:8998/ui

①session的创建需求：使用livy-session方式，创建一个名为spark的session。
curl -XPOST 'http://spark:8998/sessions' -H "Content-Type:application/json" --data '{"kind":"spark"}'

返回：
{"id":0,"name":null,"appId":null,"owner":null,"proxyUser":null,"state":"starting","kind":"spark","appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":["stdout: ","\nstderr: ","\nYARN Diagnostics: "]}

②session查看
http://spark:8998/ui

③使用livy session，计算hdfs上根目录下的文件hello.txt中每个单词出现的总次数，并将结果落地到hdfs指定的目录下
curl -XPOST 'http://spark:8998/sessions/0/statements' -H 'Content-Type:application/json' -d '{"code":"sc.textFile(\"/tmp/LICENSE\").flatMap(_.split(\" \")).map((_,1)).reduceByKey(_+_).saveAsTextFile(\"/tmp/wc\")"}'
返回：
{"id":0,"code":"sc.textFile(\"/tmp/LICENSE\").flatMap(_.split(\" \")).map((_,1)).reduceByKey(_+_).saveAsTextFile(\"/tmp/wc\")","state":"waiting","output":null,"progress":0.0,"started":0,"completed":0}

④删除session
curl -XDELETE http://spark:8998/sessions/0
{"msg":"deleted"}

2、通过使用livy-batches，可以通过rest来执行spark-submit，用于处理非交互式的请求。
①使用livy batches，运行官方案例之求圆周率PI的值。
在Linux命令行下提交如下语句：
前提：需要将计算（jar包）上传到hdfs指定的目录下。

curl -XPOST -H "Content-Type: application/json" http://JANSON01:8998/batches --data '{
"conf": {"spark.master":"yarn-cluster"}, "file": "hdfs://ns1/myjars/spark-examples_2.11-2.3.0.jar",
"className": "org.apache.spark.examples.SparkPi", "name": "Livy Pi Example",
"executorCores":1, "executorMemory":"512m", "driverCores":1, "driverMemory":"512m",
"queue":"default", "args":["100"]}'

②使用livy batches，计算hdfs上根目录下的文件hello.txt中每个单词出现的次数，并将结果落地到hdfs指定的目录下
在Linux终端下键入执行的指令：
前提：书写工程，打包成jar，并将计算（jar包）上传到hdfs指定的目录下。

curl -XPOST -H "Content-Type: application/json" http://JANSON01:8998/batches --data '{
"conf": {"spark.master":"yarn-cluster"}, "file": "hdfs://ns1/myjars/my-word-count.jar",
"className": "com.l000phone.bigdata.MyWordCount", "name": "Janson Word Count",
"executorCores":1, "executorMemory":"512m", "driverCores":1, "driverMemory":"512m",
"queue":"default", "args":["hdfs://ns1/hello.txt","hdfs://ns1/livy-batches-result"]}'

oozie 5.2.0 编译部署
find /app/maven/repository/ -name "*.lastUpdated" | xargs rm -rf
export MAVEN_OPTS="-Xms256m -Xmx4096m"

失败：
./bin/mkdistro.sh -Pspark-2 -Dhadoop.version=3.3.0 -Dhadoop.majorversion=3 -Dhbase.version=1.6.0 -Dhive.version=2.3.7 -Dpig.version=0.16.0 -Dsqoop.version=1.4.7 -Dspark.version=3.0.1 -Dspark.streaming.kafka.version=1.6.3 -Dspark.bagel.version=1.6.3 -Dspark.scala.binary.version=2.12 -Djackson.version=2.11.3 -DskipTests -e -X

成功：

修改 vim sharelib/spark/pom.xml ，修改spark-streaming-kafka 对应的scala版本为2.11 或变量：spark.scala.binary.kafka.version；
修改 spark-bagel 对应的scala版本为 2.11 或变量 spark.scala.binary.bagel.version

修改src/main/assemblies/distro.xml，注释掉其中的 Oozie core test jar 引用
vim src/main/assemblies/distro.xml


./bin/mkdistro.sh -Dmaven.test.skip=true -Pspark-2 -Dhadoop.version=3.3.0 -Dhadoop.majorversion=3 -Dhbase.version=1.6.0 -Dhive.version=2.3.7 -Dpig.version=0.16.0 -Dsqoop.version=1.4.7 -Dspark.version=2.4.7 -Dspark.streaming.kafka.version=1.6.3 -Dspark.bagel.version=1.6.3 -Dspark.scala.binary.version=2.12 -Dspark.scala.binary.kafka.version=2.11 -Dspark.scala.binary.bagel.version=2.11 -Djackson.version=2.11.3 -DskipTests -e -X

编译结果位于：/app/oozie-5.2.0/distro/target/oozie-5.2.0-distro.tar.gz

INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Oozie Main 5.2.0:
[INFO]
[INFO] Apache Oozie Main .................................. SUCCESS [ 1.012 s]
[INFO] Apache Oozie Fluent Job ............................ SUCCESS [ 0.067 s]
[INFO] Apache Oozie Fluent Job API ........................ SUCCESS [ 5.368 s]
[INFO] Apache Oozie Client ................................ SUCCESS [ 4.014 s]
[INFO] Apache Oozie Share Lib Oozie ....................... SUCCESS [ 2.111 s]
[INFO] Apache Oozie Share Lib HCatalog .................... SUCCESS [ 5.257 s]
[INFO] Apache Oozie Share Lib Distcp ...................... SUCCESS [ 0.597 s]
[INFO] Apache Oozie Core .................................. SUCCESS [ 7.763 s]
[INFO] Apache Oozie Share Lib Streaming ................... SUCCESS [ 3.003 s]
[INFO] Apache Oozie Share Lib Pig ......................... SUCCESS [ 5.341 s]
[INFO] Apache Oozie Share Lib Git ......................... SUCCESS [ 2.500 s]
[INFO] Apache Oozie Share Lib Hive ........................ SUCCESS [ 3.524 s]
[INFO] Apache Oozie Share Lib Hive 2 ...................... SUCCESS [ 3.704 s]
[INFO] Apache Oozie Share Lib Sqoop ....................... SUCCESS [ 2.426 s]
[INFO] Apache Oozie Examples .............................. SUCCESS [ 4.049 s]
[INFO] Apache Oozie Share Lib Spark ....................... SUCCESS [ 4.685 s]
[INFO] Apache Oozie Share Lib ............................. SUCCESS [ 20.230 s]
[INFO] Apache Oozie Docs .................................. SUCCESS [ 0.315 s]
[INFO] Apache Oozie WebApp ................................ SUCCESS [ 4.515 s]
[INFO] Apache Oozie Tools ................................. SUCCESS [ 2.161 s]
[INFO] Apache Oozie MiniOozie ............................. SUCCESS [ 1.991 s]
[INFO] Apache Oozie Fluent Job Client ..................... SUCCESS [ 1.554 s]
[INFO] Apache Oozie Server ................................ SUCCESS [ 3.186 s]
[INFO] Apache Oozie Distro ................................ SUCCESS [ 34.714 s]
[INFO] Apache Oozie ZooKeeper Security Tests .............. SUCCESS [ 2.289 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:07 min
[INFO] Finished at: 2020-10-30T14:32:53+08:00
[INFO] ------------------------------------------------------------------------

Oozie distro created, DATE[2020.10.30-06:30:45GMT] VC-REV[unavailable], available at [/app/oozie-5.2.0/distro/target]

oozie 5.2.0 部署
https://www.jianshu.com/p/6e2f6b4517f0

mv oozie-5.2.0 oozie-5.2.0-src
tar zxf oozie-5.2.0-src/distro/target/oozie-5.2.0-distro.tar.gz
ln -sf oozie-5.2.0 oozie

cd ${OOZIE_HOME}
mkdir libext

wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip

mv ext-2.2.zip libext
cp /app/hive/lib/mysql-connector-java-5.1.48-bin.jar libext/

cd /app/hadoop/
find -name *.jar |xargs -t -i cp {} /app/oozie/libext
rm -rf /app/oozie/libext/hsqldb-2.3.4.jar

cd /app/oozie
cp /mnt/hgfs/hadoop3/ext-2.2.zip libext/
cp /mnt/hgfs/hadoop3/mysql-connector-java-5.1.48-bin.jar libext/
cp /mnt/hgfs/hadoop3/ojdbc7-12.1.0.2.jar libext/

解压tar文件

tar zxf oozie-sharelib-5.2.0.tar.gz
tar zxf oozie-examples.tar.gz
tar zxf oozie-client-5.2.0.tar.gz

将share上传到hdfs上参考：./bin/oozie-setup.sh sharelib create -fs hdfs://spark:9000 -locallib oozie-sharelib-5.2.0.tar.gz
执行：
(base) [tekken@spark oozie]$ bin/oozie-setup.sh sharelib create -fs hdfs://spark:9000
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/app/oozie-5.2.0/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/app/oozie-5.2.0/lib/slf4j-log4j12-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/app/oozie-5.2.0/libext/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Found Hadoop that supports Erasure Coding. Trying to disable Erasure Coding for path: hdfs://spark:9000/user/oozie/share/lib
Done
the destination path for sharelib is: hdfs://spark:9000/user/oozie/share/lib/lib_20201030183446

修改oozie-site.xml
vim /app/oozie/conf/oozie-site.xml

<property>
<name>oozie.service.JPAService.jdbc.driver</name>
<value>com.mysql.jdbc.Driver</value>
<description>
JDBC driver class.
</description>
</property>
<property>
<name>oozie.service.JPAService.jdbc.url</name>
<value>jdbc:mysql://spark:3306/oozie?createDatabaseIfNotExist=true&useSSL=false&characterEncoding=UTF-8</value>
<description>
JDBC URL.
</description>
</property>
<property>
<name>oozie.service.JPAService.jdbc.username</name>
<value>root</value>
<description>
DB user name.
</description>
</property>
<property>
<name>oozie.service.JPAService.jdbc.password</name>
<value>spark123</value>
<description>
DB user password.
IMPORTANT: if password is emtpy leave a 1 space string, the service trims the value,
if empty Configuration assumes it is NULL.
</description>
</property>

<property>
<name>oozie.service.HadoopAccessorService.hadoop.configurations</name>
<value>*=/app/hadoop/etc/hadoop</value>
<description>
Comma separated AUTHORITY=HADOOP_CONF_DIR, where AUTHORITY is the HOST:PORT of
the Hadoop service (JobTracker, YARN, HDFS). The wildcard '*' configuration is
used when there is no exact match for an authority. The HADOOP_CONF_DIR contains
the relevant Hadoop *-site.xml files. If the path is relative is looked within
the Oozie configuration directory; though the path can be absolute (i.e. to point
to Hadoop client conf/ directories in the local filesystem.
</description>
</property>

<property>
<name>oozie.service.SparkConfigurationService.spark.configurations</name>
<value>*=/app/spark/conf</value>
<description>
Comma separated AUTHORITY=SPARK_CONF_DIR, where AUTHORITY is the HOST:PORT of
the ResourceManager of a YARN cluster. The wildcard '*' configuration is
used when there is no exact match for an authority. The SPARK_CONF_DIR contains
the relevant spark-defaults.conf properties file. If the path is relative is looked within
the Oozie configuration directory; though the path can be absolute. This is only used
when the Spark master is set to either "yarn-client" or "yarn-cluster".
</description>
</property>

<property>
<name>oozie.service.WorkflowAppService.system.libpath</name>
<value>hdfs://spark:9000/user/oozie/share/lib/lib_20201030183446</value>
<description>
System library path to use for workflow applications.
This path is added to workflow application if their job properties sets
the property 'oozie.use.system.libpath' to true.
</description>
</property>

./bin/oozie-setup.sh db create -run -sqlfile oozie.sql

启动：
oozied.sh start

http://spark:11000

#Hadoop#

Hadoop数据仓库文章被收录于专栏

Hadoop数据仓库是建立在Hadoop生态系统基础上的大数据存储和处理解决方案。它可以用于将结构化、半结构化和非结构化的数据集中存储，并提供高性能的数据查询、分析和数据处理功能。

hadoop3.3源码编译集群部署

全站热榜