0.规划
两个节点：
| ip | 部署的程序 | 备注 |
| 192.168.56.2/bigdata.lzf | namenode,datanode,NodeManager，hive,presto,mysql,hive-metastore,presto-cli | 主节点 |
| 192.168.56.3/bigdata.dn1.lzf | secondarynode,resourceManager,NodeManager,hive,presto,presto-cli | 资源管理节点 |
hive由于不存在主从的问题，虽然metastore也可以部署多个。
如果有必要resourceManager也可以部署多个。
--
1.公共操作
1.1 创建用户
group add hadoop
useradd hadoop -G hadoop
1.2 设置ssh互通
使用rsa的互通
1.2.1 分别在2,3机器上执行如下（hadoop用户下)
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 0600 ~/.ssh/authorized_keys1.2.2 把3的id_rsa.pub 复制到 2的id_rsa.pub_sl
ssh bigdata.dn1.lzf
cd .ssh
scp id_rsa.pub hadoop@bigdata.lzf:~/.ssh/id_rsa.pub_sl
1.2.3 把2的 id_rsa.pub_sl 追加都2的authorized_keys
cd .ssh
cat id_rsa.pub_sl>> authorized_keys
1.2.4 把2的authorized_keys复制到到3的/home/hadoop/.ssh/下
之后进行验证：
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
在2下执行ssh bigdata.dn1.lzf
在3下执行ssh bigdata.lzf
成功，则可以继续了。
注：
a.如果有更多的节点，操作时类似的，就是把所有的id_rsa.pub合并在一起，然后追加到主机的 authorized_keys ,最后把主机authorized_keys复制到各个节点上即可。
b.至于使用rsa还是dsa看具体情况。
1.3 目录
以hadoop身份执行
mkdir -p /home/hadoop/data_hadoop/hdfs/name
mkdir -p /home/hadoop/data_hadoop/hdfs/data
mkdir -p /home/hadoop/data_hadoop/tmp
1.4 安装软件
在/home/haoop下安装
hadoop-2.8.0
1.5 环境变量配置
export JAVA_HOME=/usr/local/jdk1.8.0_131
export HADOOP_HOME=/home/hadoop/hadoop-2.8.0
export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.8.0/etc/hadoop
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_PREFIX=$HADOOP_HOME
export PATH=$PATH:$HOME/.local/bin:$HOME/bin:$JAVA_HOME/bin:$HADOOP_HOME/bin
1.6 修改各个env.sh和日志
根据单机的配置即可。
修改 etc/hadoop/log4j.properties 添加如下：
log4j.logger.org.apache.hadoop.util.NativeCodeLoader=DEBUG
2.节点配置
2.1 192.168.56.2 配置
core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://bigdata.lzf:9001</value>
<description>HDFS的URI，文件系统://namenode标识:端口号,默认是9000</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/data_hadoop/tmp</value>
<description>namenode上本地的hadoop临时文件夹</description>
</property>
<property>
<name>ipc.client.connect.max.retries</name>
<value>100</value>
<description>默认10次，现在配置100次</description>
</property>
<property>
<name>ipc.client.connect.retry.interval</name>
<value>10000</value>
<description>连接间隔1秒钟，默认是0.1秒</description>
</property>
--为了权限，例如通过beeline之类的通过访问
添加以下内容：
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
hdfs-site.xml
<property>
<name> dfs.namenode.name.dir</name>
<value>/home/hadoop/data_hadoop/hdfs/name</value>
<description>namenode上存储hdfs名字空间元数据 </description>
</property>
<property>
<name> dfs.datanode.data.dir</name>
<value>/home/hadoop/data_hadoop/hdfs/data</value>
<description>datanode上数据块的物理存储位置</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>副本个数，配置默认是3,应小于datanode机器数量</description>
</property>
<property>
<name>dfs.namenode.rpc-address</name>
<value>bigdata.lzf:9001</value>
<description>RPC address that handles all clients requests。有人说需要和fs.defaultFS 一样端口</description>
</property>
<property> <name>dfs.namenode.http-address</name> <value>bigdata.lzf:50070</value> <description> The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>bigdata.dn1.lzf:50090</value> </property>
slaves
#有两个节点
bigdata.lzf
bigdata.dn1.lzf
yarn-site.xml
#只是用于nodemanager
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
2.2 .192.168.56.3 配置
core-site.xml
<property>
<name>fs.defaultFS</name>
<value>hdfs://bigdata.lzf:9001</value>
<description>HDFS的URI，文件系统://namenode标识:端口号,默认是9000</description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/data_hadoop/tmp</value>
<description>namenode上本地的hadoop临时文件夹</description>
</property>
<property>
<name>ipc.client.connect.max.retries</name>
<value>100</value>
<description>默认10次，现在配置100次</description>
</property>
<property>
<name>ipc.client.connect.retry.interval</name>
<value>10000</value>
<description>连接间隔1秒钟，默认是0.1秒</description>
</property>
--为了权限，例如通过beeline之类的通过访问
添加以下内容：
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
hdfs-site.xml
<property>
<name> dfs.namenode.name.dir</name>
<value>/home/hadoop/data_hadoop/hdfs/name</value>
<description>namenode上存储hdfs名字空间元数据 </description>
</property>
<property>
<name> dfs.datanode.data.dir</name>
<value>/home/hadoop/data_hadoop/hdfs/data</value>
<description>datanode上数据块的物理存储位置</description>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
<description>副本个数，配置默认是3,应小于datanode机器数量</description>
</property>
<property>
<name>dfs.namenode.rpc-address</name>
<value>bigdata.lzf:9001</value>
<description>RPC address that handles all clients requests。有人说需要和fs.defaultFS 一样端口</description>
</property>
<property> <name>dfs.namenode.http-address</name> <value>bigdata.lzf:50070</value> <description> The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port. </description> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>bigdata.dn1.lzf:50090</value> </property>
slaves
#有两个节点
bigdata.lzf
bigdata.dn1.lzf
yarn-site.xml
#resourceManager,nodemanager
<configuration>
<property> <name>yarn.resourcemanager.address</name> <value>bigdata.dn1.lzf:8032</value> <description>资源管理器地址</description> </property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>bigdata.dn1.lzf</value>
<description>资源管理器节点名称 </description>
</property>
<property>
<name>yarn.resourcemanager.scheduler.class</name>
<value> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
<description>调度器类 </description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>bigdata.dn1.lzf:8099</value>
<description>用于管理集群的资源，可以通过浏览器访问 </description>
</property>
<property>
<name>yarn.nodemanager.webapp.address</name>
<value>bigdata.dn1.lzf:8042</value>
<description>用于管理节点，可以通过浏览器访问 </description>
</property>
</configuration>
3.初始化
在2节点上执行
hdfs namenode -format
不需要执行hdfs secondarynamenode -format
4.启动
4.1启动dfs
在主节点2行执行start-dfs.sh即可：
[hadoop@bigdata sbin]$ ./start-dfs.sh
17/07/21 17:28:44 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... 17/07/21 17:28:44 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library Starting namenodes on [bigdata.lzf] bigdata.lzf: starting namenode, logging to /home/hadoop/hadoop-2.8.0/logs/hadoop-hadoop-namenode-bigdata.lzf.out bigdata.dn1.lzf: starting datanode, logging to /home/hadoop/hadoop-2.8.0/logs/hadoop-hadoop-datanode-bigdata.dn1.lzf.out bigdata.lzf: starting datanode, logging to /home/hadoop/hadoop-2.8.0/logs/hadoop-hadoop-datanode-bigdata.lzf.out Starting secondary namenodes [bigdata.dn1.lzf] bigdata.dn1.lzf: starting secondarynamenode, logging to /home/hadoop/hadoop-2.8.0/logs/hadoop-hadoop-secondarynamenode-bigdata.dn1.lzf.out 17/07/21 17:29:05 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... 17/07/21 17:29:05 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
然后可以在2,3上使用jps，分别可以看到以下内容
5620 Jps 5239 NameNode 5373 DataNode
---
4069 DataNode 4261 Jps 4167 SecondaryNameNode
4.2 启动yarn
然后在节点3上执行：start-yarn.sh
注意：3节点才是配置为资源管理器节点的，所以只能从3启动，而不是从2
译注: 也许以后的版本不需要如此了！
如果在2上执行start-yarn.sh也可以启动yarn集群，但rm就是2，而不是3了。
5.测试
创建目录
[hadoop@bigdata sbin]$ hadoop fs -mkdir /tmp 17/07/21 17:33:03 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... 17/07/21 17:33:03 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library [hadoop@bigdata sbin]$ hadoop fs -ls / 17/07/21 17:33:16 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... 17/07/21 17:33:16 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library Found 1 items drwxr-xr-x - hadoop supergroup 0 2017-07-21 17:33 /tmp
---
上传文件
[hadoop@bigdata sbin]$ hadoop fs -copyFromLocal -f start-dfs.sh hdfs://bigdata.lzf:9001/tmp 17/07/21 17:35:29 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... 17/07/21 17:35:29 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library
查看
这个是在3上执行的，前面创建目录和上传是在2执行的。
[hadoop@bigdata ~]$ hadoop fs -tail hdfs://bigdata.lzf:9001/tmp/start-dfs.sh17/07/21 17:41:18 DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library...17/07/21 17:41:18 DEBUG util.NativeCodeLoader: Loaded the native-hadoop library----------------------------------------# quorumjournal nodes (if any)SHARED_EDITS_DIR=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.namenode.shared.edits.dir 2>&-)case "$SHARED_EDITS_DIR" inqjournal://*) JOURNAL_NODES=$(echo "$SHARED_EDITS_DIR" | sed 's,qjournal://\([^/]*\)/.*,\1,g; s/;/ /g; s/:[0-9]*//g') echo "Starting journal nodes [$JOURNAL_NODES]" "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \ --config "$HADOOP_CONF_DIR" \ --hostnames "$JOURNAL_NODES" \ --script "$bin/hdfs" start journalnode ;;esac#---------------------------------------------------------# ZK Failover controllers, if auto-HA is enabledAUTOHA_ENABLED=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.ha.automatic-failover.enabled)if [ "$(echo "$AUTOHA_ENABLED" | tr A-Z a-z)" = "true" ]; then echo "Starting ZK Failover Controllers on NN hosts [$NAMENODES]" "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \ --config "$HADOOP_CONF_DIR" \ --hostnames "$NAMENODES" \ --script "$bin/hdfs" start zkfcfi# eof
http访问
50070 默认是访问集群中的数据节点
http://bigdata.lzf:50070/explorer.html#/tmp 可以直接访问 hdfs://bigdata.lzf:9001/tmp在的文件。
50075 用户访问具体的数据节点
http://bigdata.lzf:50075,http://bigdata.dn1.lzf:50075
hadoop 的 wordcount 测试
hadoop jar /home/hadoop/hadoop-2.8.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.0.jar grep hdfs://bigdata.lzf:9001/tmp/hadoop-hadoop-namenode-bigdata.lzf.log output '(lzf)'
17/07/24 11:44:04 INFO mapreduce.Job: Counters: 29 File System Counters FILE: Number of bytes read=604458 FILE: Number of bytes written=1252547 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1519982 HDFS: Number of bytes written=0 HDFS: Number of read operations=12 HDFS: Number of large read operations=0 HDFS: Number of write operations=5 Map-Reduce Framework Combine input records=0 Combine output records=0 Reduce input groups=0 Reduce shuffle bytes=0 Reduce input records=0 Reduce output records=0 Spilled Records=0 Shuffled Maps =0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=0 Total committed heap usage (bytes)=169222144 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Output Format Counters Bytes Written=0
根据测试，输出的内容会放在/user/hadoop/output目录下，事先不创建也没有关系。
6.遇到问题
6.1 ip地址变更的问题
由于其中的主节点原来是承担了所有的角色，而且ip地址和现有的也不同。
其次，重新格式化之后，并没有删除掉原来的元数据目录。
所以，只好手动删除掉
rm -Rf /home/hadoop/data_hadoop/hdfs/namerm -Rf /home/hadoop/data_hadoop/hdfs/datarm -Rf /home/hadoop/data_hadoop/hdfs/tmpmkdir -p /home/hadoop/data_hadoop/hdfs/namemkdir -p /home/hadoop/data_hadoop/hdfs/datamkdir -p /home/hadoop/data_hadoop/tmp
6.2 区分namenode和secondarynamenode 的关键
hdfs-site.xml中改配置如下
<property>
<name>dfs.namenode.http-address</name> <value>bigdata.lzf:50070</value> <description> The address and the base port where the dfs namenode web ui will listen on. If the port is 0 then the server will start on a free port. </description></property><property> <name>dfs.namenode.secondary.http-address</name> <value>bigdata.dn1.lzf:50090</value></property> 通过这个最简单的配置，了解hadoop中集群的简单工作原理。
后续的研究，包括：
1.增加数据节点
2.搭建ha-hadoop集群
3.搭建基于yarn的presto集群