1. 介绍

alluxio这种master-slave结构必然涉及到单点故障。所以配置master的容错是必须的。

如果alluxio还没装好请参考我的其他文章。

本文前提是你已经配好了HA的HDFS。

hadoop配置HA我没写文章，建议参考官方文档：HDFS High Availability

我们采用的实验配置如下:

hostname	role
a1(10.8.12.16)	alluxio-master,alluxio-worker,hadoop master,hadoop slave
a2(10.8.12.17)	alluxio-standby-master,alluxio-worker,hadoop master-standby,hadoop slave
a3(10.8.12.18)	alluxi-worker,hadoop-slave

注意，设定为master和standby master的要可以互相无密码登入哦~~~

2. 准备工作

这里假设你已经提前安装好了JDK、支持HA的HDFS以及ZK集群。

这里特别强调一点：请在配置zk集群的时候设置zoo.cfg的时候注意把initLimit设置的大点，默认是10，我设置成100.

如果不这样操作，当你alluxio集群中worker的内存比较大的时候，会无法启动worker，看worker.log有报错：

2016-09-21 16:57:24,174 WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-09-21 16:57:24,513 ERROR logger.type (MetricsConfig.java:loadConfigFile) - Error loading metrics configuration file.
2016-09-21 16:57:24,754 INFO  server.Server (Server.java:doStart) - jetty-7.x.y-SNAPSHOT
2016-09-21 16:57:24,769 INFO  handler.ContextHandler (ContextHandler.java:startContext) - started o.e.j.s.ServletContextHandler{/metrics/json,null}
2016-09-21 16:57:24,814 INFO  handler.ContextHandler (ContextHandler.java:startContext) - started o.e.j.w.WebAppContext{/,file:/home/appadmin/alluxio-1.2.0/core/server/src/main/webapp/},/home/appadmin/alluxio-1.2.0/core/server/src/main/webapp
2016-09-21 16:57:27,864 INFO  server.AbstractConnector (AbstractConnector.java:doStart) - Started SelectChannelConnector@0.0.0.0:30000
2016-09-21 16:57:27,864 INFO  logger.type (UIWebServer.java:startWebServer) - Alluxio Worker Web service started @ 0.0.0.0/0.0.0.0:30000
2016-09-21 16:57:27,865 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:27,869 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (0) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:27,919 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:27,920 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (1) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:27,970 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:27,970 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (2) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:28,021 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:28,021 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (3) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:28,171 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:28,173 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (4) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:28,723 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:28,723 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (5) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:29,724 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:29,724 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (6) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:30,725 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:30,725 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (7) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:31,725 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:31,726 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (8) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:32,726 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:32,726 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (9) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:33,726 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:33,727 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (10) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:34,727 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:34,727 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (11) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:35,728 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:35,728 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (12) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:36,728 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:36,729 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (13) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:37,729 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:37,729 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (14) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:38,729 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:38,730 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (15) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:39,730 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:39,731 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (16) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:40,731 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:40,731 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (17) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:41,732 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:41,732 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (18) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:42,732 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:42,732 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (19) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:43,733 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:43,733 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (20) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:44,733 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:44,734 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (21) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:45,734 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:45,735 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (22) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:46,735 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:46,735 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (23) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:47,736 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:47,736 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (24) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:48,736 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:48,737 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (25) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:49,737 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:49,737 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (26) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:50,738 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:50,738 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (27) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:51,738 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:51,739 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (28) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:52,739 INFO  logger.type (AbstractClient.java:connect) - Alluxio client (version 1.2.0) is trying to connect with BlockMasterWorker master @ /10.8.12.16:19998
2016-09-21 16:57:52,739 ERROR logger.type (AbstractClient.java:connect) - Failed to connect (29) to BlockMasterWorker master @ /10.8.12.16:19998 : java.net.ConnectException: Connection refused
2016-09-21 16:57:52,740 ERROR logger.type (BlockWorker.java:start) - Failed to get a worker id from block master
alluxio.exception.ConnectionFailedException: Failed to connect to BlockMasterWorker master @ /10.8.12.16:19998 after 29 attempts
    at alluxio.AbstractClient.connect(AbstractClient.java:186)
    at alluxio.AbstractClient.retryRPC(AbstractClient.java:291)
    at alluxio.worker.block.BlockMasterClient.getId(BlockMasterClient.java:109)
    at alluxio.worker.WorkerIdRegistry.registerWithBlockMaster(WorkerIdRegistry.java:60)
    at alluxio.worker.block.BlockWorker.start(BlockWorker.java:168)
    at alluxio.worker.AlluxioWorker.startWorkers(AlluxioWorker.java:354)
    at alluxio.worker.AlluxioWorker.start(AlluxioWorker.java:326)
    at alluxio.worker.AlluxioWorker.main(AlluxioWorker.java:84)
2016-09-21 16:57:52,741 ERROR logger.type (AlluxioWorker.java:main) - Uncaught exception while running Alluxio worker, stopping it and exiting.
java.lang.RuntimeException: alluxio.exception.ConnectionFailedException: Failed to connect to BlockMasterWorker master @ /10.8.12.16:19998 after 29 attempts
    at com.google.common.base.Throwables.propagate(Throwables.java:160)
    at alluxio.worker.block.BlockWorker.start(BlockWorker.java:171)
    at alluxio.worker.AlluxioWorker.startWorkers(AlluxioWorker.java:354)
    at alluxio.worker.AlluxioWorker.start(AlluxioWorker.java:326)
    at alluxio.worker.AlluxioWorker.main(AlluxioWorker.java:84)
Caused by: alluxio.exception.ConnectionFailedException: Failed to connect to BlockMasterWorker master @ /10.8.12.16:19998 after 29 attempts
    at alluxio.AbstractClient.connect(AbstractClient.java:186)
    at alluxio.AbstractClient.retryRPC(AbstractClient.java:291)
    at alluxio.worker.block.BlockMasterClient.getId(BlockMasterClient.java:109)
    at alluxio.worker.WorkerIdRegistry.registerWithBlockMaster(WorkerIdRegistry.java:60)
    at alluxio.worker.block.BlockWorker.start(BlockWorker.java:168)
    ... 3 more

查看zk上节点信息，也会发现相关的节点没有创建出来.查看zk日志，也会有错误，说connect refused的错误。

3. 添加hadoop和HDFS相关配置文件

将hadoop的配置文件core-site.xml和hdfs-site.xml拷贝至$ALLUXIO_HOME/conf下

4. client配置

客户端使用时需要添加额外的JAVA选项

alluxio.zookeeper.enabled=true
alluxio.zookeeper.address=[zookeeper_hostname1]:2181,[zookeeper_hostname2]:2181,[zookeeper_hostname3]:2181

5. 完整配置一览

5.1 alluxio-env.sh(master)

ALLUXIO_MASTER_HOSTNAME=${ALLUXIO_MASTER_HOSTNAME:-"10.8.12.16"}
ALLUXIO_WORKER_MEMORY_SIZE=${ALLUXIO_WORKER_MEMORY_SIZE:-"100GB"}
ALLUXIO_RAM_FOLDER=${ALLUXIO_RAM_FOLDER:-"/home/appadmin/ramdisk"}
ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-"hdfs://ns/alluxio/"}

5.2 alluxio-env.sh(standby master-1)

ALLUXIO_MASTER_HOSTNAME=${ALLUXIO_MASTER_HOSTNAME:-"10.8.12.17"}
ALLUXIO_WORKER_MEMORY_SIZE=${ALLUXIO_WORKER_MEMORY_SIZE:-"100GB"}
ALLUXIO_RAM_FOLDER=${ALLUXIO_RAM_FOLDER:-"/home/appadmin/ramdisk"}
ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-"hdfs://ns/alluxio/"}

5.3 alluxio-env.sh(alluxio worker)

ALLUXIO_WORKER_MEMORY_SIZE=${ALLUXIO_WORKER_MEMORY_SIZE:-"100GB"}
ALLUXIO_RAM_FOLDER=${ALLUXIO_RAM_FOLDER:-"/home/appadmin/ramdisk"}
ALLUXIO_UNDERFS_ADDRESS=${ALLUXIO_UNDERFS_ADDRESS:-"hdfs://ns/alluxio/"}

5.4 alluxio-site.properties(所有节点)

alluxio.underfs.hdfs.configuration=/home/appadmin/hadoop-2.7.2/etc/hadoop/core-site.xml
alluxio.zookeeper.enabled=true
alluxio.zookeeper.address=10.8.12.16:2181,10.8.12.17:2181,10.8.12.18:2181
alluxio.master.journal.folder=hdfs://ns/alluxio/journal
# 端口超时也稍微设置大点，否则使用ZK的时候会有端口关闭的问题
alluxio.security.authentication.socket.timeout.ms=3000000
# 心跳超时设置大点，否则alluxio failover的时候worker可能会来不及切换到新master
alluxio.worker.block.heartbeat.timeout.ms=60000

6. 测试容错

6.1 启动集群方法

在master上和standby master上分别执行以下操作启动master进程

# 所有节点上的master都会启动
alluxio-start.sh master

在各个worker上分别执行以下操作来启动worker进程

alluxio-start.sh worker SudoMount

这里貌似一定要用该命令分别启动worker，否则就会出问题

6.2 进程一览

a1（10.8.12.16）节点的进程：

a2（10.8.12.17）节点的进程：

a3（10.8.12.18）节点的进程：（我这里多了个AlluxioMaster，因为我a3也设置了和其他节点无密码登入，所以会多开了这个，实际上不需要）

6.3 zk一览

发现16这台才是leader。

我们访问下web界面，发现只有16这台上才有 worker的信息：

访问下17这台，可以访问，但是为空！

18这台由于我们在alluxio-env.sh当中没有设置ALLUXIO_MASTER_HOSTNAME，所以是不会成为standby master的，虽然进程已经启动！！

PS：如果同时启动3台master，实际上也只会随机选择2台来做为主备。。。因此多了没有用。

6.4 namenode发生failover

首先我们测试namenode中一个挂掉，引发fail over是否对alluxio集群有影响。

首先看下namenode的情况：

我们跑到nn2节点上，查看下namenode的进程，然后杀死他（O.O 好可怕~~）

kill之后查看nn1状态：

然后我们看下alluxio是否正常，我们简单通过alluxio runTests命令来测试：

是不是很坑爹？竟然一个namenode挂掉了，alluxio竟然没有识别出来，仍然去挂掉的namenode上去获取datanode信息。还好报错给了我们提示：

The current failed datanode replacement policy is DEFAULT, and a client may configure this via 'dfs.client.block.write.replace-datanode-on-failure.policy' in its configuration.

原来这是个HDFS上的配置问题。这个问题的原因是我们只有3个节点，而且正好又设置了3个备份，当1个datanode挂了就会存在无法写入的问题。解决方法：

6.4.1 使得复制份数小于datanode数量（推荐）

<!--原来是3台节点3个复制，现在改成3个节点2个复制-->
        <property>
            <name>dfs.replication</name>
                <value>2</value>
        </property>

6.4.2 修改repllace策略

我们修改下hdfs-site.xml文件添加如下选项：

<property>
  <name>dfs.client.block.write.replace-datanode-on-failure.policy</name>
  <value>NEVER</value>
</property>

我们关闭下alluxio集群和dfs集群，然后重启下再模拟以上操作。这里推荐6.4.1的方法，因为那个方法是我验证过的，没问题~

6.4.3 重新复制新HDFS文件到alluxio配置目录

这一步不要忘记！！

6.5 alluxio failover测试

先在ZK客户端上确认下leader是nn1这台。

现在我们尝试把16这台上的alluxio master进程杀死O.O

再检查ZK发现多了台17，不过原来的旧的节点没删（这个我就觉得alluxio做的不太好啊，都死掉的节点了，查了下发现以前tachyon就有人提了PR解决这个问题了，怎么现在alluxio又有这个问题了？见[TACHYON-961] Delete previous leader znodes before taking leadership）。看来我也要去提问题了。

然后在a2这台做为standby master重新运行alluxio runTests命令

Blog of Kami Wan

No one is always busy. It just depends on what number you are on their priority list

alluxio基于HDFS的master容错配置