hadoop中怎么强制发送心跳
答案:1 悬赏:40 手机版
解决时间 2021-04-15 20:45
- 提问者网友：椧運幽默
- 2021-04-14 20:19
hadoop中怎么强制发送心跳
最佳答案
- 五星知识达人网友：洎扰庸人
- 2021-04-14 21:03
datenode以固定周期向namenode发送心跳，namenode如果在一段时间内没有收到心跳，就会标记datenode为宕机。
此段时间的计算公式是：
timeout = 2 * heartbeat.recheck.interval + 10 * dfs.heartbeat.interval
默认 heartbeat.recheck.interval 是5分钟，dfs.heartbeat.interval是3秒，所以DN挂了后，NN要确定DN为DEAR需要10min30s，也就是630S自动标识为dead。
dfs.heartbeat.interval
3
Determines datanode heartbeat interval in seconds.
dfs.heartbeat.recheck.interval
20000
Determines when machines are marked dead 单位：毫秒！！！
2014-04-02 12:24:33,464 INFO org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager: Setting heartbeat recheck interval to 30000 since dfs.namenode.stale.datanode.interval is less than dfs.namenode.heartbeat.recheck-interval
预计检测时间是=2*20+10*3=70，结果发现一台DN down了后，NN检测到的时间还是超过了70秒，最终标记DN为dead的时间是660秒，也就是说这项配置根本就没起作用。接下来我把rechekc的参数设置成100000（也就是100秒），最终发现NN检测到DN挂了确实是用了2*100+10*3 = 230秒。接下来再来解决第一个疑问，
看看什么是dfs.namenode.stale.datanode.interval，这里引进了一个状态叫做“stale”
DataNodes are marked as stale if it does not send heartbeat message to NameNode within the timeout configured using the configuration parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds).
dfs.namenode.stale.datanode.interval = 30000
而且默认情况下这个stale特性是关闭的，还需要把
dfs.namenode.check.stale.datanode = true
只有把这两项都配置上去，再结合上面的heartbeat才能正常work，否则的话就把dfs.heartbeat.recheck.interval设置成30秒以上吧。因为如果时间太多，那么对于NN来说是要不断更新状态的，负载太高。可以看一段dfs.namenode.stale.datanode.interval的说明
Default time interval for marking a datanode as "stale", i.e., if the namenode has not received heartbeat msg from a datanode for more than this time interval, the datanode will be marked and treated as "stale" by default. The stale interval cannot be too small since otherwise this may cause too frequent change of stale states. We thus set a minimum stale interval value (the default value is 3 times of heartbeat interval) and guarantee that the stale interval cannot be less than the minimum value. A stale data node is avoided during lease/block recovery. It can be conditionally avoided for reads (see dfs.namenode.avoid.read.stale.datanode) and for writes (see dfs.namenode.avoid.write.stale.datanode).
此段时间的计算公式是：
timeout = 2 * heartbeat.recheck.interval + 10 * dfs.heartbeat.interval
默认 heartbeat.recheck.interval 是5分钟，dfs.heartbeat.interval是3秒，所以DN挂了后，NN要确定DN为DEAR需要10min30s，也就是630S自动标识为dead。
2014-04-02 12:24:33,464 INFO org.apache.hadoop.hdfs.server.blockmanagement.HeartbeatManager: Setting heartbeat recheck interval to 30000 since dfs.namenode.stale.datanode.interval is less than dfs.namenode.heartbeat.recheck-interval
预计检测时间是=2*20+10*3=70，结果发现一台DN down了后，NN检测到的时间还是超过了70秒，最终标记DN为dead的时间是660秒，也就是说这项配置根本就没起作用。接下来我把rechekc的参数设置成100000（也就是100秒），最终发现NN检测到DN挂了确实是用了2*100+10*3 = 230秒。接下来再来解决第一个疑问，
看看什么是dfs.namenode.stale.datanode.interval，这里引进了一个状态叫做“stale”
DataNodes are marked as stale if it does not send heartbeat message to NameNode within the timeout configured using the configuration parameter "dfs.namenode.stale.datanode.interval" in seconds (default value is 30 seconds).
dfs.namenode.stale.datanode.interval = 30000
而且默认情况下这个stale特性是关闭的，还需要把
dfs.namenode.check.stale.datanode = true
只有把这两项都配置上去，再结合上面的heartbeat才能正常work，否则的话就把dfs.heartbeat.recheck.interval设置成30秒以上吧。因为如果时间太多，那么对于NN来说是要不断更新状态的，负载太高。可以看一段dfs.namenode.stale.datanode.interval的说明
Default time interval for marking a datanode as "stale", i.e., if the namenode has not received heartbeat msg from a datanode for more than this time interval, the datanode will be marked and treated as "stale" by default. The stale interval cannot be too small since otherwise this may cause too frequent change of stale states. We thus set a minimum stale interval value (the default value is 3 times of heartbeat interval) and guarantee that the stale interval cannot be less than the minimum value. A stale data node is avoided during lease/block recovery. It can be conditionally avoided for reads (see dfs.namenode.avoid.read.stale.datanode) and for writes (see dfs.namenode.avoid.write.stale.datanode).
我要举报
如以上问答信息为低俗、色情、不良、暴力、侵权、涉及违法等信息，可以点下面链接进行举报！
大家都在看
推荐资讯