cloudera learning4:Hadoop集群规划

时间:2023-12-24 23:48:13

涉及到一些关于硬件的东西,我也不是很懂,记录下来有待以后学习。

Hadoop集群一般都是由小到大,刚开始可能只有4到6个节点,随着存储数据的增加,计算量的增大,内存需求的增加,集群慢慢变大。

比如按照数据存储量增大集群,每个星期数据存储3TB数据,HDFS的block备份数为3,则集群就需要9TB的磁盘,一般还要再预估25%buffer。如果一台机器的存储量为16*3T,则大概每个月往集群中增加1台机器。

如何进行硬件选择?一般Hadoop节点分成管理节点(master node)和工作节点(work node)。管理节点上跑NameNode,Standby NameNode,ResourceManager,SecondaryNameNode。工作节点上跑DataNode,NodeManager,impala server进程。

work nodes的推荐配置:

中级配置(deep storage, 1Gb Ethernet ):

– 16 x 3TB SATA II hard drives, in a non-RAID, JBOD* configuraGon – 1 or 2 of the 16 drives for the OS, with RAID-1 mirroring
– 2 x 6-core 2.9GHz CPUs, 15MB cache
– 256GB RAM

– 2x1 Gigabit Ethernet

高级配置(high memory, spindle dense, 10Gb Ethernet ):

– 24 x 1TB Nearline/MDL SAS hard drives, in a non-RAID, JBOD* configuraGon

– 2 x 6-core 2.9GHz CPUs, 15MB cache – 512GB RAM (or more)
– 1x10 Gigabit Ethernet

Work Node不推荐RAID,不推荐Blade Servers。

master node的推荐配置:

Carrier-class hardware

Dual power supplies

Dual Ethernet cards
– Bonded to provide failover

RAIDed hard drives

Reasonable amount of RAM
– 64 GB for clusters of 20 nodes or less

– 96 GB for clusters of up to 300 nodes

– 128 GB for larger clusters

不推荐部署在虚拟化的主机上,因为虚拟化会带了很多不确定性,比如虚拟的三个server,实际的存储可能在一个物理server上,给hdfs的block备份带来风险。

Network推荐:

Nodes are connected to a top-of-rack switch

Nodes should be connected at a minimum speed of 1Gb/sec

Consider 10Gb/sec connecAons in the following cases:

– Clusters storing very large amounts of data

– Clusters in which typical jobs produce large amounts of intermediate
data

Racks are interconnected via core switches
Core switches should connect to top-of-rack switches at 10Gb/sec or faster

Beware of oversubscripAon in top-of-rack and core switches

Consider bonded Ethernet to miAgate against failure

Consider redundant top-of-rack and core switches 

用域名,避免用IP地址,最好配DNS.

OS建议选centos or RedHat Enterprise Linux (RHEL) 

磁盘划分越多越好,避免LVM(Logical Volume Manager),设置noatime。

存储的文件size越大越好。

OS,network,system time, user and group和component版本等等的配置,可以通过Cloudera Manager Host Inspector 进行check。