集群管理指南描述了如何使用Cloudera Manager在Cloudera企业部署中配置和管理集群。Cloudera Enterprise Hadoop管理员管理资源、主机、高可用性以及备份和恢复配置。Cloudera Manager管理控制台是管理员用来监视和管理集群的主要工具。您还可以将Cloudera Manager API用于集群管理任务。
Performance Management
Disable the tuned Service
- Ensure that the tuned service is started:
systemctl start tuned
- Turn the tuned service off:
tuned-adm off
- Ensure that there are no active profiles:
The output should contain the following line:tuned-adm list
No current active profile
- Shutdown and disable the tuned service:
systemctl stop tuned systemctl disable tuned
Disabling Transparent Hugepages
To see whether transparent hugepages are enabled, run the following commands and check the output:
$ cat defrag_file_pathname
$ cat enabled_file_pathname
- [always] never means that transparent hugepages is enabled.
- always [never] means that transparent hugepages is disabled.
To disable transparent hugepages on reboot, add the following commands to the /etc/rc.d/rc.local file on all cluster hosts:
- RHEL/CentOS 7.x:
echo never > /sys/kernel/mm/transparent_hugepage/enabled echo never > /sys/kernel/mm/transparent_hugepage/defrag
- RHEL/CentOS 6.x
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
Setting the vm.swappiness Linux Kernel Parameter
To view your current setting for vm.swappiness, run:
cat /proc/sys/vm/swappiness
To set vm.swappiness to 1, run:
sudo sysctl -w vm.swappiness=1
Improving Performance in Shuffle Handler and IFile Reader
Best Practices for MapReduce Configuration
Send a heartbeat as soon as a task finishes
<property> <name>mapreduce.tasktracker.outofband.heartbeat</name> <value>true</value> </property>
Reduce the interval for JobClient status reports on single node systems
<property> <name>jobclient.progress.monitor.poll.interval</name> <value>10</value> </property>
Tune the JobTracker heartbeat interval
Tuning the minimum interval for the TaskTracker-to-JobTracker heartbeat to a smaller value may improve MapReduce performance on small clusters.
<property> <name>mapreduce.jobtracker.heartbeat.interval.min</name> <value>10</value> </property>
Start MapReduce JVMs immediately
Map . Reduce .slowstart.completed.maps属性指定在调度任何Reduce任务之前必须完成的任务中映射任务的比例。对于需要快速周转的小型作业,将此值设置为0可以提高性能;较大的值(高达50%)可能适合较大的作业。
<property> <name>mapred.reduce.slowstart.completed.maps</name> <value>0</value> </property>
Decrease Reserve Space
Cloudera recommends reducing the root user block reservation from 5% to 1% for the DataNode volumes. To set reserved space to 1% with the tune2fs command:
# tune2fs -m 1 /dev/sde1