Cluster Testing

In order to check that the configuration has been set up correctly, start up HDFS. To start a Hadoop cluster you will need to start both the HDFS and YARN cluster.

Format a new distributed filesystem:

$ $HADOOP_PREFIX/bin/hdfs namenode -format <cluster_name>

Start the HDFS with the following command, run on the designated NameNode:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh –config $HADOOP_CONF_DIR –script hdfs start namenode

Run a script to start DataNodes on all slaves:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh –config $HADOOP_CONF_DIR –script hdfs start datanode

Start the YARN with the following command, run on the designated ResourceManager:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh –config $HADOOP_CONF_DIR start resourcemanager

Run a script to start NodeManagers on all slaves:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh –config $HADOOP_CONF_DIR start nodemanager

Start a standalone WebAppProxy server. If multiple servers are used with load balancing it should be run on each of them:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start proxyserver –config $HADOOP_CONF_DIR

Start the MapReduce JobHistory Server with the following command, run on the designated server:

$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh start historyserver –config $HADOOP_CONF_DIR

Hadoop Shutdown

Stop the NameNode with the following command, run on the designated NameNode:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh –config $HADOOP_CONF_DIR –script hdfs stop namenode

Run a script to stop DataNodes on all slaves:

$ $HADOOP_PREFIX/sbin/hadoop-daemon.sh –config $HADOOP_CONF_DIR –script hdfs stop datanode

Stop the ResourceManager with the following command, run on the designated ResourceManager:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh –config $HADOOP_CONF_DIR stop resourcemanager

Run a script to stop NodeManagers on all slaves:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh –config $HADOOP_CONF_DIR stop nodemanager

Stop the WebAppProxy server. If multiple servers are used with load balancing it should be run on each of them:

$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh stop proxyserver –config $HADOOP_CONF_DIR

Stop the MapReduce JobHistory Server with the following command, run on the designated server:

$ $HADOOP_PREFIX/sbin/mr-jobhistory-daemon.sh stop historyserver –config $HADOOP_CONF_DIR