Installing Ganglia 3.x on Ubuntu 12.04.1

While Ganglia is a well known monitoring solution, it was difficult to find and regroup information for a successful and clean install on a 12.04.1 Ubuntu cluster of 30 nodes.  In particular, linking the UI and Apache was a bit hackish until I found the symbolic link trick (ie the alias).

What is Ganglia?

Ganglia is BSD-licensed open-source project implementing a scalable distributed monitoring system for high-performance computing systems such as clusters and Grids. It leverages widely used technologies such as XML for data representation, XDR for compact, portable data transport, and RRD tool for data storage and visualization. It has very low per-node overheads and high concurrency. The implementation is robust, has been ported to an extensive set of operating systems and processor architectures, and is currently in use on thousands of clusters around the world.

Overview/Concepts

Major components

There are 3 main parts to Ganglia: The nodes generating the statistics, the node(s) collecting the statistics, and a web front-end for displaying them.

• gmond: A process running on the monitored nodes. For Ubuntu, this is installed with the ganglia-monitor package:

sudo apt-get install ganglia-monitor

        Configuration in /etc/gmond.conf, and associated service is ganglia-monitor.

• gmetad: A process running on one or multiple machines that collects the statistics sent by the various gmond processes in the associated grid. For Ubuntu, this is the package ganglia-webfrontend package:

         sudo apt-get install ganglia-webfrontend

        Configuration in /etc/gmetad.conf, and associated service is gmetad.

• A web UI: The web front end is installed/contained within the same package as gmetad. The UI is used to display the collected data. 

One of the tricky things about Ganglia is how it connects to itself.  Any gmond processes sends its collected information towards a gmetad.  Generally, the  collector node(s) is both a node that will generate statistics and a collection point. Such a node will run the the same daemon gmond but will be tweaked to also run the gmetad daemon. In turn, the gmetad daemon provides data to the web front end.

On a multi-nodes deployment, it is possible to verify (or test) the gmond process and to receive an XML dump:

telnet localhost 8649

Note: The above example uses the default port.  This port can be changed in /etc/gmond.conf.

Installation

On each Ubuntu nodes used to collect statistics:

# Install service ganglia-monitor -> gmond
sudo apt-get install ganglia-monitor 

For the Ubuntu node that collects data and that is used to run the web UI:

# Install ganglia frontend + gmetad
sudo apt-get install ganglia-webfrontend

Note that the above is not sufficient for ganglia to be accessible via HTTP.  The Ganglia UI must be deployed within a web application server (e.g. Apache tomcat) to be accessible.  

Note:  Apache tomcat is installed as a dependency when the ganglia-frontend package is installed. It it also possible to install it separately by entering the following command:

sudo apt-get install apache2

In order to deploy and run Ganglia in Apache tomcat, it is required to copy the apache.conf file from /etc/ganglia-webfrontend/apache.conf to /etc/apache2/sites-enabled/:

sudo cp /etc/ganglia-webfrontend/apache.conf /etc/apache2/sites-enabled/ganglia.conf

Note: The file is renamed to clarify what it is used for.

The /etc/ganglia-webfrontend/apache.conf contains a simple alias for /ganglia towards /sur/share/ganglia-webfrontend.

In my opinion, it is simpler and cleaner to create a symbolic link from etc/ganglia-webfrontend/apache.conf to /etc/apache2/sites-enabled/ such as:

cd /etc/apache2/sites-enabled
sudo ln -s /etc/ganglia-webfrontend/apache.conf ganglia.conf
sudo service apache2 stop
sudo service apache2 start

With the above, and the Apache tomcat server running, the Ganglia UI is available at: <host ip>/ganglia.

Stop/start

Uing Ubuntu services:

sudo service gmetad start
sudo service ganglia-monitor start

Monitoring hadoop 1.0.4

Ganglia can be used to monitor a Hadoop cluster.

In file  $HADOOP_HOME/conf/hadoop-metrics2.properties, enable the various sink properties referring to ganglia (remove '#'). Near the end of the file, when specifying the addresses to sink to, use the same address as the mcast_join value found in /etc/gmond.conf, i.e if using multicast:

namenode.sink.ganglia.servers=239.2.11.71:8649
datanode.sink.ganglia.servers=239.2.11.71:8649
tasktracker.sink.ganglia.servers=239.2.11.71:8649
...

Monitoring HBase 0.94.5

Ganglia can also be used to monitor an HBase cluster.

In the file $HBASE_HOME/conf/hadoop-metrics.properties, enable the various configuration options for ganglia.

In the example below, ganglia 3.1.7 is used, and ganglia collects data using multicast (mcast_join value set to =239.2.11.71):

    hbase.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
  hbase.period=10
  hbase.servers=239.2.11.71:8649
  jvm.class=org.apache.hadoop.metrics.ganglia.GangliaContext31
  jvm.period=10
  jvm.servers=239.2.11.71:8649

References

Ganglia official wiki: http://sourceforge.net/apps/trac/ganglia/wiki/ganglia_readme