Hopefully the comments within them should be sufficient, a summary of the contents and usage runs as follows:

blackHoles.sh - Keeps an eye out for nodes that run multiple jobs (over 10) with a walltime of 0 seconds, and offlines any it finds. We run it via a 5 minute cron job on our PBS server.

get/plot_141_temps.sh - Queries our worker nodes for their temperature (via IPMI) and can take actions (powerdown) if a predetermined number of nodes exceed a specified threshold. If a temperature rise occurs, the worker nodes are shut down first, and each disk server is then monitored until its temperature crosses a threshold. We run this via a cron job on our head node. plot_141_temps requires gnuplot to produce the graphs.

glasgow-dashboard.tar.bz2 - Source code for our dashboards. Most of this should just work if you drop it into a directory on a php-enabled web server. Exceptions are: pbswebmon needs to be installed to get the table of Batch System Jobs; the temperature plot will obviously not work, unless you install get/plot_141_temps.sh above.


I'm wary of offering support for these, as they really are very Glasgow specific. That said, I'll try to provide help where requested.


Cheers
Mike