COMPUTER HELP AND TIPS.

Showing posts with label MAUI. Show all posts

The concept of Processor Equivalent

Learned a new concept of Process Equivalent or PE. It is a measure of the actual impact of a set of requested resources by a job on the total resources system wide.

Taken from MAUI Admin Guide by Cluster Resources

PE =MAX (ProcsRequestedByJob / TotalConfiguredProcs,
MemoryRequestedByJob / TotalConfiguredMemory,
DiskRequestedByJob / TotalConfiguredDisk,
SwapRequestedByJob / TotalConfiguredSwap) * TotalConfiguredProcs

Assume a homogeneous 100 node system with 4 processors and 1 GB of memory per node. A job is submitted requesting 2 processors and 768 MB of memory. The PE for this job would be calculated as:

PE = MAX(2/(100*4), 768/(100*1024)) * (100*4) = 3.

This result makes sense since the job would be consuming 3/4 of the memory on a 4 processor node. The calculation works equally well on homogeneous or heterogeneous systems, uniprocessor or large way SMP systems.

Overview of MAUI Scheduler Commands

Finally, I thought I summarise the documentation of MAUI Schduler Commands in one nice blog entry "Overview of MAUI Scheduler Commands". Hopefully, this will help summarise the commands used for system administration of the MAUI Scheduler

MAUI Installation in xCAT 2.x

What is MAUI?
Maui Cluster Scheduler (a.k.a. Maui Scheduler) is our first generation cluster scheduler. Maui is an advanced policy engine used to improve the manageability and efficiency of machines ranging from clusters of a few processors to multi-teraflop supercomputers.

Taken and modified from https://xcat.wiki.sourceforge.net/Maui

Step 1: Download the tarball

Go to http://clusterresources.com/, create an account and download here:
http://www.clusterresources.com/product/maui/index.php?
Untar in /tmp

Step 2: Setup Torque for xCAT

# cd /opt/torque
# ln -s x86_64/bin .
# ln -s x86_64/lib .
# ln -s x86_64/sbin .
# export PATH=$PATH:/opt/torque/x86_64/bin/

Step 3: Setup MAUI

# cd maui-3.2.6p21
# ./configure --prefix=/opt/maui --with-pbs=/opt/torque/
# make -j8
# make install
# cp /opt/xcat/share/xcat/netboot/add-on/torque/moab /etc/init.d/maui
(Edit /etc/init.d/maui so that all MOAB is MAUI and all moab becomes maui)
# service start maui
# chkconfig --level 345 maui on

Step 4: Configure MAUI

# touch /etc/profile.d/maui.sh
# vim maui (Type: export PATH=$PATH:/opt/maui/bin)
# source /etc/profile.d/maui.sh
# vim /usr/local/maui/maui.cfg
Change: RMCFG[] TYPE=PBS@...@ to: RMCFG[] TYPE=PBS
# service maui restart

Now run:

# showq

You should see all of the processors. Next try running a job to make sure that maui picks it up.

Dealing with stuck jobs and Torque and MAUI

This is a add-on for the blog entry "Manually Deleting Torque amd PBS jobs using MAUI"

1. Force the Torque Server or MOM to send an obituary of the job ID to the server

# qsig -s 0 job_id

2. Using the momctl command on the compute nodes where the job is listed. You can use a tracejob to check which nodes the job has been send to

# momctl -c job_id -h compute_node_1

3i. Setting the qmgr server setting mom_job_sync to True might help prevent jobs from hanging.

# qmgr -c "set server mom_job_sync = True"

3ii. To verify that the setting in 3i is in, you can use trhe command

# qmgr -c "p s"

4. The final option. If all else fail, do a

qdel -p job_id

For more information, see Adaptive Computing Website Section 11.1.7 Stuck Jobs

Manually Deleting Torque amd PBS jobs using MAUI

Tracing Jobs
To trace a job with MAUI commands including the nodes the jobs are residing, you can use the commands

# showq -r

Alternatively, you can use the MAUI commands to trace the job activity

# trace job_id

Deleting Jobs
To delete a job with MAUI commands, you can use the commands,

# canceljob job_id

Alternatively, you can also use PBS commands to delete a job

# qdel job_id

PBS mom control
If not able to delete a stale job which has no process, you can use the momctl command

# momctl

If you are unable to delete the stale job with has no process, you can use momctl to do diagnostic. Basically The momctl command allows remote shutdown, reconfiguration, diagnostics, and querying of the pbs_mom daemon. For more information on momctl, do look at momctl by http://www.clusterresources.com/:

Example 1: Diagnosis of pbs_mom

# momctl -h node1 -d 1

Example 2: Cycle the pbs_mom on node 1

# momctl -h nod1 -C

Manually deleting the jobs
To manually delete the jobs, you should shutdown the pbs server

# service pbs_server stop

Remove the job spool files

# rm /var/spool/pbs/server_priv/jobs/111.host.SC 
# rm /var/spool/pbs/server_priv/jobs/111.host.JB

Restart the pbs_server

# service pbs_server restart

Further Information:

Deleting PBS/Maui Jobs

MAUI: Cannot send request to server ...... (server may not be running)

ERROR: cannot send request to server ................42559 (server may not be running)
ERROR: cannot request service (status)

A few assumptions for this blog entry. You are using Torque and MAUI only for the resource manager and scheduler respectively.

Several things to check.

Make sure your /etc/hostname reflect the correct entries for all the head and compute nodes
For the Torque configuration, do follow the configuration seen in the blog entry Setting up Torque Server on xCAT 2.x. Pay particular attendtion to Step 2 and 3
For the MAUI configuration, do follow the configuration seen in the blog entry MAUI Installation in xCAT 2.x
Check the logs for MAUI at /usr/local/maui/log

After following Pointer 1-4, you are still unable to solve, you can get some hints if you can go to

# cd /opt/maui/sbin/

and type

./maui

In my particular situation, I encounter server must be started on host 'xxx.xxx.xxx.xxx.xxx' (currently on 'xxx') which is my short-name. I resolve it quickly by going to /usr/local/maui/maui.cfg and edit the

SERVERHOST = xxxx