Unable to clear the offline mode for Compute node using pbsnodes command

If you try to make a node online after you deliberately off-line the node perhaps for some repair work and unable to online again. Instead on your Torque Server,  you see errors such as these

PBS_Server: LOG_ERROR::is_request, bad attempt to connect from
192.168.1.1:1023 (address not trusted - check entry in server_priv/nodes

As the log says, there is some issue in the server_priv configuration. If all your nodes are working well except that, you should take a look at the client settings. Maybe your server_priv has the wrong configration.

Once you have cleared the error, just fire off the command

# pbsnodes -c node_name

Encountering Unauthorised Request for Qmgr command

Taken from the Torque Users Group http://www.clusterresources.com/pipermail/torqueusers/2011-05rch/003408.html


If you encounter "qmgr obj=que svr=default: Unauthorized Request" whenever you type the qmgr command, it is due to /var/spool/pbs/server_name not in sync with the name associated with eth0. To mitigate the problem,

1. server_name must match the first no-localhost name in /etc/hosts
2. You can check your hostname on your Linux Box, by typing hostname

Using Torque to launch Interactive Session

Some simple uses of Torque commands

To launch an Interactive session asking for 4 nodes with 8 cores each

$ qsub -N demo -I -l nodes=4:ppn=8,walltime=01:00:00 scriptfile

where

• -I:  Interactive Session
• nodes: Server
• ppn: Cores

To launch Interactive X11 Session

$ qsub -I -X -l nodes=4:ppn=8,walltime:01:00:00 scriptfile

If you are using windows as your client and you wish to display X11 on your windows, do look at
using Putty, X11 to forward X11

Setting up Torque Server on xCAT 2.x

Modified from xCAT 2 Advanced Cookbook
Step 1: Setup Torque Server

# cd /tmp
# wget http://www.clusterresources.com/downloads/torque/torque-2.3.0.tar.gz
# tar zxvf torque-2.3.0.tar.gz
# cd torque-2.3.0
# CFLAGS=-D__TRR ./configure \
--prefix=/opt/torque \
--exec-prefix=/opt/torque/x86_64 \
--enable-docs \
--disable-gui \
--with-server-home=/var/spool/pbs \
--enable-syslog \
--with-scp \
--disable-rpp \
--disable-spool

# make

# make install

Step 2: Configure Torque

# cd /opt/torque/x86_64/lib
# ln -s libtorque.so.2.0.0 libtorque.so.0
# echo "/opt/torque/x86_64/lib" >>/etc/ld.so.conf.d/torque.conf
# ldconfig
# cp -f /opt/xcat/share/xcat/netboot/add-on/torque/xpbsnodes /opt/torque/x86_64/bin/
# cp -f /opt/xcat/share/xcat/netboot/add-on/torque/pbsnodestat /opt/torque/x86_64/bin/

Create /etc/profile.d/torque.sh:

# vim /etc/profile.d/torque.sh

Type:

export PBS_DEFAULT=n00 (where n00 is the Head Node)
export PATH=/opt/torque/x86_64/bin:$PATH

Step 3. Define Nodes

# cd /var/spool/pbs/server_priv
# vim nodes

Type:

n01 np=8 (where np is the number of core for the server)

Step 4: Setup and Start Service

# cp -f /opt/xcat/share/xcat/netboot/add-on/torque/pbs /etc/init.d/
# cp -f /opt/xcat/share/xcat/netboot/add-on/torque/pbs_mom /etc/init.d/
# cp -f /opt/xcat/share/xcat/netboot/add-on/torque/pbs_sched /etc/init.d/
# cp -f /opt/xcat/share/xcat/netboot/add-on/torque/pbs_server /etc/init.d/
# chkconfig --del pbs
# chkconfig --del pbs_mom
# chkconfig --del pbs_sched
# chkconfig --level 345 pbs_server on
# service pbs_server start

Step 5: Edit pbs_mom, pbs_sched, pbs_server

# vim /etc/init.d/pbs_sched
(Ensure your path is correct for BASE_PBS_PREFIX=/opt/torque) 
# vim /etc/init.d/pbs_mom 
(Ensure your path is correct for chmod 777 /var/spool/pbs/spool /var/spool/pbs/undelivered)
(Ensure your path is correct for chmod o+t /var/spool/pbs/spool /var/spool/pbs/undelivered)
# vim /etc/init.d/pbs_server
(Ensure your path is correct for BASE_PBS_PREFIX=/opt/torque)
(Ensure your path is correct for PBS_HOME=/var/spool/pbs)

Step 6: Install pbstop

# cp -f /opt/xcat/share/xcat/netboot/add-on/torque/pbstop /opt/torque/x86_64/bin/
# chmod 755 /opt/torque/x86_64/bin/pbstop

Step 7: Install Perl Curses for pbstop

# yum install perl-Curses (You will need RPMForge Repository enabled. Do check Installing RPMForge)

Step 8: Create a Torque Default Queue

# qmgr

create queue dque
set queue dque queue_type = Execution
set queue dque enabled = True
set queue dque started = True
set server scheduling = True
set server default_queue = dque
set server log_events = 127
set server mail_from = adm
set server query_other_jobs = True
set server resources_default.walltime = 00:01:00
set server scheduler_iteration = 60
set server node_pack = False
set server keep_completed=300

Step 9: Setup Torque Clients (x86_64) (using manual installation)

Ensure all the /etc/hosts contains the head and compute node

# vim /etc/hosts(include all the hosts)

# pscp -r /opt/torque compute:/opt/
# pscp -r /var/spool/pbs compute:/var/spool/

Change the sticky bit for the /var/log/pbs/spool & /var/log/spool/undelivered

# chmod 777 /var/spool/pbs/spool /var/spool/pbs/undelivered
# chmod o+t /var/spool/pbs/spool /var/spool/pbs/undelivered

(Ensure your path is correct for chmod 777 /var/spool/pbs/spool /var/spool/pbs/undelivered)
(Ensure your path is correct for chmod o+t /var/spool/pbs/spool /var/spool/pbs/undelivered)

Step 10: Start pbs_mom for Torque Client Node
Go to the Head Node

# pscp /etc/init.d/pbs_mom compute:/etc/init.d/

Edit pbs_mom

# vim /etc/init.d/pbs_mom

BASE_PBS_PREFIX=/opt/torque
chmod 777 /var/spool/pbs/spool /var/spool/pbs/undelivered
chmod o+t /var/spool/pbs/spool /var/spool/pbs/undelivered

# service pbs_mom start

Step 11: Start the pbs_mom, pbs_server, pbs_sched services at Head Node

# service pbs_mom start
# service pbs_sched start
# service pbs_server start

Step 12: Check the PBS is working
Go to Head Node

# pbstop
# pbsnodes -a (You should see some the nodes as "free")

Torque Resource Manager Server Parameters

Useful Information on Torque Resource Manager Server Parameters. Do take a look
Torque Resource Manager Appendix B: Server Parameters

qdel: Server could not connect to MOM

If you encounter this error "qdel: Server could not connect to MOM" and you are using OpenPBS or PBS Pro, your users might encounter a problem where you are unable to kill the job.

Usually the error above there is a problem on pbs_mom on the compute node.

Step 1: Check the Queue and check the node that the jobs lands on

# qstat -a (for summary)
# qstat -n (You will see where the nodes the job lands)

Step 2: Try to kill as cluster administrator.

# qdel 

*If you are not able to delete the job somehow
Step 3: Try restarting PBS Mom on the client

# service pbs_mom restart

Step 4: If Step 3 is not workable, it might be due to connection issues or hardware problems
Try

# ssh compute_node_1

If you cannot, you have to remote KVM into the server to take a look

Dealing with stuck jobs and Torque and MAUI

This is a add-on for the blog entry "Manually Deleting Torque amd PBS jobs using MAUI"

1. Force the Torque Server or MOM to send an obituary of the job ID to the server

# qsig -s 0 job_id

2. Using the momctl command on the compute nodes where the job is listed. You can use a tracejob to check which nodes the job has been send to

# momctl -c job_id -h compute_node_1

3i. Setting the qmgr server setting mom_job_sync to True might help prevent jobs from hanging.

# qmgr -c "set server mom_job_sync = True"

3ii. To verify that the setting in 3i is in, you can use trhe command

# qmgr -c "p s"

4. The final option. If all else fail, do a

qdel -p job_id

For more information, see Adaptive Computing Website Section 11.1.7 Stuck Jobs

Manually Deleting Torque amd PBS jobs using MAUI

Tracing Jobs
To trace a job with MAUI commands including the nodes the jobs are residing, you can use the commands

# showq -r

Alternatively, you can use the MAUI commands to trace the job activity

# trace job_id

Deleting Jobs
To delete a job with MAUI commands, you can use the commands,

# canceljob job_id

Alternatively, you can also use PBS commands to delete a job

# qdel job_id

PBS mom control
If not able to delete a stale job which has no process, you can use the momctl command

# momctl

If you are unable to delete the stale job with has no process, you can use momctl to do diagnostic. Basically The momctl command allows remote shutdown, reconfiguration, diagnostics, and querying of the pbs_mom daemon. For more information on momctl, do look at momctl by http://www.clusterresources.com/:

Example 1: Diagnosis of pbs_mom

# momctl -h node1 -d 1

Example 2: Cycle the pbs_mom on node 1

# momctl -h nod1 -C

Manually deleting the jobs
To manually delete the jobs, you should shutdown the pbs server

# service pbs_server stop

Remove the job spool files

# rm /var/spool/pbs/server_priv/jobs/111.host.SC 
# rm /var/spool/pbs/server_priv/jobs/111.host.JB

Restart the pbs_server

# service pbs_server restart

Further Information:

1. Deleting PBS/Maui Jobs

Testing Torque Server Configuration

After Setting up Torque Server Setting up Torque Server on xCAT 2.x, you may want to verify whether the Torque Server Configuration is correct. Taken from 1.6 Testing Server Configuration (Adaptive Computing)

To verify whether your configuration works, do do the following test

1. Verify all queues are properly configured

# qstat -q

2. View additional server configuration

# qmgr -c 'p s'

3.verify all nodes are correctly reporting

# pbsnodes -a

4 Submit a basic job - DO NOT RUN AS ROOT

# su - testuser
$ echo "sleep 30" | qsub

5. Verify jobs display

# qstat

MAUI: Cannot send request to server ...... (server may not be running)

ERROR: cannot send request to server ................42559 (server may not be running)
ERROR: cannot request service (status)

A few assumptions for this blog entry. You are using Torque and MAUI only for the resource manager and scheduler respectively.

Several things to check.

1. Make sure your /etc/hostname reflect the correct entries for all the head and compute nodes
2. For the Torque configuration, do follow the configuration seen in the blog entry Setting up Torque Server on xCAT 2.x. Pay particular attendtion to Step 2 and 3
3. For the MAUI configuration, do follow the configuration seen in the blog entry MAUI Installation in xCAT 2.x
4. Check the logs for MAUI at /usr/local/maui/log

After following Pointer 1-4, you are still unable to solve, you can get some hints if you can go to

# cd /opt/maui/sbin/

and type

./maui

In my particular situation, I encounter server must be started on host 'xxx.xxx.xxx.xxx.xxx' (currently on 'xxx') which is my short-name. I resolve it quickly by going to /usr/local/maui/maui.cfg and edit the

SERVERHOST = xxxx

Torque Error - Address already in use (98) in scan_for_exiting, cannot bind to port 464 in client_to_svr - too many retries

pbs_mom;Svr;pbs_mom;LOG_ERROR:: Address already in use (98) in scan_for_exiting, cannot bind to port 464 in client_to_svr - too many retries

One cause for this is very high traffic on the network not allowing the mom and the server to communicate properly. One common case are job scripts that incessantly run qstat. You will be surprise that sometimes users input some of these qstat scripts that cause the error

Dynamic server provisioning with xCAT and TORQUE

I have found a good article "Dynamic server provisioning with xCAT and TORQUE" from IBM Developers Network > Open Source > Technical Library

"This article describes a solution for building a dynamically provisioned high-performance computing (HPC) cluster system using the Extreme Cloud Administration Toolkit (xCAT) and Tera-scale Open-source Resource and QUEue Manager (TORQUE) open source packages. The xCAT is a leading solution for dynamically provisioning compute, storage, and network resources. TORQUE is a workload and resource-management system that manages batch jobs and compute nodes and schedules the execution of those jobs."

Jobs submitted placed on the queue and not able to run on HPC

Sometimes, when out users submit job via MOAB using Torque as a resource manager, their jobs is just not able to run even though there are resources to accomdoate the run. Of course there are many ways to troubleshoot. Here is how I troubleshoot 1 of them.

Step 1: Use the command checkjob on the process ID of the job that is stuck.
# checkjob 100001
--------------------------------------
Partition List:      xxxxxxxx
Flags:                 RESTARTABLE
Attr:                   checkpoint
StartPriority:       94
rejected for CPU              - (null)
rejected for State              - (null)
NOTE: job req cannot run in partition xxxxx (available procs do not meet requirements : 0 of 32 procs found)
---------------------------------------
Here's the key hint under NOTE. There is no node suitable for this user. But why? Launch another command


# qstat -f 100001
----------------------------------
euser = xxxxx
egroup = xxxxx
queue_rank = xxxxx
queue_type = xxxxx
etime = xxxxxxxx
submit_args = -1 nodes2:ppn=16 ./run.sh
-----------------------------------
Note the key issue is that users submitted a job requesting 2 nodes with 16 cores each. This does not exist in our cluster configuration. Hence it is not able to run....

Using MOAB and Torque Commands together to analyse the problem is useful indeed.