LSF-to-PBSPro

AFRL's HAWK is changing its workload manager to PBSPro. Here are some things that may help make this transition easier.

Commonly Used Commands

The chart below shows the commonly used commands in LSF and their PBSPro equivalents.

LSF - PBSPro Command Comparison
LSF	PBSPro	Description
bqueues	qstat -Q	Show Queue information.
bqueues	qstat -Qf	Show Queue information.
bsub < script_name	qsub script_name	Submit a job to the queue for execution.
bhist	tracejob	These provide a complete view of job history, similar to LSF bhist.
bhist	qstat -xf
bjobs	qstat -a	Show all jobs currently running.
bstop	qhold	Hold a job.
bresume	qrls	Resume a job placed on hold by a user.
bkill	qdel	Kill a job.

Sample PBSPro Script

Your job script can contain the directives needed to request resources or set attributes for your job. Additional scripts can be found on each system at $SAMPLES_HOME, when available. Items in italics are meant to be replaced with your personal preferences (ncpus must be greater than 0, and resource items include all applications which require licenses, and the applications xpatch and guava). The -A option is optional, and will use $ACCOUNT as the default if it is not specified.

Sample Serial PBSPro Script

#!/bin/csh
#PBS -l walltime=h:mm:ss
#PBS -l select=ncpus=1
#PBS -l resource=x
#PBS -l job_type={SMP|MPI|MIX}
#PBS -j oe
#PBS -m abe
#PBS -M user1@afrl.hpc.mil, user2@your.domain
#PBS -A account_number (16-character account number)
#
#  Change to $WORK_DIR directory and copy input file.
#  We're using the archive command to pull information from
#  $ARCHIVE_HOME. The -C option tells the archive command which
#  directory in $ARCHIVE_HOME to look into. If you had a file in
#  $ARCHIVE_HOME/foo/bar, you would give the -C option foo/bar.
#
cd $WORK_DIR
archive get -C {directory in $ARCHIVE_HOME} {filename}
#
#  Run the analysis.
date
./my_job
date
#
#  Archive output. First we tar up $WORK_DIR to compress
#  the data and make it easier to transfer, then we use the
#  archive command to push it to $ARCHIVE_HOME. The -C option works
#  the same way here as it did above, except in this case,
#  you would give it a path to where you wanted to store
#  the files. More information on the archive command can
#  be found in the archive man page (man archive).
#
tar cvf ../{output filename}.tar .
archive put -C {directory in $ARCHIVE_HOME} ../{output filename}.tar
#
#  Exit the script.

Re-writing your Scripts

If you have used LSF in the past, your scripts will need to be adjusted for Hawk's new PBSPro workload manager. This chart will help you make those changes.

LSF - PBSPro Submission Script Directive Comparison
LSF	PBSPro	Description
#BSUB -q queue_name	#PBS -q queue_name	Queue job will be submitted to
#BSUB -n x	#PBS -l select=ncpus=x	Number of Processors ≥ 1
#BSUB -W h:mm	#PBS -l walltime=h:mm:ss	Wallclock Time
#BSUB -a {SMP\|MPI\|MIX}	#PBS -l job_type={SMP\|MPI\|MIX}	Job type of SMP, MPI, or MIX
#BSUB -J job_name	#PBS -N job_name	Job Name
#BSUB -o output_file_name	#PBS -o output_file_name	Redirect stdout to specified file
#BSUB -e error_file_name	#PBS -e error_file_name	Redirect stderr to specified error output file name
#BSUB -P account_number	#PBS -A account_number	Account number
#BSUB -u email1, email2	#PBS -M email1, email2	Specify e-mail information
#BSUB -B	#PBS -m b	Send e-mail at job start
#BSUB -N	#PBS -m e	Send e-mail at job end

Submitting a Job to the Queue

To queue a job, run the command:

% qsub script_name

To run the job using modifiers instead of specifying information in the script (the information preceded by "#PBS" in the script above), use the following chart:

PBSPro Submission Command Modifiers
Command	Directives	Description
qsub	-I	Submits a job Interactively Cannot be included in your script. Must be used during submission.
	-o output_file_name	Name of output file
	-e error_file_name	Name of error file
	-M email1, email2	Validate email for any PBSPro informational notices.
	-m b	Send e-mail when job begins execution **
	-m e	Send e-mail when job terminates**
	-m a	Send e-mail when job is aborted by PBSPro**
	-m abe	Send e-mail when job begins, terminates, and/or is aborted**

**NOTE**
To ensure all requested e-mails are sent, it is necessary to use multiple directives in one line, as only the final "-m" is read by PBSPro. For example, to have an e-mail sent at execution and termination of your job, the qsub command would be qsub -be.

Additional Information

Vendor documentation, including: PBSPro Vendor Guide, PBSPro User Training, and the AFRL Local Workload Management System User Guide, can be found on the protected AFRL User website at:

https://user.afrl.hpc.mil/documentation/index.php

Please be aware that the vendor documentation is not on the publically accessible site, and Kerberos authorization is required.

If you have any questions, please call the Consolidated Customer Assistance Center (CCAC) at 800-CCAC-039 (877 222-2039).

The appearance of hyperlinks does not constitute endorsement by the Department of Defense of this website or the information, products, or services contained therein. For other than authorized activities such as military exchanges and morale, welfare and recreation sites, the Department of Defense does not exercise any editorial control over the information you may find at these locations. Such links are provided consistent with the stated purpose of this DoD website.