Skip Nav

SGI Ice X (Spirit) User Guide

Table of Contents

1. Introduction

1.1. Document Scope and Assumptions

This document provides an overview and introduction to the use of the SGI Ice X (Spirit) located at the AFRL DSRC, along with a description of the specific computing environment on Spirit. The intent of this guide is to provide information to enable the average user to perform computational tasks on the system. To receive the most benefit from the information provided here, you should be proficient in the following areas:

  • Use of the UNIX operating system
  • Use of an editor (e.g., vi or emacs)
  • Remote usage of computer systems via network or modem access
  • A selected programming language and its related tools and libraries

1.2. Policies to Review

All policies are discussed in the AFRL DSRC Policy Guide. All users running at the AFRL DSRC are expected to know, understand, and follow the policies discussed. If you have any questions about AFRL DSRC's policies, please contact the HPC Help Desk.

1.3. Obtaining Accounts

The process of getting an account on the HPC systems at any of the DSRCs begins with getting an account on the HPCMP Portal to the Information Environment, commonly called a "pIE User Account." If you do not yet have a pIE User Account, please visit HPC Centers: Obtaining An Account and follow the instructions there. If you need assistance with any part of this process, please contact the HPC Help Desk at

1.4. Requesting Assistance

The HPC Help Desk is available to help users with unclassified problems, issues, or questions. Analysts are on duty 8:00 a.m. - 11:00 p.m. Eastern, Monday - Friday (excluding Federal holidays).

For more detailed contact information, please see our Contact Page.

2. System Configuration

2.1. System Summary

Spirit is an SGI ICE X System. The login nodes and compute nodes are populated with 2.6-GHz Intel E5 Sandy Bridge 64-bit 8-core processors. Spirit uses a dedicated Numalink v5 communications network for MPI messages and IO traffic. Spirit uses Lustre to manage its high-speed, parallel, Infiniband FDR file system. Spirit has 4590 compute nodes that share memory only on the node; nodes do not share memory. Each compute node has two 8-core processors (16 cores) with its own Redhat Enterprise Linux (RHEL) operating system and 32 GBytes of DDR3 memory, with no user-accessible swap space. Spirit has 2.7 PBytes (formatted) of disk storage.

Spirit's intended use is as a batch-scheduled HPC system, and its login nodes are not to be used for large computational (e.g. memory, IO, long executions) work. All executions that require large amounts of system resources must be sent to the compute nodes by batch job submission.

Node Configuration
Login Nodes Compute Nodes
Total Nodes 8 4590
Operating System RHEL
Cores/Node 16
Core Type Intel E5 Sandy Bridge
Core Speed 2.6 GHz
Memory/Node 64 GBytes 32 GBytes
Accessible Memory/Node 62 GBytes 29 GBytes
Memory Model Shared on node. Shared on node.
Distributed across cluster.
Interconnect Type FDR 14x Infiniband FDR 14x Infiniband;
Enhanced LX Hypercube
File Systems on Spirit
Path Capacity Type
1.2 PBytesLustre
1.2 PBytesLustre

2.2. Processor

Spirit uses Intel Sandy Bridge E5-2670 64-bit processors on its login and compute nodes. Login nodes run at 2.6 GHz and have two 8-core processors for a total of 16 cores per node. Each processor has 8x32 KBytes of L1 instruction cache, 8x32 KBytes of L1 data cache, 8x256 KBytes of L2 cache, and access to a 20-MByte L3 cache that is shared among all 8 cores of the processor.

2.3. Memory

Spirit uses both shared and distributed memory models. Memory is shared among all the cores on a node, but is not shared among the nodes across the cluster.

Each login node contains 64 GBytes of main memory. All memory and cores on the node are shared among all users who are logged in. Therefore, users should not use more than 8 GBytes of memory at any one time.

Each compute node contains 29 GBytes of user-accessible shared memory.

2.4. Operating System

Spirit's operating system is SGI's Performance Suite, which is a combination of the Redhat Enterprise Linux (RHEL) Operating System and SGI-specific optimizations, utilities, and tools. More information on SGI Performance Suite can be found on SGI's Performance Suite page.

The compute nodes can provide access to dynamically shared objects and most of the typical Linux commands and basic functionality, either by compiling your application (in the case of shared objects) or by including the command in the PBS batch submission script.

2.5. File Systems

Spirit has the following file systems available for user storage:

/home is a locally mounted Lustre file system with a formatted capacity of 1.2 PBytes. All users have a home directory located on this file system, which can be referenced by the environment variable $HOME.

/workspace is a locally mounted Lustre file system that is tuned for high-speed I/O performance and has a formatted capacity of 1.2 PBytes. All users have a work directory located on this file system, which can be referenced by the environment variable $WORKDIR.

2.6. Peak Performance

Spirit is rated at 62.83 HABUs and 1.5 peak PFLOPS.

3. Accessing the System

3.1. Kerberos

A Kerberos client kit must be installed on your desktop to enable you to get a Kerberos ticket. Kerberos is a network authentication tool that provides secure communication by using secret cryptographic keys. Only users with a valid HPCMP Kerberos authentication can gain access to Spirit. More information about installing Kerberos clients on your desktop can be found at HPC Centers: Kerberos & Authentication.

3.2. Logging In

The system host name for the Spirit cluster is, which will redirect the user to one of eight (8) login nodes. Hostnames and IP addresses to these nodes are available upon request from the HPC Help Desk.

The preferred way to login to Spirit is via ssh, as follows:

% ssh

Kerberized rlogin is also allowed.

3.3. File Transfers

File transfers to DSRC systems (except transfers to the local archive system) must be performed using Kerberized versions of the following tools: scp, ftp, sftp, and mpscp.

The command below uses secure copy (scp) to copy a single local file into a destination directory on a Spirit login node. The mpscp command is similar to the scp command, but has a different underlying means of data transfer, and may enable greater transfer rate. The mpscp command has the same syntax as scp.

% scp local_file

Both scp and mpscp can be used to send multiple files. This command transfers all files with the .txt extension to the same destination directory.

% scp *.txt

The example below uses the secure file transfer protocol (sftp) to connect to Spirit, then uses the sftp cd and put commands to change to the destination directory and copy a local file there. The sftp quit command ends the sftp session. Use the sftp help command to see a list of all sftp commands.

% sftp

sftp> cd target_dir
sftp> put local_file
sftp> quit

The Kerberized file transfer protocol (kftp) command differs from sftp in that your username is not specified on the command line, but given later when prompted. The kftp command may not be available in all environments.

% kftp

username> user
kftp> cd target_dir
kftp> put local_file
kftp> quit

Windows users may use a graphical file transfer protocol (ftp) client such as Filezilla.

4. User Environment

4.1. User Directories

4.1.1. Home Directory

Each user is allocated a home directory (the current working directory immediately after login) with an initial disk quota of 100 GBytes of permanent storage that is backed up. Your home directory can be referenced locally with the $HOME environment variable from all nodes in the system.

You may submit a request to increase your disk space quota by contacting our Service Center. You must supply the following information for evaluation of the request by the system administrators and the AFRL DSRC management:

  • Amount of system resource requested
  • Length of time requested for the increase
  • Special deadlines for the project
  • Explanation of the attempts to work within limits
4.1.2. Work Directory

Spirit has one large file system (/workspace) for the temporary storage of data files needed for executing programs. You may access your personal working directory under /workspace by using the $WORKDIR environment variable, which is set for you upon login. Your $WORKDIR directory has no disk quotas, and files stored there do not affect your permanent file quota usage. Because of high usage, the /workspace file system tends to fill up frequently. Please review the Workspace Policy and be mindful of your disk usage.

REMEMBER: /workspace is a "scratch" file system and is not backed up. You are responsible for managing files in your $WORKDIR by backing up files to the MSAS and deleting unneeded files when your jobs end.

All of your jobs should execute from your $WORKDIR directory, not $HOME. While not technically forbidden, jobs that are run from $HOME are subject to disk space quotas and have a much greater chance of failing if problems occur with that resource. Jobs that are run entirely from your $WORKDIR directory are more likely to complete, even if all other resources are temporarily unavailable.

If you use $WORKDIR in your batch scripts, you must be careful to avoid having one job accidentally contaminate the files of another job. One way to avoid this is to use the $JOBDIR (or $WORK_DIR) directory, which is unique to each job on the system. The $JOBDIR directory is not subject to the Workspace Policy until the job exits the workload management system.

4.1.3. Center Directory

The Center-Wide File System (CWFS) provides file storage that is accessible from Spirit's login nodes, and from all nodes of the Utility Server. The CWFS permits file transfer between Spirit and the Utility Server using simple copy commands. Each user has their own directory in the CWFS. The name of your CWFS directory may vary between machines and between centers, but the environment variable $CENTER will always refer to this directory.

The example below shows how to copy a file from your work directory on Spirit to your work directory on the Utility Server using the CWFS. The CWFS ($CENTER) is mounted on both Spirit and the Utility Server.

While logged into Spirit, copy your file from your Spirit work directory to the CWFS.

% cp $WORKDIR/filename $CENTER

Then, log into the Utility Server and copy your file from the CWFS to your Utility Server work directory.

% cp $CENTER/filename $WORKDIR

For more information about using the Utility Server, see the Utility Server Quick Reference and the Utility Server User Guide.

4.2. Shells

The following shells are available on Spirit: csh, bash, ksh, tcsh, zsh and sh. To change your default shell, please modify your preferred shell entry in pIE ( Once changed in pIE, your preferred shell will become your default shell on the Spirit cluster within 1-2 working days.

4.3. Environment Variables

A number of environment variables are provided by default on all HPCMP high performance computing (HPC) systems. We encourage you to use these variables in your scripts where possible. Doing so will help to simplify your scripts and reduce portability issues if you ever need to run those scripts on other systems.

4.3.1. Login Environment Variables

The following environment variables are common to both the login and batch environments:

Common Environment Variables
Variable Description
$ARCHIVE_HOME Your directory on the archive server.
$ARCHIVE_HOST The host name of the archive server.
$BC_HOST The generic (not node specific) name of the system.
$CC The currently selected C compiler. This variable is automatically updated when a new compiler environment is loaded.
$CENTER Your directory on the Center-Wide File System (CWFS).
$COST_HOME This variable contains the path to the base directory of the default installation of the Common Open Source Tools (COST) installed on a particular compute platform. (See BC policy FY13-01 for COST details.)
$CSI_HOME The directory containing the following list of heavily used application packages: ABAQUS, Accelrys, ANSYS, CFD++, Cobalt, EnSight, Fluent, GASP, Gaussian, LS-DYNA, MATLAB, and TotalView, formerly known as the Consolidated Software Initiative (CSI) list. Other application software may also be installed here by our staff.
$CXX The currently selected C++ compiler. This variable is automatically updated when a new compiler environment is loaded.
$DAAC_HOME The directory containing DAAC-supported visualization tools: ParaView, VisIt, EnSight, and ezViz.
$F77 The currently selected Fortran 77 compiler. This variable is automatically updated when a new compiler environment is loaded.
$F90 The currently selected Fortran 90 compiler. This variable is automatically updated when a new compiler environment is loaded.
$HOME Your home directory on the system.
$JAVA_HOME The directory containing the default installation of JAVA.
$KRB5_HOME The directory containing the Kerberos utilities.
$PET_HOME The directory containing the tools formerly installed and maintained by the PETTT staff. This variable is deprecated and will be removed from the system in the future. Certain tools will be migrated to $COST_HOME, as appropriate.
$PROJECTS_HOME A common directory where group-owned and supported applications and codes may be maintained for use by members of a group. Any project may request a group directory under $PROJECTS_HOME.
$SAMPLES_HOME The Sample Code Repository. This is a collection of sample scripts and codes provided and maintained by our staff to help users learn to write their own scripts. There are a number of ready-to-use scripts for a variety of applications.
$WORKDIR Your work directory on the local temporary file system (i.e., local high-speed disk).
4.3.2. Batch-Only Environment Variables

In addition to the variables listed above, the following variables are automatically set only in your batch environment. That is, your batch scripts will be able to see them when they run. These variables are supplied for your convenience and are intended for use inside your batch scripts.

Batch-Only Environment Variables
Variable Description
$BC_CORES_PER_NODE The number of cores per node for the compute node on which a job is running.
$BC_MEM_PER_NODE The approximate maximum user-accessible memory per node (in integer MBytes) for the compute node on which a job is running.
$BC_MPI_TASKS_ALLOC The number of MPI tasks allocated for a job.
$BC_NODE_ALLOC The number of nodes allocated for a job.

4.4. Modules

Software modules are a very convenient way to set needed environment variables and include necessary directories in your path so that commands for particular applications can be found. Spirit uses "modules" to initialize your environment with system commands and libraries and compiler suites,

A number of modules are loaded automatically as soon as you log in. To see the modules which are currently loaded, run "module list". To see the entire list of available modules, run "module avail". You can modify the configuration of your environment by loading and unloading modules. For complete information on how to do this, see the Modules User Guide.

4.5. Archive Usage

All of our HPC systems share an online Mass Storage Archival system (MSAS) that currently has more than 43 TBytes of Tier 1 archival storage (disk cache) and 6 PBytes of Tier 2 high-speed archival storage utilizing a robotic tape library. Every user is given an account on the MSAS.

Kerberized login and ftp are allowed into the MSAS system. Locally developed utilities may be used to transfer files to and from the MSAS as well as to create and delete directories, rename files, and list directory contents. For convenience, the environment variable $ARCHIVE_HOME can be used to reference your MSAS archive directory when using archive commands. The command getarchome can be used to display the value of $ARCHIVE_HOME for any user.

4.5.1. Archival Command Synopsis

A synopsis of the main archival utilities is listed below. For information on additional capabilities, see the Archive User Guide or read the online man pages that are available on each system. These commands are non-Kerberized and can be used in batch submission scripts if desired.

Copy one or more files from the MSAS:

archive get [-C path] [-s] file1 [file2...]

List files and directory contents on the MSAS:

archive ls [lsopts] [file/dir ...]

Create directories on the MSAS:

archive mkdir [-C path] [-m mode] [-p] [-s] dir1 [dir2 ...]

Copy one or more files to the MSAS:

archive put [-C path] [-D] [-s] file1 [file ...]

5. Program Development

5.1. Programming Models

Spirit supports three base programming models: Message Passing Interface (MPI), Shared-Memory (SMP, SHMEM), and Open Multi-Processing (OpenMP). A Hybrid MPI/OpenMP programming model is also supported. MPI and SHMEM are examples of the message- or data-passing models, while OpenMP only uses shared memory on a node by spawning threads.

5.1.1. Message Passing Interface (MPI)

SGI's Message Passing Toolkit (MPT) supports the MPI 2.2 standard, as documented by the MPI Forum.

MPI establishes a practical, portable, efficient, and flexible standard for message passing that makes use of the most attractive features of a number of existing message-passing systems, rather than selecting one of them and adopting it as the standard. See "man mpi" for additional information.

A copy of the MPI 2.2 Standard, in PDF format, can be found at

When creating an MPI program on Spirit, ensure that the following actions are taken:

  • Make sure the Message Passing Toolkit (module mpt) is loaded. To check this, run the "module list" command. If the mpt module is not listed, use the command, "module load mpt".
  • Make sure the source code includes one of the following lines :

    INCLUDE "mpif.h" ## for Fortran, or
    #include <mpi.h> ## for C/C++

To compile an MPI program, use the following examples:

ifort -o mpi_program mpi_program.f -lmpi ## for Fortran
icc -o mpi_program mpi_program.c -lmpi ## for C
icpc -o mpi_program mpi_program.c -lmpi -lmpi++ ## for C++

To run an MPI program within a batch script, use the following command:

mpiexec_mpt ./mpi_program [user_arguments]

The mpiexec_mpt command launches executables across a set of compute nodes allocated to your job and, by default, utilizes all cores and nodes available to your job. When each member of the parallel application has exited, mpiexec_mpt exits.

A common concern for MPI users is the need for more memory for each process. By default, one MPI process is started on each core of a node. This means that on Spirit, the available memory on the node is split sixteen ways. To allow an individual process to use more of the node's memory, you need to start fewer processes on that node. To accomplish this, the user must request more nodes from PBS, but only run on a certain number of them. For example, the following select statement requests 8 nodes, with 16 cores per node, but only uses 8 of those cores for MPI processes:

#PBS -l select=8:ncpus=16:mpiprocs=8

To tell the system to only start your job across those 8 cores, the following mpiexec_mpt command can be used:

mpiexec_mpt -np ${BC_MPI_TASKS_ALLOC} ./mpi_program [user_arguments]

For more information about mpiexec_mpt, type "man mpiexec_mpt".

5.1.2. Shared Memory (SHMEM)

These logically shared, distributed-memory access routines provide high-performance, high-bandwidth communication for use in highly parallelized scalable programs. The SHARED MEMORY data-passing library routines are similar to the MPI library routines: they pass data between cooperating parallel processes. The SHMEM data-passing routines can be used in programs that perform computations in separate address spaces and that explicitly pass data to and from different processes in the program.

The SHMEM routines minimize the overhead associated with data-passing requests, maximize bandwidth, and minimize data latency. Data latency is the length of time between a process initiating a transfer of data and that data becoming available for use at its destination.

SHMEM routines support remote data transfer through put operations that transfer data to a different process and get operations that transfer data from a different process. Other supported operations are work-shared broadcast and reduction, barrier synchronization, and atomic memory updates. An atomic memory operation is an atomic read and update operation, such as a fetch and increment, on a remote or local data object. The value read is guaranteed to be the value of the data object just prior to the update. See "man intro_shmem" for details on the SHMEM library.

When creating a SHMEM program on Spirit, ensure that the following actions are taken:

  • Make sure the Message Passing Toolkit (module mpt) is loaded. To check this, run the "module list" command. If the mpt module is not listed, use the command, "module load mpt" to load it.
  • The source code includes one of the following lines:

    INCLUDE 'mpp/shmem.fh' ## for Fortran, or
    #include <mpp/shmem.h> ## for C

  • The compile command includes an option to reference the SHMEM library.

To compile a SHMEM program, use the following examples:

ifort -o shmem_program shmem_program.f90 -lsma -lmpi ## for Fortran
icc -o shmem_program shmem_program.c -lsma -lmpi ## for C
ipcp -o shmem_program shmem_program.c -lsma -lmpi++ ## for C++

To launch a SHMEM program, use the mpiexec_mpt command, as follows:

mpiexec_mpt -np N ./shmem_program [user_arguments]

where N is the number of processes being started, with each process utilizing one core. For more information about mpiexec_mpt, type "man mpiexec_mpt".

5.1.3. Open Multi-Processing (OpenMP)

OpenMP is an application programming interface (API) that supports multi-platform shared memory multiprocessing programming in C, C++ and Fortran. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior. OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications.

When creating an OpenMP program on Spirit, take the following actions:

  • If using OpenMP functions (for example, omp_get_wtime), ensure that the source code includes the line, "USE omp_lib", or includes one of the following:

    INCLUDE 'omp.h' ## for Fortran, or
    #include <omp.h> ## for C
  • Ensure that the compile command includes an option to reference the OpenMP library.

To compile an OpenMP program, use the following examples:

For C codes:

icc -openmp -o OpenMP_program OpenMP_program.c ## Intel
cc -fopenmp -o OpenMP_program OpenMP_program.c ## GNU

For Fortran codes:

ifort -openmp -o OpenMP_program OpenMP_program.f ## Intel
gfortran -fopenmp -o OpenMP_program -fopenmp OpenMP_program.f ## GNU

To run an OpenMP program within a batch script, you also need to set the $OMP_NUM_THREADS environment variable to the number of threads in the team. For example:

./OpenMP_program [user_arguments]

In the example above, the application starts OpenMP_program on one node and spawns a total of sixteen threads. Since Spirit has sixteen cores per compute node, this yields 1 thread per core.

5.1.4. Hybrid MPI/OpenMP

An application built with the hybrid model of parallel programming can run on Spirit using both OpenMP and Message Passing Interface (MPI).

When creating a hybrid (MPI/OpenMP) program on Spirit, follow the instructions in the MPI and OpenMP sections above for creating your program. Then use the compilation instructions for OpenMP.

To run a hybrid program within a batch script, set $OMP_NUM_THREADS equal to the number of threads in the team. Then launch your program using mpiexec_mpt as follows:

mpiexec_mpt -np N omplace ./mpi_program [user_arguments]

Where N is the number of MPI tasks.

5.2. Available Compilers

The following compiler suites are available on Spirit:

Compiler Commands
Command Vendor Language Serial/Parallel
gcc GNU C Serial/Parallel
g++ GNU C++ Serial/Parallel
gfortran GNU Fortran Serial/Parallel
icc Intel C Serial/Parallel
icpc Intel C++ Serial/Parallel
ifort Intel Fortran Serial/Parallel
5.2.1. GCC Compiler Programming Environment

The GCC Compiler Programming Environment can be accessed by loading the module gnu-compilers/[version_level].

module load gnu-compilers/[version_level]

5.2.2. Intel Compiler Programming Environment

The Intel Compiler Programming Environment can be accessed by loading the module intel-compilers/[version_level].

module load intel-compilers/[version_level]

5.3. Relevant Modules

By default, Spirit loads the Intel compiler and SGI MPT environments for you.

For more information on using modules, see the Modules User Guide.

5.4. Libraries

5.4.1. Intel Math Kernel Libraries (Intel MKL)

Spirit provides the Intel Math Kernel Libraries (Intel MKL), a set of numerical routines tuned specifically for Intel platform processors. The routines, which are available via both FORTRAN and C interfaces, include:

  • LAPACK plus BLAS (Levels 1, 2, and 3)
  • ScaLAPACK plus PBLAS (Levels 1, 2, and 3)
  • Fast Fourier Transform (FFT) routines for single-precision, double-precision, single-precision complex, and double-precision complex data types
  • Discrete Fourier Transforms (DFTs)
  • Fast Math and Fast Vector Library
  • Vector Statistical Library Functions (VSL)
  • Vector Transcendental Math Functions (VML)

The MKL is loaded by default with any Intel compiler module.

Linking to the Intel Math Kernel Libraries can be complex and is beyond the scope of this introductory guide. Documentation explaining the full feature set along with instructions for linking can be found at the Intel Math Kernel Library documentation page.

Intel also makes a link advisor available to assist users with selecting proper linker and compiler options:

5.4.2. Additional Math Libraries

There is also an extensive set of Math libraries available in the $PET_HOME/MATH directory on Spirit. Information about these libraries may be found on the Baseline Configuration Web site at BC Policy FY06-01.

5.5. Debuggers

5.5.1. gdb

The GNU Project Debugger (gdb) is a source level debugger that can be invoked either with a program for execution or a running process id. To launch your program under gdb for debugging, use:

gdb a.out corefile

To attach gdb to a program that is already executing on this node, use the following command:

gdb a.out pid

For more information, the GDB manual can be found at

5.5.2. TotalView

TotalView is a debugger that supports threads, MPI, OpenMP, C/C++, and Fortran, mixed-language codes, advanced features like on-demand memory leak detection, other heap allocation debugging features, and the Standard Template Library Viewer (STLView). Unique features like dive, a wide variety of breakpoints, the Message Queue Graph/Visualizer, powerful data analysis, and control at the thread level are also available.

Follow these steps to use TotalView on Spirit via a UNIX X-Windows interface:

  1. Ensure that an X server is running on your local system. Linux users will likely have this by default, but MS Windows users will need to install a third party X Windows solution. There are various options available.
  2. For Linux users, connect to Spirit using ssh -Y. Windows users will need to use PuTTY with X11 forwarding enabled (Connection->SSH->X11->Enable X11 forwarding).
  3. Compile your program on Spirit with the "-g" option.
  4. Submit an interactive job:

    qsub -l ncpus=4 -l walltime=00:30:00 -q debug -X -I

  5. After a short while the following message will appear:

    qsub: waiting for job NNNNNNNN to start
    qsub: job NNNNNNN ready

  6. You are now logged into an interactive batch session.
  7. Load the TotalView module:

    module load totalview

  8. Start program execution:

    mpiexec_mpt -tv -np 4 ./my_mpi_prog.exe arg1 arg2...

  9. After a short delay, the TotalView windows will pop up. Click "GO" to start program execution.

For more information on using TotalView, see the TotalView Documentation page.

5.6. Code Profiling and Optimization

Profiling is the process of analyzing the execution flow and characteristics of your program to identify sections of code that are likely candidates for optimization, which increases the performance of a program by modifying certain aspects for increased efficiency.

5.6.1. SGI perfcatch

SGI perfcatch will run an MPI or SHMEM program with a wrapper profiling library that prints communication and synchronization call profiling information to a summary file upon program completion.

To use perfcatch, insert the perfcatch command in front of the executable name:

mpiexec_mpt -np N perfcatch ./mpi_program [user_arguments]

Where N is the number of MPI tasks.

Man pages are available for perfcatch. Additional information can be found in Chapter 9 of SGI's Message Passing Toolkit User Guide.

Additional profiling options are available. See "man perfcatch" for additional instrumentation options.

5.6.2. Additional Profiling Tools

There is also a set of profiling tools available in the $PET_HOME/pkgs directory on Spirit. Information about these tools may be found on the Baseline Configuration Web site at BC Policy FY13-01.

5.6.3. Program Development Reminders

If an application is not programmed for distributed memory, then only the cores on a single node can be used. This is limited to 16 cores on Spirit.

Keep the system architecture in mind during code development. For instance, if your program requires more memory than is available on a single node, then you will need to parallelize your code so that it can function across multiple nodes.

5.6.4. Performance Optimization Methods

Optimization generally increases compilation time and executable size, and may make debugging difficult. However, it usually produces code that runs significantly faster. The optimizations that you can use will vary depending on your code and the system on which you are running.

Note: Before considering optimization, you should always ensure that your code runs correctly and produces valid output.

In general, there are four main categories of optimization:

  • Global Optimization
  • Loop Optimization
  • Interprocedural Analysis and Optimization(IPA)
  • Function Inlining
Global Optimization

A technique that looks at the program as a whole and may perform any of the following actions:

  • Performed on code over all its basic blocks
  • Performs control-flow and data-flow analysis for an entire program
  • Detects all loops, including those formed by IF and GOTOs statements and performs general optimization.
  • Constant propagation
  • Copy propagation
  • Dead store elimination
  • Global register allocation
  • Invariant code motion
  • Induction variable elimination
Loop Optimization

A technique that focuses on loops (for, while, etc.,) in your code and looks for ways to reduce loop iterations or parallelize the loop operations. The following types of actions may be performed:

  • Vectorization - rewrites loops to improve memory access performance. Some compilers may also support automatic loop vectorization by converting loops to utilize low-level hardware instructions and registers if they meet certain criteria.
  • Loop unrolling - (also known as "unwinding") replicates the body of loops to reduce loop branching overhead and provide better opportunities for local optimization.
  • Parallelization - divides loop operations over multiple processors where possible.
Interprocedural Analysis and Optimization (IPA)

A technique that allows the use of information across function call boundaries to perform optimizations that would otherwise be unavailable.

Function Inlining

Function Inlining is a technique that seeks to reduce function call and return overhead. It is:

  • Used with functions that are called numerous times from relatively few locations.
  • Allows a function call to be replaced by a copy of the body of that function.
  • May create opportunities for other types of optimization
  • May not be beneficial.

Improper use of this form of optimization may increase code size and actually result in less efficient code.

6. Batch Scheduling

6.1. Scheduler

The Portable Batch System (PBS) is currently running on Spirit. It schedules jobs and manages resources and job queues, and can be accessed through the interactive batch environment or by submitting a batch request. PBS is able to manage both single and multiprocessor jobs.

6.2. Queue Information

The following table describes the PBS queues available on Spirit:

Queue Descriptions and Limits
Priority Queue
Max Wall
Clock Time
Max Cores
Per Job
Highest urgent Urgent 168 Hours 36720 Jobs belonging to DoD HPCMP Urgent Projects.
Down Arrow for decreasing priority debug Debug 1 Hour
1/2 Hour
User testing;In the debug queue, you may use 736 cores for 1 hour or 1472 cores for 1/2 hour.
high High 168 Hours 36720 Jobs belonging to DoD HPCMP High Priority Projects.
challenge Challenge 168 Hours 36720 Jobs belonging to DoD HPCMP Challenge Projects.
standard Standard 168 Hours 36720 Standard jobs.
transfer N/A 12 Hours 1 Data transfer for user jobs.
Lowest background Background 120 Hours 32 Unrestricted Access - no allocation charge.

6.3. Interactive Logins

When you log in to Spirit, you will be running in an interactive shell on a login node. The login nodes provide login access for Spirit and support such activities as compiling, editing, and general interactive use by all users. Please note the AFRL DSRC Use Policy. The preferred method to run resource intensive executions is to use an interactive batch session.

6.4. Interactive Batch Sessions

To use the interactive batch environment, you must first acquire an interactive batch shell. This is done by executing a qsub command with the "-I" option from within the interactive environment. For example,

qsub -l ncpus=# -q queue_name -l walltime=HHH:MM:SS -I

Your batch shell request will be placed in the desired queue and scheduled for execution. This may take a few minutes because of the system load. Once your shell starts, you can run or debug interactive applications, post-process data, etc. This session will also be shared with other users.

At this point, you can launch parallel applications onto your assigned set of compute nodes by using the mpiexec_mpt command. You can also run interactive commands or scripts on this service node, but you should limit your memory and cpu usage.

6.5. Batch Request Submission

PBS batch jobs are submitted via the qsub command. The format of this command is:

qsub [ options ] batch_script_file

qsub options may be specified on the command line or imbedded in the batch script file by lines beginning with "#PBS".

For a more thorough discussion of PBS Batch Submission, see the Spirit PBS Guide.

6.6. Batch Resource Directives

Batch resource directives allow you to specify to PBS how your batch jobs should be run, and what resources your job requires. Although PBS has many directives, you only need to know a few to run most jobs.

The basic syntax of PBS directives is as follows:

#PBS option[[=]value]

where some options may require values to be included. For example, to set the number of cores for your job, you might specify the following:

#PBS -l ncpus=16

The following directives are required for all jobs:

Required PBS Directives
Option Value Description
-A project_ID Name of the project
-q queue_name Name of the queue
-l ncpus=# Number of cores
-l walltime=HHH:MM:SS Maximum wall time

A more complete listing of batch Resource Directives is available in the Spirit PBS Guide.

6.7. Sample Scripts

While it is possible to include all PBS directives at the qsub command-line, the preferred method is to embed the PBS directives within the batch request script using "#PBS". The following is a sample batch script:

  # Declare the project under which this job run will be charged. (required)
  # Users can find eligible projects by typing "show_usage" on the command line.
  #PBS -A Project_ID
  # Request 1 hour of wallclock time for execution (required).
  #PBS -l walltime=01:00:00
  # Request 16 cores (required).
  #PBS -l ncpus=16
  # Submit job to debug queue (required).
  #PBS -q debug
  # Declare a jobname.
  #PBS -N myjob
  # Send standard output (stdout) and error (stderr) to the same file.
  #PBS -j oe
  # Change to the $JOBDIR directory.
  cd $JOBDIR
  # Check MSAS availability. If not available, then wait.
  archive stat -s
  # Retrieve executable program from the MSAS.
  archive get -C $ARCHIVE_HOME/project_name program.exe
  # Retrieve input data file from the MSAS.
  archive get -C $ARCHIVE_HOME/project_name/input
  # Execute a parallel program.
  mpiexec_mpt -np $BC_MPI_TASKS_ALLOC ./my_program < > projA-7.out
  # Check MSAS availability. If not available, then wait.
  archive stat -s
  # Create a new subdirectory on the MSAS.
  archive mkdir -C $ARCHIVE_HOME/project_name output7
  # Transfer output file back to the MSAS.
  archive put -C $ARCHIVE_HOME/project_name/output7 projA-7.out
  # Clean up unneeded files from working storage.
  rm -r projA-7

Additional examples are available in the Spirit PBS Guide and in the Sample Code Repository ($SAMPLES_HOME) on Spirit.

6.8. PBS Commands

The following commands provide the basic functionality for using the PBS batch system:

qsub: Used to submit jobs for batch processing.

qsub [qsub options] my_job_script

qstat: Used to check the status of submitted jobs.

qstat PBS_JOBID #check one job
qstat -u my_user_name #check all of user's jobs

qdel: Used to kill queued or running jobs.


A more complete list of PBS commands is available in the Spirit PBS Guide.

6.9. Advance Reservations

An Advance Reservation Service (ARS) is available on Spirit for reserving cores for use, starting at a specific date/time, and lasting for a specific number of hours. The ARS is accessible via most modern web browsers at Authenticated access is required. An ARS User Guide is available online once you have logged in.

7. Software Resources

7.1. Application Software

A complete listing with installed versions can be found on our software page. The general rule for all COTS software packages is that the two latest versions will be maintained on our systems. For convenience, modules are also available for most COTS software packages.

7.2. Useful Utilities

The following utilities are available on Spirit:

Spirit Utilities
Utility Description
archive Perform basic file-handling operations on the MSAS
mpscp High-performance remote file copy
qpeek Display spooled stdout and stderr for an executing batch job.
qview Display information about batch jobs and queues
show_queues Report current batch queue status, usage, and limits
show_storage Display MSAS allocation and usage by subproject
show_usage Display CPU allocation and usage by subproject Strip DOS end-of-record control characters from a text file

7.3. Sample Code Repository

The Sample Code Repository is a directory that contains examples for COTS batch scripts, building and using serial and parallel programs, data management, and accessing and using serial and parallel math libraries. The $SAMPLES_HOME environment variable contains the path to this area, and is automatically defined in your login environment.

Sample Code Repository on Spirit
Use of the application name resource.
application_namesREADME and list of valid strings for application names intended for use in every PBS script preamble. The HPCMP encourages applications not specifically named in the list to be denoted as "other".
Application-specific examples; interactive job submit scripts; use of the application name resource; software license use.
abaqusInstructions for using the abaqus automatic batch script generator as well a sample input deck and sample script for running abaqus jobs. The abaqus module must be loaded.
accelrysInstructions for using the accelrys automatic batch script generator as well a sample input deck and sample script for running accelrys jobs. The accelrys module must be loaded.
ale3dInstructions, sample PBS job script, and sample input data file for executing the ALE3D application on Spirit. The ale3d module needs to be loaded prior to use.
ansysInstructions for using the ansys automatic batch script generator as well a sample input deck and sample script for running ansys jobs. The ansys module must be loaded.
cart3dA series of examples to learn how to use the CART3D application. Follow the instructions in README.txt. Sample PBS jobs scripts are provided as well as sample input data files. Membership in the Unix group wpcart3d is required.
cfdppInstructions for using the cfd++ automatic batch script generator as well sample input data and sample script for running cfd++ jobs. The cfd++ module must be loaded.
CFXInstructions for using the CFX automatic batch script generator as well a sample input deck and sample script for running CFX jobs. The cfx module must be loaded.
cobaltInstructions for using the cobalt automatic batch script generator. Sample job script for the COBALT application. Tar files for two test cases. The cobalt module must be loaded.
cthInstructions and PBS submission script to execute CTH jobs. The cth module should be loaded prior to use.
espressoInstructions and PBS submission script to execute espresso jobs. The espresso module should be loaded prior to use.
fieldviewInstructions and PBS submission script to execute fieldview jobs. The fieldview module should be loaded prior to use.
fluentInstructions for using the fluent automatic batch script generator. Sample job script for the fluent application. The fluent module must be loaded.
fun3dREADME file with instructions on how to execute the FUN3D application. A PBS job script and example input files in a tar file are provided. Membership in the Unix group fun3d is required to use the tar file.
gamessInstructions for using the gamess automatic batch script generator, input data file, and sample PBS job script for the gamess application. The gamess module should be loaded.
gaspBrief instructions, two sample PBS job scripts, and two sample input data files to execute the GASP application. The GASP module must be loaded to run the application. Membership in the Unix group wpgasp is required to be able to use the input data tar files.
gaspexPBS job script and sample data archive for using the gaspex application. The gaspex module should be loaded.
gaussianInstructions for executing the gaussian PBS job script generation tool to run gaussian on Spirit. Also gaussian PBS job script and input data file.
lsdynaInstructions for using the ls-dyna automatic batch script generator. Sample job script for the LS_DYNA application. The lsdyna module should be loaded.
matlabInstructions to execute an interactive MATLAB job and a .m script to execute in it. A matlab module should be loaded.
ncarInstructions on how to use the NCAR Graphics tool. The appropriate ncar module must be loaded beforehand.
openfoamSample PBS job scripts to execute the OPENFOAM application. Instructions on using OPENFOAM are in the scripts. In some cases, the openfoam module must be loaded.
sierraInstructions on how to run the SIERRA application. Membership in the sierra group is required to use sierra.
sqlite3Instructions on how to use the SQLITE3 database tool.
starccm+Instructions to use "submit_starccm" to create a PBS job script for starccm+, plus input data files PBS job scripts that have already been generated. One of the starccm modules should be loaded prior to use.
subversionREADME presenting instructions and other information on how to use the SUBVERSION tool to organize software versions.
vaspSample input file and PBS job script for the VASP application.
Archiving and retrieving files; Lustre striping; file searching; $WORKDIR use.
MPSCP_to_DMSInstructions and sample scripts on using the mpscp utility for transferring files to/from file archive.
OST_StripesInstructions and examples for striping large files on the Lustre file systems.
Transfer_QueuePBS batch script examples for data transfer using the transfer queue.
Transfer_Queue_with_Archive_CommandsExample and instructions on recommended best practice to stage data from mass storage using the transfer queue prior to job execution, then processing using that data, then passing output data back to mass storage using the transfer queue again.
MPI, OpenMP, and hybrid examples; single-core jobs; large memory jobs; running multiple applications within a single batch job.
Hybrid_ExampleSample Fortran and C codes and makefile for compiling hybrid MPI/OpenMP codes, and sample scripts for running hybrid MPI/OpenMP jobs.
mpicInstructions for running multiple serial processes on multiple cores.
Mix_Serial_with_ParallelDemonstration of how to use $PBS_NODEFILE to set several host lists to run serial tasks and parallel tasks on different nodes in the same PBS job. SGI's MPT and mpirun must be used.
MPI_ExampleSample code and makefile for compiling MPI code to run on Spirit, and sample script for running MPI jobs.
Multiple_exec_one_communicatorExample using 3 binaries that shows how to compile and execute a heterogeneous application on Spirit. SGI's MPT and mpirun must be used.
Multiple_ParallelDemonstration of how to set up and run multiple MPI tasks on different nodes in the same PBS job. SGI's MPT and mpirun must be used.
OpenMP_ExampleSample C code and makefile for compiling OpenMP code to run on Spirit, and sample scripts for running OPenMP jobs.
Serial_processing_1C and Fortran serial program examples and sample scripts for running multiple instances of a serial program across nodes using ssh.
Serial_processing_2Fortran serial program example and sample scripts for running multiple serial compute tasks simultaneously across cores of a compute node.
Basic code compilation; debugging; use of library files; static vs. dynamic linking; Makefiles; Endian conversion.
BLACS_ExampleSample BLACS Fortran program, compile script and PBS submission script. The BLACS are from Netlib's ScaLAPACK library in $PET_HOME.
Core_FilesInstructions and source code for viewing core files with Totalview and gdb. This sample uses the Gnu compilers.
ddtInstructions and sample programs for using the DDT debugger.
Endian_ConversionText file discussing what to do to be able to use binary data generated on a non-X86_64 platform.
Memory_UsagePresents a routine callable from Fortran or C for determining how much memory a process is using.
MKL_BLACSSample BLACS Fortran program, compile script and PBS submission script. The BLACS are from Intel's Math Kernel Library (MKL).
MKL-ScaLAPACK_ExampleSample ScaLAPACK Fortran program, compile script and PBS job script. The ScaLAPACK solver, BLACS communication, and supporting LAPACK and BLAS routines are all from Intel's MKL.
MPI_CompilationDiscussion of how to compile MPI codes on Spirit using Intel and Gnu compilers and linking in Intel's IMPI or SGI's MPT. Includes notes on support for each MPI by the available compilers.
ScaLAPACK_ExampleSample ScaLAPACK Fortran program, compile script and PBS submission scripts. The linear solver routines are from Netlib's ScaLAPACK library in $PET_HOME. The LAPACK routines are from Netlib's LAPACK library in $PET_HOME, and the BLAS are from Netlib's LAPACK library in $COST_HOME.
Timers_FortranSerial Timers using Fortran Intrinsics f77 and f90/95.
Use of modules; customizing the login environment; use of common environment variables to facilitate portability of work between systems.
modulesSample README, module description file, and module template for creation of software modules on Spirit.
Basic batch scripting; use of the transfer queue; job arrays; job dependencies; Secure Remote Desktop; job monitoring.
Hybrid_ExampleSample Fortran and C codes and makefile for compiling hybrid MPI/OpenMP codes, and sample scripts for running hybrid MPI/OpenMP jobs.
Job_Array_ExampleSample code to generate binary and data and job script for using job arrays.
MPI_ExampleSample code and makefile for compiling MPI code to run on Spirit, and sample script for running MPI jobs.
OpenMP_ExampleSample C code and makefile for compiling OpenMP code to run on Spirit, and sample scripts for running OPenMP jobs.
Transfer_QueuePBS batch script examples for data transfer using the transfer queue.

8. Links to Vendor Documentation

SGI Documentation Home:
SGI Message Passing Toolkit User Guide

Red Hat Home:
Novell SUSE Linux Enterprise Server:

GNU Home:
GNU Compiler:
Intel Compiler Documentation: