Skip Nav

Appro Shared Memory System (Lancer) User Guide

Table of Contents

1. Introduction

1.1. Document Scope and Assumptions

This document provides an overview and introduction to the use of the Appro Shared Memory system (Lancer) located at the AFRL DSRC, along with a description of the specific computing environment on Lancer. The intent of this guide is to provide information to enable the average user to perform computational tasks on the system. To receive the most benefit from the information provided here, you should be proficient in the following areas:

  • Use of the UNIX operating system
  • Use of an editor (e.g., vi or emacs)
  • Remote usage of computer systems via network or modem access
  • A selected programming language and its related tools and libraries

1.2. Policies to Review

Users are expected to be aware of the following policies for working on Lancer.

1.2.1. Data Backup

Users are expected to provide their own data backup. Administrators will not backup any user data. Files not archived by the user within 30 days will be deleted and cannot be retrieved.

1.3. Obtaining Accounts

Authorized DOD and Contractor personnel may request an account on Lancer by submitting a proposal to the AFRL DSRC via email to sp-proposal@ccac.hpc.mil. The proposal should include the following information:

  • HPC experience and level of required support
  • Project suitability for a Shared Memory system
  • Project contribution to the DoD mission and/or HPC technical advancement
  • Proposed workload

Direct any questions regarding this non-allocated system to sp-proposal@ccac.hpc.mil.

1.4. Requesting Assistance

The Consolidated Customer Assistance Center (CCAC) is available to help users with unclassified problems, issues, or questions. Analysts are on duty 8:00 a.m. - 11:00 p.m. Eastern, Monday - Friday (excluding Federal holidays).

For more detailed contact information, please see our Contact Page.

2. System Configuration

2.1. System Summary

Lancer is an Appro GreenBlade system. The login and compute nodes are populated with Intel Xeon E5-2670 Sandy Bridge processors. Lancer uses 10 Gigabit Ethernet as its high-speed network for I/O traffic.

Lancer has 192 TBytes of Xyratex Lustre storage and uses ScaleMP vSMP Foundation software to aggregate its 160 compute nodes into separate virtual images. Each compute node contains two 8-core Sandy Bridge Intel Xeon E5-2670 processors and 128 GBytes of DDR3 memory. Lancer has 192 TBytes (formatted) of disk storage.

Configuration of the virtual images is shown below and may change over time.

Lancer is intended as a batch-scheduled HPC system. Its login nodes are not to be used for large computational (e.g. memory, I/O, long executions) work. All executions that require large amounts of system resources must be sent to the compute nodes by batch job submission.

Node Configuration
Login Nodes Compute Nodes
Total Nodes 2 160
Operating System RHEL 6 RHEL 6
Virtualization/Hypervisor n/a ScaleMP vSMP Foundation
Cores/Node 16 16
Core Type Intel Xeon E5-2670
Sandy Bridge
Intel Xeon E5-2670
Sandy Bridge
Core Speed 2.6 GHz 2.6 GHz
Memory/Node 64 GBytes 128 GBytes
Accessible Memory/Node 62 GBytes 30 GBytes
Memory Model Shared on node. Shared on node.
Distributed across cluster.
Virtual Machine 1, 3, & 4 Virtual Machine 2
Compute Nodes Compute Nodes
Total Nodes 32 64
Total Cores 512 1024
Total Memory 4 TBytes 8 TBytes
Total Accessible Memory 3.3 TBytes 6.5 TBytes
Accessible Memory/Node 6.6 GBytes 6.6 GBytes
File Systems on Lancer
Path Capacity Type
/home
($HOME)
Lustre
/workspace
($WORKDIR)
192 TBytes Lustre
/p/cwfs
/p/cwfs1 thru /p/cwfs6
($CENTER)
800 TBytes PanFS

2.2. Processor

Lancer uses 2.6-GHz Intel Xeon E5-2670 Sandy Bridge processors on its login and compute nodes.

2.3. Memory

Each login node contains 64 GBytes of main memory. All memory and cores on the node are shared among all users who are logged in. Therefore, users should not use more than 8 GBytes of memory at any one time.

Each compute node contains 6.6 GBytes of user-accessible shared memory.

2.4. Operating System

The operating system on Lancer is RHEL 6.2.

2.5. Virtualization/Hypervisor

Lancer uses ScaleMP vSMP Foundation to maintain multiple virtual systems on one physical system. Each virtual image (v1, v2, v3, and v4) has a different configuration to handle a user's shared memory needs.

Lancer uses ScaleMP vSMP Foundation software to aggregate its 160 compute nodes into separate virtual images. Each compute node contains two 8-core Sandy Bridge Intel Xeon E5-2670 processors and 128 GBytes of DDR3 memory.

2.6. File Systems

Lancer has the following file systems available for user storage:

/home

This path is also accessible using the env variable $HOME. $HOME is used for day-to-day items (i.e. small binaries, scripts) and should not be used to run jobs or as an archival location.

/workspace

This path is also accessible using $WORKDIR.

/p/cwfs1 thru /p/cwfs6

This path is directed to the Center-Wide File system, which is meant for short-term storage (no longer than 30 days).

3. Accessing the System

3.1. Kerberos

A Kerberos client kit must be installed on your desktop to enable you to get a Kerberos ticket. Kerberos is a network authentication tool that provides secure communication by using secret cryptographic keys. Only users with a valid HPCMP Kerberos authentication can gain access to Lancer. More information about installing Kerberos clients on your desktop can be found at HPC Centers: Kerberos & Authentication.

3.2. Logging In

3.2.1. Kerberized SSH

% ssh lancer-l01.afrl.hpc.mil

3.3. File Transfers

File transfers to DSRC systems (except transfers to the local archive system) must be performed using Kerberized versions of the following tools: scp, ftp, sftp, and mpscp.

4. User Environment

4.1. User Directories

4.1.1. Home Directory

This is the directory referenced by the $HOME environment variable.

4.1.2. Work Directory

This is the directory referenced by the $WORKDIR environment variable.

4.1.3. Center Directory

The Center-Wide File System (CWFS) provides file storage that is accessible from Lancer's login nodes, and from all nodes of the Utility Server. The CWFS permits file transfer between Lancer and the Utility Server using simple copy commands. Each user has their own directory in the CWFS. The name of your CWFS directory may vary between machines and between centers, but the environment variable $CENTER will always refer to this directory.

The example below shows how to copy a file from your work directory on Lancer to your work directory on the Utility Server using the CWFS. The CWFS ($CENTER) is mounted on both Lancer and the Utility Server.

While logged into Lancer, copy your file from your Lancer work directory to the CWFS.

% cp $WORKDIR/filename $CENTER

Then, log into the Utility Server and copy your file from the CWFS to your Utility Server work directory.

% cp $CENTER/filename $WORKDIR

For more information about using the Utility Server, see the Utility Server Quick Reference and the Utility Server User Guide.

4.2. Shells

The following shells are available on Lancer: csh, bash, ksh, tcsh, and sh. The user's default shell is bash.

4.3. Environment Variables

A number of environment variables are provided by default on all HPCMP high performance computing (HPC) systems. We encourage you to use these variables in your scripts where possible. Doing so will help to simplify your scripts and reduce portability issues if you ever need to run those scripts on other systems.

4.3.1. Login Environment Variables
Common Environment Variables
Variable Description
$ARCHIVE_HOME Your directory on the archive server.
$ARCHIVE_HOST The host name of the archive server.
$CENTER Your directory on the Center-Wide File System (CWFS).
$CSI_HOME The directory containing the following list of heavily used application packages.
$HOME Your home directory on the system.
$JAVA_HOME The directory containing the default installation of JAVA.
$WORKDIR Your work directory on the local temporary file system (i.e., local high-speed disk).
$CPU Identifies the type of CPU on the system.
$ACCOUNT Identifies current account information.

4.4. Modules

Software modules are a very convenient way to set needed environment variables and include necessary directories in your path so that commands for particular applications can be found. Lancer uses "modules" to initialize your environment with system commands and libraries and compiler suites,

A number of modules are loaded automatically as soon as you log in. To see the modules which are currently loaded, run "module list". To see the entire list of available modules, run "module avail". You can modify the configuration of your environment by loading and unloading modules. For complete information on how to do this, see the Modules User Guide.

4.5. Archive Usage

Files not archived by the user within 30 days will be deleted and cannot be retrieved.

All of our HPC systems share an online Mass Storage Archival system (MSAS) that currently has more than 43 TBytes of Tier 1 archival storage (disk cache) and 6 PBytes of Tier 2 high-speed archival storage utilizing a robotic tape library. Every user is given an account on the MSAS.

Kerberized login and ftp are allowed into the MSAS system. Locally developed utilities may be used to transfer files to and from the MSAS as well as to create and delete directories, rename files, and list directory contents. For convenience, the environment variable $ARCHIVE_HOME can be used to reference your MSAS archive directory when using archive commands. The command getarchome can be used to display the value of $ARCHIVE_HOME for any user.

4.5.1. Archival Command Synopsis

A synopsis of the main archival utilities is listed below. For information on additional capabilities, see the Archive User Guide or read the online man pages that are available on each system. These commands are non-Kerberized and can be used in batch submission scripts if desired.

Copy one or more files from the MSAS:

archive get [-C path] [-s] file1 [file2...]

List files and directory contents on the MSAS:

archive ls [lsopts] [file/dir ...]

Create directories on the MSAS:

archive mkdir [-C path] [-m mode] [-p] [-s] dir1 [dir2 ...]

Copy one or more files to the MSAS:

archive put [-C path] [-D] [-s] file1 [file ...]

5. Program Development

5.1. Programming Models

Lancer supports two base programming models: Message Passing Interface (MPI) and Open Multi-Processing (OpenMP).

5.1.1. Open Multi-Processing (OpenMP)

OpenMP is an application programming interface (API) that supports multi-platform shared memory multiprocessing programming in C, C++ and Fortran. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior. OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications.

When creating an OpenMP program on Lancer, ensure that the following actions are taken:

  • If using OpenMP functions (for example, omp_get_wtime), ensure that the source code includes the line, "USE omp_lib".

To compile an OpenMP program, use the following examples:

For C codes:

icc -o OpenMP_program -mp=nonuma OpenMP_program.c ## PGI
icc -o OpenMP_program -mp OpenMP_program.c ## PathScale
cc -o OpenMP_program -fopenmp OpenMP_program.c ## GNU

For Fortran codes:

ifort -o OpenMP_program -mp=nonuma OpenMP_program.f ## PGI
ifort -o OpenMP_program -mp OpenMP_program.f ## PathScale
ifort -Openmp -o OpenMP_program -mp ## Intel
gfortran -o OpenMP_program -fopenmp OpenMP_program.f ## GNU

To run an OpenMP program within a batch script, you may also want to set the $OMP_NUM_THREADS environment variable to the number of threads in the team. For example:

setenv OMP_NUM_THREADS 4

If the number of threads is not set by the user, the default value is "8".

5.2. Available Compilers

At this time, the Intel Compiler Programming Environment is available on Lancer.

5.2.1. Intel Compiler Programming Environment

The Intel Compiler Programming Environment can be accessed by swapping out the default module for the module PrgEnv-intel/[version level].

module swap PrgEnv-cray PrgEnv-intel/[version level]

5.3. Libraries

5.3.1. Gamess

GAMESS is a program for ab initio quantum chemistry. To initiate Gamess, run the following command:

submit_gamess [input_file][ext_file]

For more information on using Gamess, please see the AFRL DSRC Software Page.

5.3.2. Amber

To initiate Amber, run the following command:

submit_sander [options and names for input and output]

For more information on using Amber, please see the AFRL DSRC Software Page.

5.4. Debuggers

5.4.1. DDT

DDT is a graphical debugging tool for scientists and software engineers developing parallel applications. DDT fully supports NVIDIA CUDA with the ability to debug code running on the CPU and GPU. To initiate DDT:

module load DDT/ddt

For more information on using DDT, please see the AFRL DSRC Software Page.

5.4.2. TotalView

TotalView is a debugger that supports threads, MPI, OpenMP, C/C++, and Fortran, mixed-language codes, advanced features like on-demand memory leak detection, other heap allocation debugging features, and the Standard Template Library Viewer (STLView). Unique features like dive, a wide variety of breakpoints, the Message Queue Graph/Visualizer, powerful data analysis, and control at the thread level are also available.

Follow these steps to use TotalView on Lancer via a UNIX X-Windows interface:

  1. Ensure that an X server is running on your local system. Linux users will likely have this by default, but MS Windows users will need to install a third party X Windows solution. There are various options available.
  2. For Linux users, connect to Lancer using ssh -Y. Windows users will need to use PuTTY with X11 forwarding enabled (Connection->SSH->X11->Enable X11 forwarding).
  3. Compile your program on Lancer with the "-g" option.
  4. Submit an interactive job:

    qsub -l ncpus=4 -l walltime=00:30:00 -q debug -X -I

  5. After a short while the following message will appear:

    qsub: waiting for job NNNNNNNN to start
    qsub: job NNNNNNN ready


  6. You are now logged into an interactive batch session.
  7. Load the TotalView module:

    module load totalview

  8. Start program execution:

    mpiexec_mpt -tv -np 4 ./my_mpi_prog.exe arg1 arg2...

  9. After a short delay, the TotalView windows will pop up. Click "GO" to start program execution.

For more information on using TotalView, see the TotalView Documentation page.

6. Batch Scheduling

6.1. Scheduler

The Portable Batch System (PBS) is currently running on Lancer. It schedules jobs and manages resources and job queues, and can be accessed through the interactive batch environment or by submitting a batch request.

6.2. Queue Information

The following table describes the PBS queues available on Lancer:

Queue Descriptions and Limits
Priority Queue
Name
Job
Class
Max Wall
Clock Time
Max Cores
Per Job
Comments
Highest debug Debug 1 Hour 512 User diagnostic jobs
Down Arrow for decreasing priority standard Standard 168 Hours 1280 Non Challenge User Jobs
Lowest transfer n/a 12 Hours 1 File Transfers

6.3. Batch Request Submission

PBS batch jobs are submitted via the qsub command. The format of this command is:

qsub [ options ] batch_script_file

qsub options may be specified on the command line or imbedded in the batch script file by lines beginning with "#PBS".

For a more thorough discussion of PBS batch submission, see the PBS User Guide.

6.4. Batch Resource Directives

A more complete listing of batch resource directives is available in the PBS User Guide.

6.5. Sample Scripts

The following script is for an OpenMP job using one thread per core on a single node and running for 20 hours in the standard queue:

#!/bin/csh
## Required Directives ------------------------------------
#PBS -l select=1:ncpus=8:mpiprocs=8
#PBS -l walltime=20:00:00

## Optional Directives ------------------------------------
#PBS -l job_type=SMP
#PBS -q standard
#PBS -A project_ID
#PBS -N testjob4
#PBS -j oe
#PBS -M my_email@yahoo.com
#PBS -m be

## Execution Block ----------------------------------------
# Environmental Setup
# Change to job-specific subdirectory
cd $WORK_DIR

# Stage input data from archive
archive get -C $ARCHIVE_HOME/my_data_dir "*.dat"

# Copy the executable from $HOME
cp $HOME/my_ompprog.exe a.out

## Launching ----------------------------------------------
# Threads per rank set by OMP_NUM_THREADS or omplace -nt
setenv OMP_NUM_THREADS 8
omplace -vv -nt 8 ./a.out > output_file

## Cleanup ------------------------------------------------
# archive your results
archive put -p -C $ARCHIVE_HOME/my_output_dir *.out output*

6.6. PBS Commands

The following commands provide the basic functionality for using the PBS batch system:

qsub: Used to submit jobs for batch processing.

qsub [qsub options] my_job_script

qstat: Used to check the status of submitted jobs.

qstat PBS_JOBID #check one job
qstat -u my_user_name #check all of user's jobs

qdel: Used to kill queued or running jobs.

qdel PBS_JOBID

A more complete list of PBS commands is available in the PBS User Guide.

7. Links to Vendor Documentation

Appropriate documentation from the vendor can be found in /global/documentation. In addition, the following vendor documentation may be useful.

GNU Home: http://www.gnu.org
GNU Compiler: http://gcc.gnu.org

Fortran Home: http://www.fortran.com