Appro Shared Memory System (Lancer) User Guide
Table of Contents
- 1. Introduction
- 1.1. Document Scope and Assumptions
- 1.2. Policies to Review
- 1.2.1. Data Backup
- 1.3. Obtaining Accounts
- 1.4. Requesting Assistance
- 2. System Configuration
- 2.1. System Summary
- 2.2. Processor
- 2.3. Memory
- 2.4. Operating System
- 2.5. Virtualization/Hypervisor
- 2.6. File Systems
- 3. Accessing the System
- 3.1. Kerberos
- 3.2. Logging In
- 3.2.1. Kerberized SSH
- 3.3. File Transfers
- 4. User Environment
- 4.1. User Directories
- 4.1.1. Home Directory
- 4.1.2. Work Directory
- 4.1.3. Center Directory
- 4.2. Shells
- 4.3. Environment Variables
- 4.3.1. Login Environment Variables
- 4.4. Modules
- 4.5. Archive Usage
- 4.5.1. Archival Command Synopsis
- 5. Program Development
- 5.1. Programming Models
- 5.1.1. Open Multi-Processing (OpenMP)
- 5.2. Available Compilers
- 5.2.1. Intel Compiler Programming Environment
- 5.3. Libraries
- 5.3.1. Gamess
- 5.3.2. Amber
- 5.4. Debuggers
- 5.4.1. DDT
- 5.4.2. TotalView
- 6. Batch Scheduling
- 6.1. Scheduler
- 6.2. Queue Information
- 6.3. Batch Request Submission
- 6.4. Batch Resource Directives
- 6.5. Sample Scripts
- 6.6. PBS Commands
- 7. Links to Vendor Documentation
- 7.1. GNU
1.1. Document Scope and Assumptions
This document provides an overview and introduction to the use of the Appro Shared Memory system (Lancer) located at the AFRL DSRC and a description of the specific computing environment on Lancer. The intent of this guide is to provide information that will enable the average user to perform computational tasks on the system. To receive the most benefit from the information provided here, you should be proficient in the following areas:
- Use of the UNIX operating system
- Use of an editor (e.g., vi or emacs)
- Remote usage of computer systems via network or modem access
- A selected programming language and its related tools and libraries
1.2. Policies to Review
Users are expected to be aware of the following policies for working on Lancer.
1.2.1. Data Backup
Users are expected to provide their own data backup. Administrators will not backup any user data. Files not archived by the user within 30 days will be deleted and cannot be retrieved.
1.3. Obtaining Accounts
Authorized DOD and Contractor personnel may request an account on Lancer by submitting a proposal to the AFRL DSRC via email to firstname.lastname@example.org. The proposal should include the following information:
- HPC experience and level of required support
- Project suitability for a Shared Memory system
- Project contribution to the DoD mission and/or HPC technical advancement
- Proposed workload
Direct any questions regarding this non-allocated system to email@example.com.
1.4. Requesting Assistance
The Consolidated Customer Assistance Center (CCAC) is available to help users with unclassified problems, issues, or questions. Analysts are on duty 8:00 a.m. - 11:00 p.m. Eastern, Monday - Friday (excluding Federal holidays).
For more detailed contact information, please see our Contact Page.
2. System Configuration
2.1. System Summary
Lancer is an Appro GreenBlade system. The login and compute nodes are populated with Intel Xeon E5-2670 Sandy Bridge processors. Lancer uses 10 Gigabit Ethernet as its high-speed network for I/O traffic.
Lancer has 192 TBytes of Xyratex Lustre storage and uses ScaleMP vSMP Foundation software to aggregate its 160 compute nodes into separate virtual images. Each compute node contains two 8-core Sandy Bridge Intel Xeon E5-2670 processors and 128 GBytes of DDR3 memory. Lancer has 192 TBytes (formatted) of disk storage.
Configuration of the virtual images is shown below and may change over time.
Lancer is intended as a batch-scheduled HPC system. Its login nodes are not to be used for large computational (e.g. memory, I/O, long executions) work. All executions that require large amounts of system resources must be sent to the compute nodes by batch job submission.
|Login Nodes||Compute Nodes|
|Operating System||RHEL 6||RHEL 6|
|Virtualization/Hypervisor||n/a||ScaleMP vSMP Foundation|
|Core Type||Intel Xeon E5-2670
|Intel Xeon E5-2670
|Core Speed||2.6 GHz||2.6 GHz|
|Memory/Node||64 GBytes||128 GBytes|
|Accessible Memory/Node||62 GBytes||30 GBytes|
|Memory Model||Shared on node.||Shared on node.
Distributed across cluster.
|Virtual Machine 1, 3, & 4||Virtual Machine 2|
|Comput Nodes||Compute Nodes|
|Total Memory||4 TBytes||8 TBytes|
|Total Accessible Memory||3.3 TBytes||6.5 TBytes|
|Accessible Memory/Node||6.6 GBytes||6.6 GBytes|
/p/cwfs1 thru /p/cwfs6
Lancer uses 2.6-GHz Intel Xeon E5-2670 Sandy Bridge processors on its login and compute nodes.
Each login node contains 64 GBytes of main memory. All memory and cores on the node are shared among all users who are logged in. Therefore, users should not use more than 8 GBytes of memory at any one time.
Each compute node contains 6.6 GBytes of user-accessible shared memory.
2.4. Operating System
The operating system on Lancer is RHEL 6.2.
Lancer uses ScaleMP vSMP Foundation to maintain multiple virtual systems on one physical system. Each virtual image (v1, v2, v3, and v4) has a different configuration to handle a user's shared memory needs.
Lancer uses ScaleMP vSMP Foundation software to aggregate its 160 compute nodes into separate virtual images. Each compute node contains two 8-core Sandy Bridge Intel Xeon E5-2670 processors and 128 GBytes of DDR3 memory.
2.6. File Systems
Lancer has the following file systems available for user storage:
This path is also accessible using the env variable $HOME. $HOME is used for day-to-day items (i.e. small binaries, scripts) and should not be used to run jobs or as an archival location.
This path is also accessible using $WORKDIR.
/p/cwfs1 thru /p/cwfs6
This path is directed to the Center-Wide File system, which is meant for short-term storage (no longer than 30 days).
3. Accessing the System
A Kerberos client kit must be installed on your desktop to enable you to get a Kerberos ticket. Kerberos is a network authentication tool that provides secure communication by using secret cryptographic keys. Only users with a valid HPCMP Kerberos authentication can gain access to Spirit. More information about installing Kerberos clients on your desktop can be found at HPC Centers: Kerberos & Authentication.
3.2. Logging In
3.2.1. Kerberized SSH
% ssh lancer-l01.afrl.hpc.mil
3.3. File Transfers
File transfers to DSRC systems (except transfers to the local archive system) must be performed using Kerberized versions of the following tools: scp, ftp, sftp, and mpscp.
4. User Environment
4.1. User Directories
4.1.1. Home Directory
This is the directory referenced by the $HOME environment variable.
4.1.2. Work Directory
This is the directory referenced by the $WORKDIR environment variable.
4.1.3. Center Directory
The Center-Wide File System (CWFS) provides file storage that is accessible from Lancer's login nodes, and from all nodes of the utility server. The CWFS permits file transfer between Spirit and the utility server using simple copy commands. Each user has their own directory in the CWFS. The name of your CWFS directory may vary between machines and between centers, but the environment variable $CENTER will always refer to this directory.
The example below shows how to copy a file from your work directory on Spirit to your work directory on the utility server using the CWFS. The CWFS ($CENTER) is mounted on both Lancer and the utility server.
While logged into Lancer, copy your file from your Lancer work directory to the CWFS.
% cp $WORKDIR/filename $CENTER
Then, log into the utility server and copy your file from the CWFS to your utility server work directory.
% cp $CENTER/filename $WORKDIR
4.3. Environment Variables
A number of environment variables are provided by default on all HPCMP high performance computing (HPC) systems. We encourage you to use these variables in your scripts where possible. Doing so will help to simplify your scripts and reduce portability issues if you ever need to run those scripts on other systems.
4.3.1. Login Environment Variables
|$ARCHIVE_HOME||Your directory on the archive server.|
|$ARCHIVE_HOST||The host name of the archive server.|
|$CENTER||Your directory on the Center-Wide File System (CWFS).|
|$CSI_HOME||The directory containing the following list of heavily used application packages.|
|$HOME||Your home directory on the system.|
|$JAVA_HOME||The directory containing the default installation of JAVA.|
|$WORKDIR||Your work directory on the local temporary file system (i.e., local high-speed disk).|
|$CPU||Identifies the type of CPU on the system.|
|$ACCOUNT||Identifies current account information.|
Software modules are a very convenient way to set needed environment variables and include necessary directories in your path so that commands for particular applications can be found. Spirit uses "modules" to initialize your environment with system commands and libraries and compiler suites,
A number of modules are loaded automatically as soon as you log in. To see the modules which are currently loaded, run "module list". To see the entire list of available modules, run "module avail". You can modify the configuration of your environment by loading and unloading modules. For complete information on how to do this, see the Modules User Guide.
4.5. Archive Usage
Files not archived by the user within 30 days will be deleted and cannot be retrieved.
All of our HPC systems share an online Mass Storage Archival system (MSAS) that currently has more than 43 TBytes of Tier 1 archival storage (disk cache) and 6 PBytes of Tier 2 high-speed archival storage utilizing a robotic tape library. Every user is given an account on the MSAS.
Kerberized login and ftp are allowed into the MSAS system. Locally developed utilities may be used to transfer files to and from the MSAS as well as to create and delete directories, rename files, and list directory contents. For convenience, the environment variable $ARCHIVE_HOME can be used to reference your MSAS archive directory when using archive commands. The command getarchome can be used to display the value of $ARCHIVE_HOME for any user.
4.5.1. Archival Command Synopsis
A synopsis of the main archival utilities is listed below. For information on additional capabilities, see the Archive User Guide or read the online man pages that are available on each system. These commands are non-Kerberized and can be used in batch submission scripts if desired.
Copy one or more files from the MSAS:
archive get [-C path] [-s] file1 [file2...]
List files and directory contents on the MSAS:
archive ls [lsopts] [file/dir ...]
Create directories on the MSAS:
archive mkdir [-C path] [-m mode] [-p] [-s] dir1 [dir2 ...]
Copy one or more files to the MSAS:
archive put [-C path] [-D] [-s] file1 [file ...]
5. Program Development
5.1. Programming Models
Lancer supports two base programming models: Message Passing Interface (MPI) and Open Multi-Processing (OpenMP).
5.1.1. Open Multi-Processing (OpenMP)
OpenMP is an application programming interface (API) that supports multi-platform shared memory multiprocessing programming in C, C++ and Fortran. It consists of a set of compiler directives, library routines, and environment variables that influence run-time behavior. OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications.
When creating an OpenMP program on Lancer, ensure that the following actions are taken:
- If using OpenMP functions (for example, omp_get_wtime), ensure that the source code includes the line, "USE omp_lib".
To compile an OpenMP program, use the following examples.
For C codes:
icc -o OpenMP_program -mp=nonuma OpenMP_program.c ## PGI
icc -o OpenMP_program -mp OpenMP_program.c ## PathScale
cc -o OpenMP_program -fopenmp OpenMP_program.c ## GNU
For Fortran codes:
ifort -o OpenMP_program -mp=nonuma OpenMP_program.f ## PGI
ifort -o OpenMP_program -mp OpenMP_program.f ## PathScale
ifort -Openmp -o OpenMP_program -mp ## Intel
gfortran -o OpenMP_program -fopenmp OpenMP_program.f ## GNU
To run an OpenMP program within a batch script, you may also want to set the $OMP_NUM_THREADS environment variable to the number of threads in the team. For example:
setenv OMP_NUM_THREADS 4
If the number of threads is not set by the user, the default value is "8".
5.2. Available Compilers
At this time, the Intel Compiler Programming Environment is available on Lancer.
5.2.1. Intel Compiler Programming Environment
The Intel Compiler Programming Environment can be accessed by swapping out the default module for the module PrgEnv-intel/[version level].
module swap PrgEnv-cray PrgEnv-intel/[version level]
GAMESS is a program for ab initio quantum chemistry. To initiate Gamess, run the following command:
For more information on using Gamess, please see the AFRL DSRC Software Page.
To initiate Amber, run the following command:
submit_sander [options and names for input and output]
For more information on using Amber, please see the AFRL DSRC Software Page.
DDT is a graphical debugging tool for scientists and software engineers developing parallel applications. DDT fully supports NVIDIA CUDA with the ability to debug code running on the CPU and GPU. To initiate DDT:
module load DDT/ddt
For more information on using DDT, please see the AFRL DSRC Software Page.
TotalView is a debugger that supports threads, MPI, OpenMP, C/C++, and Fortran, mixed-language codes, advanced features like on-demand memory leak detection, other heap allocation debugging features, and the Standard Template Library Viewer (STLView). Unique features like dive, a wide variety of breakpoints, the Message Queue Graph/Visualizer, powerful data analysis, and control at the thread level are also available.
Follow these steps to use TotalView on Lancer via a UNIX X-Windows interface:
- Ensure that an X server is running on your local system. Linux users will likely have this by default, but MS Windows users will need to install a third party X Windows solution. There are various options available.
- For Linux users, connect to Spirit using ssh -Y. Windows users will need to use PuTTY with X11 forwarding enabled (Connection->SSH->X11->Enable X11 forwarding).
- Compile your program on Spirit with the "-g" option.
- Submit an interactive job:
qsub -l ncpus=4 -l walltime=00:30:00 -q debug -X -I
- After a short while the following message will appear:
qsub: waiting for job NNNNNNNN to start
qsub: job NNNNNNN ready
- You are now logged into an interactive batch session.
- Load the TotalView module:
module load totalview
- Start program execution:
mpiexec_mpt -tv -np 4 ./my_mpi_prog.exe arg1 arg2...
- After a short delay, the TotalView windows will pop up. Click "GO" to start program execution.
For more information on using TotalView, see the TotalView Documentation page.
6. Batch Scheduling
The Portable Batch System (PBS) is currently running on Lancer. It schedules jobs and manages resources and job queues, and can be accessed through the interactive batch environment or by submitting a batch request.
6.2. Queue Information
The following table describes the PBS queues available on Spirit:
|Highest||debug||Debug||1 Hour||512||User diagnostic jobs|
|standard||Standard||168 Hours||1280||Non Challenge User Jobs|
|Lowest||transfer||n/a||12 Hours||1||File Transfers|
6.3. Batch Request Submission
PBS batch jobs are submitted via the qsub command. The format of this command is:
qsub [ options ] batch_script_file
qsub options may be specified on the command line or imbedded in the batch script file by lines beginning with "#PBS".
For a more thorough discussion of PBS Batch Submission, see the PBS User Guide.
6.4. Batch Resource Directives
A complete listing of batch Resource Directives is available in the PBS User Guide.
6.5. Sample Scripts
The following script is for an OpenMP job using one thread per core on a single node and running for 20 hours in the standard queue:
#!/bin/csh ## Required Directives ------------------------------------ #PBS -l select=1:ncpus=8:mpiprocs=8 #PBS -l walltime=20:00:00 ## Optional Directives ------------------------------------ #PBS -l job_type=SMP #PBS -q standard #PBS -A project_ID #PBS -N testjob4 #PBS -j oe #PBS -M firstname.lastname@example.org #PBS -m be ## Execution Block ---------------------------------------- # Environmental Setup # Change to job-specific subdirectory cd $WORK_DIR # Stage input data from archive archive get -C $ARCHIVE_HOME/my_data_dir "*.dat" # Copy the executable from $HOME cp $HOME/my_ompprog.exe a.out ## Launching ---------------------------------------------- # Threads per rank set by OMP_NUM_THREADS or omplace -nt setenv OMP_NUM_THREADS 8 omplace -vv -nt 8 ./a.out > output_file ## Cleanup ------------------------------------------------ # archive your results archive put -p -C $ARCHIVE_HOME/my_output_dir *.out output*
6.6. PBS Commands
The following commands provide the basic functionality for using the PBS batch system:
qsub: Used to submit jobs for batch processing.
qsub [qsub options] my_job_script
qstat: Used to check the status of submitted jobs.
qstat PBS_JOBID #check one job
qstat -u my_user_name #check all of user's jobs
qdel: Used to kill queued or running jobs.
A more complete list of PBS commands is available in the PBS User Guide.
7. Links to Vendor Documentation
Appropriate documentation from the vendor can be found in /global/documentation. In addition, the following vendor documentation may be useful.