Introduction to Linux & HPC

• Who?

  • Unit of Lehigh's Library & Technology Services within the Center for Innovation in Teaching & Learning

• Our Mission

  • We enable Lehigh Faculty, Researchers and Scholars achieve their goals by providing various computational resources; hardware, software, and storage; consulting and training.

• Research Computing Staff

  • Steve Anthony, System Administrator
  • Dan Brashler, CAS Senior Computing Consultant
  • Sachin Joshi, Data Analyst & Visualization Specialist
  • Alex Pacheco, Manager & XSEDE Campus Champion

• Computational Science and Engineering
  • Gain understanding, mainly through the analysis of mathematical models implemented on computers.
  • Construct mathematical models and quantitative analysis techniques, using computers to analyze and solve scientific problems.
  • Typically, these models require large amount of floating-point calculations not possible on desktops and laptops.
  • The field's growth drove the need for HPC and benefited from it.
• HPC
  • High Performance Computing (HPC) is computation at the forefront of modern technology, often done on a supercomputer.
• Supercomputer
  • A supercomputer is a computer at the frontline of current processing capacity, particularly speed of calculation.

• HPC may be the only way to achieve computational goals in a given amount of time
  • Size: Many problems that are interesting to scientists and engineers cannot fit on a PC usually because they need more than a few GB of RAM, or more than a few hundred GB of disk.
  • Speed: Many problems that are interesting to scientists and engineers would take a very long time to run on a PC: months or even years; but a problem that would take a month on a PC might only take a few hours on a supercomputer

• many calculations are carried out simultaneously
• based on principle that large problems can often be divided into smaller ones, which are then solved in parallel
• Parallel computers can be roughly classified according to the level at which the hardware supports parallelism.
  • Multicore computing
  • Symmetric multiprocessing
  • Distributed computing
  • Grid computing
  • General-purpose computing on graphics processing units (GPGPU)

• Traditional Disciplines
  • Science: Physics, Chemistry, Biology, Material Science
  • Engineering: Mechanical, Structural, Civil, Environmental, Chemical
• Non Traditional Disciplines
  • Finance
    • Preditive Analytics
    • Trading
  • Humanities
    • Culturomics or cultural analytics: study human behavior and cultural trends through quantitative analysis of digitized texts, images and videos.

• Linux is an operating system that evolved from a kernel created by Linus Torvalds when he was a student at the University of Helsinki.
• It’s meant to be used as an alternative to other operating systems, Windows, Mac OS, MS-DOS, Solaris and others.
• Linux is the most popular OS used in a Supercomputer
  • From June 2017 Top 500 List

• If you are using a Supercomputer/High Performance Computer for your research, it will be based on a *nix OS.
• It is required/neccessary/mandatory to learn Linux Programming (commands, shell scripting) if your research involves use of High Performance Computing or Supercomputing resources.

• All files are arranged in a hierarchial structure, like an inverted tree.
• The top of the hierarchy is traditionally called root (written as a slash / )

• Path means a position in the directory tree.
• You can use either the relative path or absolute path
• In relative path expression
  • (one dot or period) is the current working directory
  • (two dots or periods) is one directory up
  • You can combine . and .. to navigate the filee system hierarchy.
  • the path is not defined uniquely and does depend on the current path.
  • is unique only if your current working directory is your home directory.
• In absolute path expression
  • the path is defined uniquely and does not depend on the current path
  • /tmp is unique since /tmp is the abolute path

• Linux permits the use of variables, similar to any programming language such as C,C++, Fortran etc
• A variable is a named object that contains data used by one or more applications.
• There are two types of variables, Environment and User Defined and can contain anumber, character or a string of characters.
• Environment Variables provides a simple way to share configuration settings between multiple applications and processes in Linux.
• By Convention, enviromental variables are often named using all uppercase letters.
  • PATH, LD_LIBRARY_PATH, LD_INCLUDE_PATH, TEXINPUTS, etc
• To reference a variable (environment or user defined) prepend $ to the name of the variable
• Its a good practice to protect your variable name within {...} such as ${PATH} when referencing it. (We’ll see an example in a few slides)
  • $PATH, ${LD_LIBRARY_PATH}
• The command printenv will list the current environmental variables.

• Rules for Variable Names
  1. Variable names must start with a letter or underscore
  2. Number can be used anywhere else
  3. DO NOT USE special characters such as @, #, %, $
  4. Case sensitive
• Examples
  • Allowed: VARIABLE, VAR1234able, var name, VAR
  • Not Allowed: 1VARIABLE, %NAME, $myvar, VAR@NAME
• Assigning value to a variable
  • Shell: myvar=somevalue
  • Environment: export name=value
• You can convert any Shell variable to Environment variable using the export command
  • Convert to Environment: export myvar

• man shows the manual for a command or program.
  • man pwd
• pwd: print working directory, gives the absolute path of your current location in the directory hierarchy
• cd dirname: change to folder called dirname
  • If you omit directory name, you will end up in your home directory
  • Enter cd
• mkdir dirname: create a directory called dirname
  • Create a directory, any name you want and cd to that directory
  • I would enter the command mkdir ise407 followed by cd ise407
  • Use the pwd command to check your current location

• The two most commonly used editors on Linux/Unix systems are:
  • vi or vim (vi improved)
  • emacs
• vi/vim is installed by default on Linux/Unix systems and has only a command line interface (CLI).
• emacs has both a CLI and a graphical user interface (GUI).
• Other editors that you may come across on *nix systems
  • kate: default editor for KDE.
  • gedit: default text editor for GNOME desktop environment.
  • gvim: GUI version of vim
  • pico: console based plain text editor
  • nano: GNU.org clone of pico
  • kwrite: editor by KDE.

• A scripting language or script language is a programming language that supports the writing of scripts.
• Scripting Languages provide a higher level of abstraction than standard programming languages.
• Compared to programming languages, scripting languages do not distinguish between data types: integers, real values, strings, etc.
• Scripting Languages tend to be good for automating the execution of other programs.
  • analyzing data
  • running daily backups
• They are also good for writing a program that is going to be used only once and then discarded.
• A script is a program written for a software environment that automate the execution of tasks which could alternatively be executed one-by-one by a human operator.
• The majority of script programs are “quick and dirty”, where the main goal is to get the program written quickly.

• Write a script
  • A shell script is a file that contains ASCII text.
  • Create a file, hello.sh with the following lines

#!/bin/bash
# My First Script
echo "Hello World!"

• Set permissions

chmod +x hello.sh

• Execute the script

./hello.sh

##  Hello World !

• My First Script

## #!/bin/bash
## # My First Script
## echo "Hello World!"

• The first line is called the ”ShaBang” line. It tells the OS which interpreter to use.
• Other options are:
  • sh : #!/bin/sh
  • ksh : #!/bin/ksh
  • csh : #!/bin/csh
  • tcsh: #!/bin/tcsh
• The second line is a comment. All comments begin with #.
• The third line tells the OS to print ”Hello World!” to the screen.
• Slides written in R Markdown. ## in orange implies screen output

• Double Quotation " "
  • Enclosed string is expanded
• Single Quotation ' '
  • Enclosed string is read literally
• Back Quotation ` `
  • Used for command substitution
  • Enclosed string is executed as a command
  • In bash, you can also use $( ) instead of ` `

myvar=hello
myname=Alex
echo "$myvar $myname"
echo '$myvar $myname'
echo $(pwd)

## hello Alex
## $myvar $myname
## /Users/apacheco/Tutorials/DCVS/bitbucket/lurc

• You can use C-style increment operators (let c++)
• Only integer arithmetic works in bash
• Use bc or awk for floating point arithmetic
  • echo "3.8 + 4.2" | bc
  • echo "scale=5; 2/5" | bc
  • bc <<< "scale=5; 2/5"
  • echo 3.8 4.2 | awk '{print $1*$2}'

• Shell Scripting Languages execute commands in sequence similar to programming languages such as C, Fortran, etc.
• Control constructs can change the sequential order of commands.
  • Conditionals: if
  • Loops: for, while, until

• integers and strings

• if statements can be nested and/or simplied using logical operators

if [ $a -gt 0 ]; then
  if [ $a -lt 5 ]; then
    echo "The value of a lies somewhere between 0 and 5"
  fi
fi

if [[ $a -gt 0 && $a -lt 5 ]]
  echo "The value of a lies somewhere between 0 and 5"
fi 

if [[ $a -gt 0 || $x -lt 5 ]]
  echo "a is positive or x is less than 5"
fi

• A loop is a block of code that iterates a list of commands.
• for is the basic loop construct

for arg in list
do
  some commands
done

• while (until) tests for a condition at the top of a loop, and keeps looping as long as that condition is true (false)

while [ condition ]; do
  some commands
done

until [ condition ]; do
  some commands
done

• Array elements may be initialized with the variable[xx] notation variable[xx]=1
• Initialize an array during declaration name=(firstname 'last name')
• reference an element i of an array name ${name[i]}
• print the whole array ${name[@]}
• print length of array ${#name[@]}
• print length of element i of array name ${#name[i]}
• ${#name} prints the length of the first element of the array
• Add an element to an existing array name=(title ${name[@]})
• copy an array name to an array user user=(${name[@]})
• concatenate two arrays nameuser=(${name[@]} ${user[@]})
• delete an entire array unset name
• remove an element i from an array unset name[i]
• the first array index is zero (0)

• Similar to programming languages, bash (and other shell scripting languages) can also take command line arguments
  • ./scriptname arg1 arg2 arg3 arg4 ...
  • $0,$1,$2,$3, etc: positional parameters corresponding to
  • ./scriptname,arg1,arg2,arg3,arg4,... respectively
  • for more than 9 arguments, position parameters need to be protected for e.g. ${10} OR
  • use shift N: shift positional parameters from N+1 to $# are renamed to variable names from $1 to $# - N + 1
  • $#: number of command line arguments
  • $*: all of the positional parameters, seen as a single word
  • $@: same as $* but each parameter is a quoted string.

#!/bin/bash
USAGE="USAGE: $0 <at least 1 argument>"
if [[ "$#" -lt 1 ]]; then
   echo $USAGE
   exit
fi
echo "Number of Arguments: " $#
echo "List of Arguments: " $@
echo "Name of script that you are running: " $0
echo "Command You Entered:" $0 $*
while [ "$#"  -gt 0 ]; do
  echo "Argument List is: " $@
  echo "Number of Arguments: " $#
  shift 
done

sh ./shift.sh $(seq 1 3)

## Number of Arguments:  3
## List of Arguments:  1 2 3
## Name of script that you are running:  ./shift.sh
## Command You Entered: ./shift.sh 1 2 3
## Argument List is:  1 2 3
## Number of Arguments:  3
## Argument List is:  2 3
## Number of Arguments:  2
## Argument List is:  3
## Number of Arguments:  1

• Like ”real” programming languages, bash has functions.
• A function is a subroutine, a code block that implements a set of operations, a ”black box” that performs a specified task.
• Wherever there is repetitive code, when a task repeats with only slight variations in procedure, then consider using a function.

function function_name {
  somecommands
}

OR

function_name () {
  some commands
}

• You can also pass arguments to a function.
• All function parameters or arguments can be accessed via $1, $2, $3,..., $N.
• $0 always point to the shell script name.
• $* or $@ holds all parameters or arguments passed to the function.
• $# holds the number of positional parameters passed to the function.
• Array variable called FUNCNAME contains the names of all shell functions currently in the execution call stack.
• By default all variables are global.
• Modifying a variable in a function changes it in the whole script.
• You can create a local variables using the local command

local var=value
local varName

• A function may recursively call itself.

#!/bin/bash
usage () {  echo "USAGE: $0 <integer>" ;   exit }

factorial() {
  local i=$1 ; local f
  declare -i i ; declare -i f

  if [[ "$i" -le 2 && "$i" -ne 0 ]]; then
    echo $i
  elif [[ "$i" -eq 0 ]]; then
    echo 1
  else
    f=$(( $i - 1 ))
    f=$( factorial $f )
    f=$(( $f * $i ))
    echo $f
  fi
}

if [[ "$#" -eq 0 ]]; then
  usage
else
  for i in $@ ; do
    x=$( factorial $i )
    echo "Factorial of $i is $x"
  done
fi

sh ./factorial3.sh $(seq 1 2 18)

## Factorial of 1 is 1
## Factorial of 3 is 6
## Factorial of 5 is 120
## Factorial of 7 is 5040
## Factorial of 9 is 362880
## Factorial of 11 is 39916800
## Factorial of 13 is 6227020800
## Factorial of 15 is 1307674368000
## Factorial of 17 is 355687428096000

• Maia

  • 32-core Symmetric Multiprocessor (SMP) system available to all Lehigh Faculty, Staff and Students
  • dual 16-core AMD Opteron 6380 2.5GHz CPU
  • 128GB RAM and 4TB HDD
  • Theoretical Performance: 640 GFLOPs (640 billion floating point operations per second)
  • Access: Batch Scheduled, no interactive access to Maia

  \[ GFLOPs = cores \times clock \times \frac{FLOPs}{cycle} \]

  FLOPs for various AMD & Intel CPU generation

  • Currently out of warranty.
  • Will be permanently shut down in case of hardware failure
  • No plans to replace Maia

• Sol: 80 node Shared Condominium Cluster
  • 9 nodes, dual 10-core Intel Xeon E5-2650 v3 2.3GHz CPU, 25MB Cache, 128GB RAM
  • 33 nodes, dual 12-core Intel Xeon E5-2670 v3 2.3Ghz CPU, 30 MB Cache, 128GB RAM
  • 14 nodes, dual 12-core Intel Xeon E5-2650 v4 2.3Ghz CPU, 30 MB Cache, 64GB RAM
  • 1 node, dual 8-core Intel Xeon 2630 v3 2.4GHz CPU, 20 MB Cache, 512GB RAM
  • 23 nodes, dual 18-core Intel Xeon Gold 6140 2.3GHz CPU, 24.7 MB Cache, 192GB RAM
  • 66 nVIDIA GTX 1080 GPU cards
  • 48 nVIDIA RTX 2080 GPU cards
  • 1TB HDD per node
  • 2:1 oversubscribed Infiniband EDR (100Gb/s) interconnect fabric
  • Theoretical Performance: 80.3968 TFLOPs (CPU) + 34.5876 TFLOPs (GPU)
  • Access: Batch Scheduled, interactive on login node for compiling, editing only

• Haswell (v3) and Broadwell (v4) provide 256-bit Advanced Vector Extensions SIMD instructions
  • double the base FLOPs at expense of CPU Speed
• Skylake (Gold) provide 512-bit Advanced Vector Extensions SIMD instructions
  • quadruple the base FLOPs at expense of CPU Speed

• LTS provides various storage options for research and teaching
• Some are cloud based and subject to Lehigh's Cloud Policy
• For research, LTS is deploying a 600TB refreshed Ceph storage system
• Ceph is based on the Ceph software
• Research groups can purchase a sharable project space on Ceph @ $375/TB for 5 year duration
• Ceph is in-house, built, operated and administered by LTS Research Computing Staff.
  • located in Data Center in EWFM with a backup cluster in Packard Lab
• HPC users can write job output directly to their Ceph volume
• Ceph volume can be mounted as a network drive on Windows or CIFS on Mac and Linux
  • See Ceph FAQ for more details
• Storage quota on
  • Maia: 5GB
  • Sol: 150GB

• Login to sol: ssh -X username@sol.cc.lehigh.edu
  • You should see something like [alp514@sol ~]$ if you are logged into sol
  • This is known as the command prompt
  • sol is the head/login and storage node for the monocacy cluster.
  • Do not run computation on this node.
  • Editing files, compiling code are acceptable computation on this node.
  • Running intense computation on this node causes a high load on the storage that would cause other users jobs to run slow.
  • All compute nodes are labelled as sol-[a-e][1-6][01-18]
    • [a-e] for each distinct type of node
    • [1-6] is the rack number in the HPC row
    • [01-18] is the location in the rack with 01 at the bottom and 18 at the top

• Commercial, Free and Open source software is installed on
  • Maia: /zhome/Apps
  • Sol: /share/Apps
• Software is managed using module environment
  • Why? We may have different versions of same software or software built with different compilers
  • Module environment allows you to dynamically change your *nix environment based on software being used
  • Standard on many University and national High Performance Computing resource since circa 2011
• How to use Sol/Maia Software on your linux workstation
• LTS provides licensed and open source software for Windows, Mac and Linux and Gogs, a self hosted Git Service or Github clone

• Users who prefer not to use the module environment will need to modify their .bashrc or .tcshrc files. Run module show for list variables that need modified, appended or prepended

• You can always install a software in your home directory
  • SPACK is an excellent package manager that can even create module files
• Stay compliant with software licensing
• Modify your .bashrc/.tcshrc to add software to your path, OR
• create a module and dynamically load it so that it doesn't interfere with other software installed on the system
  • e.g. You might want to use openmpi instead of mvapich2
  • the system admin may not want install it system wide for just one user
• Add the directory where you will install the module files to the variable MODULEPATH in .bashrc/.tcshrc

# My .bashrc file
export MODULEPATH=${MODULEPATH}:/home/alp514/modulefiles

• Various versions of compilers installed on Sol
• Open Source: GNU Compiler (also called gcc even though gcc is the c compiler)
  • 4.8.5 (system default), 5.3.0, 6.1.0 and 7.1.0
• Commercial: Only two seats of each
  • Intel Compiler: 16.0.3, 17.0.0, 17.0.3 and 18.0.1
  • Portland Group or PGI: 16.5, 16.10, 17.4, 17.7 and 18.3
• We are licensed to install any available version
• On Sol, all except gcc 4.8.5 are available via the module environment

[2018-02-22 08:47.27] ~/Workshop/2017XSEDEBootCamp/OpenMP
[alp514.sol-d118](842): icc -o laplacec laplace_serial.c
[2018-02-22 08:47.46] ~/Workshop/2017XSEDEBootCamp/OpenMP
[alp514.sol-d118](843): ./laplacec
Maximum iterations [100-4000]?
1000
---------- Iteration number: 100 ------------
[995,995]: 63.33  [996,996]: 72.67  [997,997]: 81.40  [998,998]: 88.97  [999,999]: 94.86  [1000,1000]: 98.67
---------- Iteration number: 200 ------------
[995,995]: 79.11  [996,996]: 84.86  [997,997]: 89.91  [998,998]: 94.10  [999,999]: 97.26  [1000,1000]: 99.28
---------- Iteration number: 300 ------------
[995,995]: 85.25  [996,996]: 89.39  [997,997]: 92.96  [998,998]: 95.88  [999,999]: 98.07  [1000,1000]: 99.49
---------- Iteration number: 400 ------------
[995,995]: 88.50  [996,996]: 91.75  [997,997]: 94.52  [998,998]: 96.78  [999,999]: 98.48  [1000,1000]: 99.59
---------- Iteration number: 500 ------------
[995,995]: 90.52  [996,996]: 93.19  [997,997]: 95.47  [998,998]: 97.33  [999,999]: 98.73  [1000,1000]: 99.66
---------- Iteration number: 600 ------------
[995,995]: 91.88  [996,996]: 94.17  [997,997]: 96.11  [998,998]: 97.69  [999,999]: 98.89  [1000,1000]: 99.70
---------- Iteration number: 700 ------------
[995,995]: 92.87  [996,996]: 94.87  [997,997]: 96.57  [998,998]: 97.95  [999,999]: 99.01  [1000,1000]: 99.73
---------- Iteration number: 800 ------------
[995,995]: 93.62  [996,996]: 95.40  [997,997]: 96.91  [998,998]: 98.15  [999,999]: 99.10  [1000,1000]: 99.75
---------- Iteration number: 900 ------------
[995,995]: 94.21  [996,996]: 95.81  [997,997]: 97.18  [998,998]: 98.30  [999,999]: 99.17  [1000,1000]: 99.77
---------- Iteration number: 1000 ------------
[995,995]: 94.68  [996,996]: 96.15  [997,997]: 97.40  [998,998]: 98.42  [999,999]: 99.22  [1000,1000]: 99.78

Max error at iteration 1000 was 0.034767
Total time was 4.099030 seconds.

• OpenMP support is built-in

[alp514.sol](1083): module show intel
-------------------------------------------------------------------
/share/Apps/share/Modules/modulefiles/toolchain/intel/16.0.3:

module-whatis    Set up Intel 16.0.3 compilers. 
conflict         pgi 
conflict         gcc 
setenv           INTEL_LICENSE_FILE /share/Apps/intel/licenses/server.lic 
setenv           IPPROOT /share/Apps/intel/compilers_and_libraries_2016.3.210/linux/ipp 
setenv           MKLROOT /share/Apps/intel/compilers_and_libraries_2016.3.210/linux/mkl 
setenv           TBBROOT /share/Apps/intel/compilers_and_libraries_2016.3.210/linux/tbb 
...
snip
...

[2018-02-22 08:47.56] ~/Workshop/2017XSEDEBootCamp/OpenMP/Solutions
[alp514.sol-d118](845): icc -qopenmp -o laplacec laplace_omp.c
[2018-02-22 08:48.09] ~/Workshop/2017XSEDEBootCamp/OpenMP/Solutions
[alp514.sol-d118](846): OMP_NUM_THREADS=4 ./laplacec
Maximum iterations [100-4000]?
1000
---------- Iteration number: 100 ------------
[995,995]: 63.33  [996,996]: 72.67  [997,997]: 81.40  [998,998]: 88.97  [999,999]: 94.86  [1000,1000]: 98.67
---------- Iteration number: 200 ------------
[995,995]: 79.11  [996,996]: 84.86  [997,997]: 89.91  [998,998]: 94.10  [999,999]: 97.26  [1000,1000]: 99.28
---------- Iteration number: 300 ------------
[995,995]: 85.25  [996,996]: 89.39  [997,997]: 92.96  [998,998]: 95.88  [999,999]: 98.07  [1000,1000]: 99.49
---------- Iteration number: 400 ------------
[995,995]: 88.50  [996,996]: 91.75  [997,997]: 94.52  [998,998]: 96.78  [999,999]: 98.48  [1000,1000]: 99.59
---------- Iteration number: 500 ------------
[995,995]: 90.52  [996,996]: 93.19  [997,997]: 95.47  [998,998]: 97.33  [999,999]: 98.73  [1000,1000]: 99.66
---------- Iteration number: 600 ------------
[995,995]: 91.88  [996,996]: 94.17  [997,997]: 96.11  [998,998]: 97.69  [999,999]: 98.89  [1000,1000]: 99.70
---------- Iteration number: 700 ------------
[995,995]: 92.87  [996,996]: 94.87  [997,997]: 96.57  [998,998]: 97.95  [999,999]: 99.01  [1000,1000]: 99.73
---------- Iteration number: 800 ------------
[995,995]: 93.62  [996,996]: 95.40  [997,997]: 96.91  [998,998]: 98.15  [999,999]: 99.10  [1000,1000]: 99.75
---------- Iteration number: 900 ------------
[995,995]: 94.21  [996,996]: 95.81  [997,997]: 97.18  [998,998]: 98.30  [999,999]: 99.17  [1000,1000]: 99.77
---------- Iteration number: 1000 ------------
[995,995]: 94.68  [996,996]: 96.15  [997,997]: 97.40  [998,998]: 98.42  [999,999]: 99.22  [1000,1000]: 99.78

Max error at iteration 1000 was 0.034767
Total time was 2.459961 seconds.

• MPI is a library and not a compiler, built or compiled for different compilers.

• Usage: <compiler> <options> <source code>

[2017-10-30 08:40.30] ~/Workshop/2017XSEDEBootCamp/MPI/Solutions
[alp514.sol](1096): mpif90 -o laplace_f90 laplace_mpi.f90 
[2017-10-30 08:40.45] ~/Workshop/2017XSEDEBootCamp/MPI/Solutions
[alp514.sol](1097): mpicc -o laplace_c laplace_mpi.c
[2017-10-30 08:40.57] ~/Workshop/2017XSEDEBootCamp/MPI/Solutions

[alp514.sol](1080): mpif90 -show
ifort -fPIC -I/share/Apps/mvapich2/2.1/intel-16.0.3/include 
  -I/share/Apps/mvapich2/2.1/intel-16.0.3/include 
  -L/share/Apps/mvapich2/2.1/intel-16.0.3/lib 
  -lmpifort -Wl,-rpath -Wl,/share/Apps/mvapich2/2.1/intel-16.0.3/lib 
  -Wl,--enable-new-dtags -lmpi

• Every MPI implementation come with their own job launcher: mpiexec (MPICH,OpenMPI & MVAPICH2), mpirun (OpenMPI) or mpirun_rsh (MVAPICH2)
• Example: mpiexec [options] <program name> [program options]
• Required options: number of processes and list of hosts on which to run program

• To run a MPI code, you need to use the launcher from the same implementation that was used to compile the code.
• For e.g.: You cannot compile code with OpenMPI and run using the MPICH and MVAPICH2's launcher
  • Since MVAPICH2 is based on MPICH, you can launch MVAPICH2 compiled code using MPICH's launcher.
• SLURM scheduler provides srun as a wrapper around all mpi launchers

[2018-02-22 08:48.27] ~/Workshop/2017XSEDEBootCamp/MPI/Solutions
[alp514.sol-d118](848): mpicc -o laplacec laplace_mpi.c
[2018-02-22 08:48.41] ~/Workshop/2017XSEDEBootCamp/MPI/Solutions
[alp514.sol-d118](849): mpiexec -n 4 ./laplacec
Maximum iterations [100-4000]?
1000
---------- Iteration number: 100 ------------
[995,995]: 63.33  [996,996]: 72.67  [997,997]: 81.40  [998,998]: 88.97  [999,999]: 94.86  [1000,1000]: 98.67
---------- Iteration number: 200 ------------
[995,995]: 79.11  [996,996]: 84.86  [997,997]: 89.91  [998,998]: 94.10  [999,999]: 97.26  [1000,1000]: 99.28
---------- Iteration number: 300 ------------
[995,995]: 85.25  [996,996]: 89.39  [997,997]: 92.96  [998,998]: 95.88  [999,999]: 98.07  [1000,1000]: 99.49
---------- Iteration number: 400 ------------
[995,995]: 88.50  [996,996]: 91.75  [997,997]: 94.52  [998,998]: 96.78  [999,999]: 98.48  [1000,1000]: 99.59
---------- Iteration number: 500 ------------
[995,995]: 90.52  [996,996]: 93.19  [997,997]: 95.47  [998,998]: 97.33  [999,999]: 98.73  [1000,1000]: 99.66
---------- Iteration number: 600 ------------
[995,995]: 91.88  [996,996]: 94.17  [997,997]: 96.11  [998,998]: 97.69  [999,999]: 98.89  [1000,1000]: 99.70
---------- Iteration number: 700 ------------
[995,995]: 92.87  [996,996]: 94.87  [997,997]: 96.57  [998,998]: 97.95  [999,999]: 99.01  [1000,1000]: 99.73
---------- Iteration number: 800 ------------
[995,995]: 93.62  [996,996]: 95.40  [997,997]: 96.91  [998,998]: 98.15  [999,999]: 99.10  [1000,1000]: 99.75
---------- Iteration number: 900 ------------
[995,995]: 94.21  [996,996]: 95.81  [997,997]: 97.18  [998,998]: 98.30  [999,999]: 99.17  [1000,1000]: 99.77
---------- Iteration number: 1000 ------------
[995,995]: 94.68  [996,996]: 96.15  [997,997]: 97.40  [998,998]: 98.42  [999,999]: 99.22  [1000,1000]: 99.78

Max error at iteration 1000 was 0.034767
Total time was 1.030180 seconds.

• All compute intensive jobs are scheduled
• Write a script to submit jobs to a scheduler
  • need to have some background in shell scripting (bash/tcsh)
• Need to specify
  • Resources required (which depends on configuration)
    • number of nodes
    • number of processes per node
    • memory per node
  • How long do you want the resources
    • have an estimate for how long your job will run
  • Which queue to submit jobs
    • SLURM uses the term partition instead of queue

• A software that manages resources (CPU time, memory, etc) and schedules job execution

  • Sol: Simple Linux Utility for Resource Management (SLURM)
  • Others: Portable Batch System (PBS)
    • Scheduler: Maui
    • Resource Manager: Torque
    • Allocation Manager: Gold

• A job can be considered as a user’s request to use a certain amount of resources for a certain amount of time

• The Scheduler or queuing system determines

  • The order jobs are executed
  • On which node(s) jobs are executed

• Ask for an amount of time that is
  • Long enough for your job to complete
  • As short as possible to increase the chance of backfilling

• The amount of installed memory less the amount that is used by the operating system and other utilities

• A general rule of thumb on most HPC resources: leave 1-2GB for the OS to run.

• Sol

• if you need to run a single core job that requires 10GB memory in the imlab partition, you need to request 2 cores even though you are only using 1 core.

• Maia: Users need to specify memory required in their submit script. Max memory that should be requested is 126GB.

• Submission
• Monitoring
• Manipulating
• Reporting

• Interactive Jobs
  • Set up an interactive environment on compute nodes for users
  • Will log you into a compute node and wait for your prompt
  • Purpose: testing and debugging code. Do not run jobs on head node!!!
    • All compute node have a naming convention sol-[a,b,c,d,e]###
    • head node is sol
• Batch Jobs
  • Executed using a batch script without user intervention
    • Advantage: system takes care of running the job
    • Disadvantage: cannot change sequence of commands after submission
  • Useful for Production runs
  • Workflow: write a script -> submit script -> take mini vacation -> analyze results

• sbatch allows for a filename pattern to contain one or more replacement symbols, which are a percent sign "%" followed by a letter (e.g. %j).

• Workflow: write a script -> submit script -> take mini vacation -> analyze results
• Batch scripts are written in bash, tcsh, csh or sh
  • ksh scripts will work if ksh is installed
• Add PBS or SLURM directives after the shebang line but before any shell commands
  • SLURM: #SBATCH DIRECTIVES
  • PBS: #PBS DIRECTIVES

• Submitting Batch Jobs

  • PBS: qsub filename
  • SLURM: sbatch filename

• qsub and sbatch can take the options for #PBS and #SBATCH as command line arguments

  • qsub -l walltime=1:00:00,nodes=1:ppn=16 -q normal filename
  • sbatch --time=1:00:00 --nodes=1 --ntasks-per-node=20 -p lts filename

• Option 1: Submit jobs as soon as previous jobs complete

• Option 2: Submit jobs with a dependency

  • SLURM: sbatch --dependency=afterok:<JobID> <Submit Script>
  • PBS: qsub -W depend=afterok:<JobID> <Submit Script>

• You want to run several serial processor jobs on

  • one node: your submit script should be able to run several serial jobs in background and then use the wait command for all jobs to finish
  • more than one node: this requires some background in scripting but the idea is the same as above

• The following scripts written by RC staff can also be used for monitoring jobs.
  • checkq: squeue with additional useful option.
  • checkload: sinfo with additional options to show load on compute nodes.
    
• load the soltools module to get access to RC staff created scripts

• Modify a job after submission but before starting:
  • scontrol update SPECIFICATION jobid=<jobid>
• Examples of SPECIFICATION are
  • add dependency after a job has been submitted: dependency=<attributes>
  • change job name: jobname=<name>
  • change partition: partition=<name>
  • modify requested runtime: timelimit=<hh:mm:ss>
  • change quality of service (when changing to imlab/engc): qos=nogpu
  • request gpus (when changing to one of the gpu partitions): gres=gpu:<1 or 2>
• SPECIFICATIONs can be combined for e.g. command to move a queued job to imlab partition and change timelimit to 48 hours for a job 123456 is
  • scontrol update partition=imlab qos=nogpu timelimit=48:00:00 jobid=123456

• sacct: displays accounting data for all jobs and job steps in the SLURM job accounting log or Slurm database

• sshare: Tool for listing the shares of associations to a cluster.

• We have created scripts based on these to provide usage reporting

  • alloc_summary.sh
    • included in your .bash_profile
    • prints allocation usage on your login shell
  • balance
    • prints allocation usage summary
  • solreport
    • obtain your monthly usage report
    • PIs can obtain usage report for all or specific users on their allocation
    • use --help for usage information

• Monthly usage summary (updated daily)
• Scheduler Status (updated every 15 mins)
• Annual Usage Reports:
  • AY 18-19 (updated daily)
  • AY 17-18
  • AY 16-17
• Usage reports restricted to Lehigh IPs