Introduction to Linux & HPC

• Who?

  • Unit of Lehigh's Library & Technology Services within the Center for Innovation in Teaching & Learning

• Our Mission

  • We enable Lehigh Faculty, Researchers and Scholars achieve their goals by providing various computational resources; hardware, software, and storage; consulting and training.

• Research Computing Staff

  • Alex Pacheco, Manager & XSEDE Campus Champion
  • Steve Anthony, System Administrator
  • Dan Brashler, CAS Senior Computing Consultant
  • Sachin Joshi, Data Analyst & Visualization Specialist

• Hardware Support
  • Provide system administration and support for Lehigh's HPC clusters.
    • 1 University owned and 3 Faculty owned
  • Assist with purchase, installation and administration of servers and clusters.
• Data Storage
  • Provide data management services including storing and sharing data.
• Software Support
  • Provide technical support for software applications, install software as requested and assist with purchase of software.
• Training & Consulting
  • Provide education and training programs to facilitate use of HPC resources and general scientific computing needs.
    • https://go.lehigh.edu/hpcseminars
  • Provide consultation and support for code development and visualization.

• Sol
  • Lehigh's Flagship High Performance Computing Cluster
  • 9 nodes, dual 10-core Intel Xeon E5-2650 v3 2.3GHz CPU, 25MB Cache, 128GB RAM
  • 33 nodes, dual 12-core Intel Xeon E5-2670 v3 2.3Ghz CPU, 30 MB Cache, 128GB RAM
  • 14 nodes, dual 12-core Intel Xeon E5-2650 v4 2.3Ghz CPU, 30 MB Cache, 64GB RAM
  • 1 node, dual 8-core Intel Xeon 2630 v3 2.4GHz CPU, 20 MB Cache, 512GB RAM
  • 24 nodes, dual 18-core Intel Xeon Gold 6140 2.3GHz CPU, 24.7 MB Cache, 192GB RAM
  • 6 nodes, dual 18-core Intel Xeon Gold 6240 2.6GHz, 24.75 MB Cache, 192GB RAM
  • 72 nVIDIA GTX 1080 & 48 nVIDIA RTX 2080TI GPU cards
  • 1TB HDD per node
  • 2:1 oversubscribed Infiniband EDR (100Gb/s) interconnect fabric
  • Theoretical Performance: 93.184 TFLOPs (CPU) + 36.130 TFLOPs (GPU)

• Monocacy: Ben Felzer, Earth & Environmental Sciences
  • Eight nodes, dual 8-core Intel Xeon E5-2650v2, 2.6GHz, 64GB RAM
    • Theoretical Performance: 2.662TFlops
• Baltrusaitislab: Jonas Baltrusaitis, Chemical Engineering
  • Three nodes, dual 16-core AMD Opteron 6376, 2.3Ghz, 128GB RAM
    • Theoretical Performance: 1.766TFlops
• Pisces: Keith Moored, Mechanical Engineering and Mechanics
  • Six nodes, dual 10-core Intel Xeon E5-2650v3, 2.3GHz, 64GB RAM, nVIDIA Tesla K80
    • Theoretical Performance: 4.416 TFlops (CPU) + 17.46TFlops (GPU)
• Pavo : decommissioned faculty cluster for prototyping future resources
  • Twenty nodes, dual 8-core Intel Xeon E5-2650v2, 2.6GHz, 64GB RAM
    • Theoretical Performance: 6.656TFlops
    • 3 nodes available for 1 hour debug runs.

• Monocacy, Baltrusaitis and Pisces: decommissioning scheduled for Sep 30, 2021.

• Sol: accessible using ssh while on Lehigh's network

  ssh username@sol.cc.lehigh.edu
  

  • Windows PC require a SSH client such as MobaXterm or Putty.
  • Mac and Linux PC's, ssh is built in to the terminal application.

• If you are not on Lehigh's network, login to the ssh gateway

  ssh username@ssh.cc.lehigh.edu
  

  and then login to sol as above

  • Alternatively,

  ssh -J username@ssh.cc.lehigh.edu username@sol.cc.lehigh.edu
  

  • Click here to learn how to configure MobaXterm to use the SSH Gateway.

• Open, Interactive HPC via the Web
  • Easy to use, plugin-free, web-based access to supercomputers
  • File Management
  • Command-line shell access
  • Job management and monitoring
  • Various Applications
• NSF-funded project
  • SI2-SSE-1534949 and CSSI-Software-Frameworks-1835725
  • Ohio Supercomputing Center
  • Deployed at dozens of sites (universities, supercomputing centers)
• At Lehigh: https://hpcportal.cc.lehigh.edu
  • Lehigh IP or VPN required

• LTS provides various storage options for research and teaching.
• Some are cloud based and subject to Lehigh's Cloud Policy.
• For research, LTS provides a 768TB storage system called Ceph.
• Ceph is based on the Ceph software.
• Research groups can purchase a sharable project space on Ceph @ $375/TB with a 5 year duration.
• Ceph is in-house, built, operated and administered by LTS Research Computing Staff.
  • located in Data Center in EWFM with a backup cluster in Packard Lab
• HPC users can write job output directly to their Ceph volume
• Ceph volume can be mounted as a network drive on Windows or CIFS on Mac and Linux
  • See Ceph FAQ for more details
• Annual HPC User account fees waived for PIs who purchase a 1TB Ceph space for life of Ceph i.e. 5 years

• Login to sol using the SSH Client or the web portal

• Linux is the Operating System installed on all HPC resources.

  • You should see something like [alp514@sol ~]$ if you are logged into sol
  • This is known as the command prompt
  • sol is the head/login for the cluster.
  • Do not run computation on this node.
  • Editing files, compiling code are acceptable computation on this node.
  • Running intense computation on this node causes a high load on the storage that would cause other users jobs to run slow.

• Linux is an operating system that evolved from a kernel created by Linus Torvalds when he was a student at the University of Helsinki.
• It’s meant to be used as an alternative to other operating systems, Windows, Mac OS, MS-DOS, Solaris and others.
• Linux is the most popular OS used in a Supercomputer

• If you are using a Supercomputer/High Performance Computer for your research, it will be based on a *nix OS.
• It is required/neccessary/mandatory to learn Linux Programming (commands, shell scripting) if your research involves use of High Performance Computing or Supercomputing resources.

• Linux distributions are tailored to different requirements such as
  • Server
  • Desktop
  • Workstation
  • Routers
  • Embedded devices
  • Mobile devices (Android is a Linux-based OS)
• Almost any software that you use on windows has a roughly equivalent software on Linux, most often multiple equivalent software
  • e.g. Microsoft Office equivalents are OpenOffice.org, LibreOffice, KOffice
  • Visit for complete list
• Linux offers you freedom, to choose your desktop environment, software.

• All files are arranged in a hierarchial structure, like an inverted tree.
• The top of the hierarchy is traditionally called root (written as a slash / )

• Path means a position in the directory tree.
• You can use either the relative path or absolute path
• In relative path expression
  • (one dot or period) is the current working directory
  • (two dots or periods) is one directory up
  • You can combine . and .. to navigate the filee system hierarchy.
  • the path is not defined uniquely and does depend on the current path.
  • is unique only if your current working directory is your home directory.
• In absolute path expression
  • the path is defined uniquely and does not depend on the current path
  • /tmp is unique since /tmp is the abolute path

• HOME : Your Home Directory on the system, /home/username
  • This is where you should be when you login the first time
  • Don't believe me, type pwd and hit enter
• PATH : List of directories to search when executing a command
  • Enter avogadro at the command prompt
  • You should see an error saying command not found
• LD_LIBRARY_PATH : List of directories to look for libraries executing code

• The two most commonly used editors on Linux/Unix systems are:
  • vi or vim (vi improved)
  • emacs
• vi/vim is installed by default on Linux/Unix systems and has only a command line interface (CLI).
• emacs has both a CLI and a graphical user interface (GUI).
• Other editors that you may come across on *nix systems
  • kate: default editor for KDE.
  • gedit: default text editor for GNOME desktop environment.
  • gvim: GUI version of vim
  • pico: console based plain text editor
  • nano: GNU.org clone of pico
  • kwrite: editor by KDE.

• A variety of commercial, free and open source software is available
  • https://go.lehigh.edu/hpcsoftware
• Software is managed using module environment
  • Why? We may have different versions of same software or software built with different compilers
  • Module environment allows you to dynamically change your *nix environment based on software being used
  • Standard on many University and national High Performance Computing resource since circa 2011
• LTS provides licensed and open source software for Windows, Mac and Linux and Gogs, a self hosted Git Service or Github clone

• Users who prefer not to use the module environment will need to modify their .bashrc or .tcshrc files. Run module show for list variables that need modified, appended or prepended

• You can always install a software in your home directory
  • SPACK is an excellent package manager that can even create module files
• Stay compliant with software licensing
• Modify your .bashrc/.tcshrc to add software to your path, OR
• create a module and dynamically load it so that it doesn't interfere with other software installed on the system
  • e.g. You might want to use openmpi instead of mvapich2
  • the system admin may not want install it system wide for just one user
• Add the directory where you will install the module files to the variable MODULEPATH in .bashrc/.tcshrc

# My .bashrc file
export MODULEPATH=${MODULEPATH}:/home/alp514/modulefiles

• A cluster is a group of computers (nodes) that works together closely

• All compute intensive jobs are scheduled
• Write a script to submit jobs to a scheduler
  • need to have some background in shell scripting (bash/tcsh)
• Need to specify
  • Resources required (which depends on configuration)
    • number of nodes
    • number of processes per node
    • memory per node
  • How long do you want the resources
    • have an estimate for how long your job will run
  • Which queue to submit jobs
    • SLURM uses the term partition instead of queue

• A software that manages resources (CPU time, memory, etc) and schedules job execution

  • Sol: Simple Linux Utility for Resource Management (SLURM)
  • Others: Portable Batch System (PBS)
    • Scheduler: Maui
    • Resource Manager: Torque
    • Allocation Manager: Gold

• A job can be considered as a user’s request to use a certain amount of resources for a certain amount of time

• The Scheduler or queuing system determines

  • The order jobs are executed
  • On which node(s) jobs are executed

• Ask for an amount of time that is
  • Long enough for your job to complete
  • As short as possible to increase the chance of backfilling

• The amount of installed memory less the amount that is used by the operating system and other utilities

• A general rule of thumb on most HPC resources: leave 1-2GB for the OS to run.

• if you need to run a single core job that requires 10GB memory in the eng partition, you need to request 2 cores even though you are only using 1 core.
  

• sbatch allows for a filename pattern to contain one or more replacement symbols, which are a percent sign "%" followed by a letter (e.g. %j).

• Submission
• Monitoring
• Manipulating
• Reporting

• Interactive Jobs
  • Set up an interactive environment on compute nodes for users
  • Will log you into a compute node and wait for your prompt
  • Purpose: testing and debugging code. Do not run jobs on head node!!!
    • All compute node have a naming convention sol-[a,b,c]###
    • head node is sol
• Batch Jobs
  • Executed using a batch script without user intervention
    • Advantage: system takes care of running the job
    • Disadvantage: cannot change sequence of commands after submission
  • Useful for Production runs
  • Workflow: write a script -> submit script -> take mini vacation -> analyze results

• SLURM: Use srun command with SBATCH Directives followed by --pty /bin/bash

    srun --time=<hh:mm:ss> --nodes=<# of nodes> --ntasks-per-node=<# of core/node> -p <queue name> --pty /bin/bash
  

  • If you have soltools module loaded, then use interact with at least one SBATCH Directive
    • interact -t 20 [Assumes -p lts -n 1 -N 18]

• Run a job interactively replace --pty /bin/bash --login with the appropriate command.

     srun -t 20 -n 1 -p im1080 --qos=nogpu $(which lammps) -in in.lj -var x 1 -var n 1
  

  • Default values are 3 days, 1 node, 18 tasks per node and lts partition

• Workflow: write a script -> submit script -> take mini vacation -> analyze results

• Submitting Batch Jobs

  sbatch filename
  

• sbatch can take the options for #SBATCH as command line arguments

  sbatch --time=1:00:00 --nodes=1 --ntasks-per-node=18 -p lts filename
  

• In sequence or serially

  • Option 1: Submit jobs as soon as previous jobs complete
  • Option 2: Submit jobs with a dependency

  sbatch --dependency=afterok:<JobID> <Submit Script>
  

• In parallel, use GNU Parallel