HPC Crash Course

Session Outline

• Introduction to HPC

Attempt at a Definition: HPC …

Trade-Offs of HPC

There is no free lunch!

“Es gibt keinen Königsweg” (1)

“Es gibt keinen Königsweg” (2)

• Master using Linux.
• Master programming for
  • a single core
  • multiple cores on a single machine
  • multiple cores on multiple machines
  • programming GPUs
• Realize that most problems
  • are embarassingly parallel

“Es gibt keinen Königsweg” (3)

• Master using Linux.
• Master programming for
  • a single core
  • multiple cores on a single machine
  • multiple cores on multiple machines
  • programming GPUs
• Realize that most problems
  • are embarassingly parallel
  • have well-solved “cores”

Types of Parallelism

Single-core level (not in focus here)

• Bit-level parallelism
  • aka bit-wise AND, OR, etc.
• Instruction-level parallelism
  • aka instruction pipelining

Programming level

• Pipeline parallelism
• Task parallelism
• Data parallelism

Pipeline Parallelism

Task Parallelism (1)

Task Parallelism (2)

Compute Nodes (1)

“Same-same (as your laptop), but different.”

Compute Nodes (2)

More differences from “consumer-grade” hardware:

Cluster Architecture

Job Scheduler and Resource Management

sequenceDiagram
    autonumber
    User-)+Scheduler: sbatch $resource_args jobscript.sh
    Scheduler->>+Scheduler: add job to queue
    User-)+Scheduler: squeue / scontrol show job
    Scheduler-->>+User: job status
    Note right of Scheduler: scheduler loop
    Scheduler-)Compute Node: start job
    Compute Node->>Compute Node: execute job
    Compute Node-)+Scheduler: job complete

File System Quotas

• Your home volume has tight quotas

• Some programs will write a lot there

• Solution: use work volume

  NAME=.cpan
  mkdir -p work/$NAME
  mv $NAME/* work/$NAME
  rmdir $NAME
  ln -sr work/$NAME $NAME

• E.g., run the above for NAME in ondemand miniconda3 R Downloads .apptainer .theano .singularity .npm .nextflow .local .debug .cpan .cache .aspera

Killed Out Of Memory Jobs

• If you get random Killed messages, your job probably needs more memory than available
• In particular:
  • On login nodes, you only have 100MB of RAM!
• Solution: request more memory!
  • More on this in the Slurm session

Introduction

• Python has a threading library with low-level primitives
• But why is there a multiprocessing? Processes?
  • Python has a global interpreter lock (GIL)
  • (maybe removed at some point but it is there)
  • Access to Python data structures is serialized
  • Multithreading: only while threads are blocked (I/O)
• Multiprocessing to the rescue:
  • Copy data to another process
  • This process can do the work

Reminder: apply/map

• Remember functional programming?
• map(func, list) -> list
  • Apply func to each element on the list to obtain a new list of same size
  • Can be parallelized if there are no side effects
• apply(func, list)
  • Apply func to each element on the list, ignoring results
  • Can be parallelized if there are no side effects

Thread Pools

• Thread pools are abstractions for parallelism
• We create a pool with N threads
• We let the thread pool process collections / lists of work
  • multiprocessing.Pool() (process pool ;-))

Example

import multiprocessing

def func(element):
    # ...

if __name__ == "__main__":
  work = list(range(1_000_000))

  with multiprocessing.Pool(processes=4) as p:
    # force single element chunks
    done_work = p.map(func, work, chunksize=1)

  print(done_work)

Caveats

def func(element, a, b):
  # ...

def func2(element_params):
  element, params = element_params
  func(element, **params)

# ...

params = {
  "a": 1,
  "b": 2,
}

real_work = list(range(1_000_000))
work_with_params = [(x, params) for x in real_work]
done_work = p.map(func, work, chunksize=1)

Introduction

• GNU parallel is a command line tool that allows you to
  • run a potentially large list of tasks
  • with a fixed number of parallel tasks at the same time
• It is similar to the earlier Python multiprocessing.ThreadPool
• Excellent Tutorial

Example

THREADS=4

parallel -t -j $THREADS \
  'md5sum {} >{}.md5' ::: *.bam

parallel -t -j $THREADS \
  'cd {//}; md5sum {/} >{/}.md5' ::: *.bam