OSG Virtual School (pilot) Materials¶

School Overview and Intro to HTC¶

View the slides (PDF, PowerPoint)

Intro to HTCondor Job Execution¶

View the slides (PDF, PowerPoint) and a recording of the lecture (YouTube)

Intro Exercises 1: Running and Viewing Simple Jobs (Strongly Recommended)¶

• Exercise 1.1: Log in to the local submit machine and look around
• Exercise 1.2: Experiment with HTCondor commands
• Exercise 1.3: Run jobs!
• Exercise 1.4: Read and interpret log files
• Exercise 1.5: Determining Resource Needs
• Exercise 1.6: Remove jobs from the queue
• Bonus Exercise 1.7: Compile and run some C code

Intro Exercises 2: Running Many HTC Jobs (Strongly Recommended)¶

• Exercise 2.1: Work with input and output files
• Exercise 2.2: Use queue N, $(Cluster), and $(Process)
• Exercise 2.3: Use queue from with custom variables
• Bonus Exercise 2.4: Use queue matching with a custom variable

Bonus Exercises: Job Attributes and Handling¶

• Bonus Exercise 3.1: Explore condor_q
• Bonus Exercise 3.2: Explore condor_status
• Bonus Exercise 3.3: A job that needs retries

Grid¶

View the slides (PDF, PowerPoint) and a recording of the lecture (YouTube)

All exercises strongly recommended!

• Exercise 1: Refresher – Submitting Multiple Jobs
• Exercise 2: Log in to the OSG Submit Server
• Exercise 3: Running jobs in the OSG
• Exercise 4: Hardware Differences in the OSG
• Exercise 5: Software Differences in the OSG

Software¶

View the slides (PDF, PowerPoint) and a recording of the lecture (YouTube)

Software Exercises 1: Basic Software and Wrapper Script Use (Strongly Recommended)¶

• Exercise 1.1: Work With Downloaded Software
• Exercise 1.2: Use a Wrapper Script To Run Software

Software Exercises 2: Specific Software Examples (Pick One)¶

• Exercise 2.1: Compiling and Running a Simple Code
• Exercise 2.2: Compiling a Research Software
• Exercise 2.3: Compiling Python and Running Jobs
• Exercise 2.4: Compiling Matlab and Running Jobs

Software Exercises 3: Using Containers in Jobs (Strongly Recommended)¶

• Exercise 3.1: Using Software in a Singularity Container

Software Exercises 4: Bonus Exercises¶

• Exercise 4.1: Using Arguments With Wrapper Scripts
• Exercise 4.2: Additional Python
• Exercise 4.3: Using Software in a Docker Container
• Exercise 4.4: Building Your Own Docker Container (Beta)
• Exercise 4.5: Using Conda Environments (Beta)

Data¶

View the slides (PDF, PowerPoint) and a recording of the lecture (YouTube)

Data Exercises 1: HTCondor File Transfer (Strongly Recommended)¶

• Exercise 1.1: Understanding a job's data needs
• Exercise 1.2: Using data compression with HTCondor file transfer
• Exercise 1.3: Splitting input

Data Exercises 2: Using Stash (Strongly Recommended)¶

• Exercise 2.1: Using a web proxy for shared input
• Exercise 2.2: Stash for shared input
• Exercise 2.3: Stash for shared output

Bonus Exercises: Shared File Systems¶

• Exercise 3.1: Shared filesystems for large input
• Exercise 3.2: Shared filesystems for large output

Bonus Topics¶

Self-Checkpointing (Fri., July 17)¶

View the slides (PDF, Keynote, PowerPoint export)

No separate exercise(s). Rather, if this applies to you: Try to get it working for your own code and let us know how it goes!

Workflows with DAGMan¶

View the slides (PPT, PDF)

• Exercise 1.1: Coordinating set of jobs: A simple DAG
• Exercise 1.2: A brief detour through the Mandelbrot set
• Exercise 1.3: A more complex DAG
• Exercise 1.4: Handling jobs that fail with DAGMan
• Bonus Exercise 4.5: HTCondor challenges

GPUs and Containers¶

View the slides (PDF, PowerPoint)

Final Talk (Fri., July 24) — What’s Next?¶

View the slides (PDF, Keynote, PowerPoint export)