OSG User School 2019 Materials¶

Monday¶

Monday Morning: Introduction to HTC and HTCondor¶

• Lecture: Introduction to HTC (PDF;PPT)
• Exercise 1.1: Log in to the local submit machine and look around
• Exercise 1.2: Experiment with basic HTCondor commands
• Exercise 1.3: Run jobs!
• Exercise 1.4: Read and interpret log files
• Exercise 1.5: Determining Resource Needs
• Exercise 1.6: Remove jobs from the queue
• Bonus Exercise 1.7: Compile and run some C code

Monday Morning: Running Many HTC Jobs¶

• Lecture: More HTCondor (PDF;PPT)
• Exercise 2.1: Work with input and output files
• Exercise 2.2: Use queue N, $(Cluster), and $(Process)
• Exercise 2.3: Use queue matching with a custom variable
• Exercise 2.4: Use queue from with custom variables

Monday Afternoon: Job Attributes and Handling¶

• Lecture: Intermediate HTCondor: Job Matching and Handling (PDF;PPT)
• Exercise 3.1: Explore condor_q
• Exercise 3.2: Explore condor_status
• Exercise 3.3: A job that needs retries

Monday Afternoon: Introduction to Distributed HTC¶

• Lecture: Introduction to DHTC (PDF; PPT)
• Exercise 4.1: Refresher - Submitting multiple jobs
• Exercise 4.2: Log in to the OSG submit machine
• Exercise 4.3: Running jobs in the OSG
• Exercise 4.4: Hardware differences in the OSG
• Exercise 4.5: Software differences in the OSG

Tuesday¶

Tuesday Morning: Troubleshooting jobs¶

• Lecture: Troubleshooting jobs (PDF; PPT)
• Exercise 1.1: Troubleshooting

Tuesday Morning: Security¶

• Lecture: Security in the OSG (PDF; PPT)
• Resource: Overview of OSG Services for Researchers and Institutions (PDF)

Tuesday Afternoon: Software Portability¶

• Lecture: Software Portability for DHTC (PDF, PPT)
• Exercise 3.1: Compiling programs for portability
• Exercise 3.2: Using a pre-compiled binary
• Exercise 3.3: Using a wrapper script
• Exercise 3.4: Pre-packaging code
• Bonus Exercise 3.5: Passing Arguments Through the Wrapper Script
• Lecture: Interpreted Languages for DHTC (PDF, PPT)
• Exercise 2.1: Pre-packaging Python
• Bonus Exercise 2.2: Python options

Wednesday¶

Wednesday Morning: Software Modules, Licensing¶

• Lecture: Considerations for licensing and programming packages (PDF, PPT)
• Exercise 1.1: Try an OSG Connect software module
• Exercise 1.2: Compile and run Matlab code

Wednesday Morning: Containers¶

• Lecture: Containers for DHTC (PDF, PPT)
• Exercise 2.1: Use Singularity from OSG Connect
• Bonus Exercise 2.2: Use Singularity to Run Tensorflow (Optional)
• Exercise 2.3: Using Docker

Wednesday Afternoon: On Your Own¶

• Ideas for activities

Thursday¶

Thursday Morning: Data Handling¶

• Lecture: Overall data considerations (PDF;PPT)
• Exercise 1.1: Understanding your data requirements
• Exercise 1.2: HTCondor file transfer and compression
• Exercise 1.3: Splitting large input data

Thursday Morning: Data Handling (continued)¶

• Lecture: Solutions for large input data (PDF; PPT)
• Exercise 2.1: Using a web proxy for large, shared input
• Exercise 2.2: Using StashCache for large, shared input
• Exercise 2.3: Using StashCache for large, unique input

Thursday Afternoon: Data Handling (continued)¶

• Lecture: Large output and shared file systems; Data summary (PDF;PPT)
• Exercise 3.1: Using a local shared filesystem for large input files
• Exercise 3.2: Using a local shared filesystem for large output files

Thursday Afternoon: Automating Workflows with HTCondor's DAGMan¶

• Lecture: HTCondor DAGMan for Workflows (PDF;PPT)
• Exercise 4.1: Coordinating set of jobs: A simple DAG
• Exercise 4.2: A brief detour through the Mandelbrot set
• Exercise 4.3: A more complex DAG
• Exercise 4.4: Handling jobs that fail with DAGMan
• Bonus Exercise 4.5: HTCondor challenges (If and only if you have time)

Friday¶

Friday Morning: From Science to Production Workflows¶

• Lecture: Optimizing Your Computing (PDF, PPT)
• Exercise 1.1: Learn about Joe’s Desired Computing Work
• Exercise 1.2: Plan Overall Workflow

Friday Morning: From Science to Production Workflows¶

• Lecture: Putting It All Together, Optimizing Workflows (PDF, PPT)
• Exercise 1.3: Execute Joe’s Workflow
• Bonus Exercise 1.4: Further Optimization and Scaling

Friday Afternoon: HTC Showcase¶

• Talk: Carolyn Voter, Civil and Environmental Engineering: High Throughput Computing in Your Backyard: Urban Hydrology Applications (PDF, PPTX)
• Talk: Sarah Sant'ana, Pscyhology: Using HTC to Understand and Communicate Appropriate Uses of Cross-Validation in Psychological Science (PDF)
• Talk: Joel Siegel, Physics: A Self-Stabilizing Metasurface Laser Sail to Explore the Stars (PDF, PPTX)
• Talk: Edgar Spalding, Botany: Measuring Plant Phenotypes with High Throughput Computing (PDF, PPTX)

Friday Afternoon: Foundations of HTC¶

• Lecture: The Principles of HTC (PDF)

Friday Afternoon: Wrap Up¶

• Lecture: Where to Go and What to Do Next (PDF)