Monday Exercise 4.3: Running jobs in the OSG¶
The goal of this exercise is to have your jobs running on the OSG and map their geographical locations.
Where in the world are my jobs? (Part 2)¶
In this version of the geolocating exercise, you will submit jobs to the OSG from
hopefully getting back much more interesting results!
You will be using the same exact payload as you did in exercise 4.1.
Gathering network information from the OSG¶
Now to create submit files that will run in the OSG!
- If not already logged in,
- Change to your local scratch directory
- Make a new directory for this exercise,
tuesday-4.3and change into it
rsyncfrom exercise 4.2 to copy over the executable and input file from the
- Re-create the submit file from exercise 4.1 except this time around change your submit file so that it submits five hundred jobs!
- Submit your file and wait for the results
Mapping your jobs¶
As before, you will be using http://www.mapcustomizer.com/ to visualize where your jobs have landed in the OSG. Copy and paste the collated results from your job output into the bulk creation text box at the bottom of the screen. Where did your jobs end up?
Once completed, move onto the next exercise: Hardware Differences in the OSG
Extra Challenge: Cleaning up your submit directory¶
If you run
ls in the directory from which you submitted your job, you may see that you now have thousands of files!
Proper data management starts to become a requirement as you start to develop truly HTC workflows;
you'll want organize your submit files, code, and input data separate from your output data.
Try editing your submit file so that all your output and error files are saved to separate directories within your submit directory.
Experiment with fewer job submissions until you're confident you have it right, then go back to submitting 500 jobs!
Submit your file and track the status of your jobs.
Did your jobs complete successfully with output and error files saved in separate directories? If not, can you find any useful information in the job logs or hold messages? If you get stuck, review the slides for submitting many jobs.