Wednesday Exercise 1.4: Pre-packaging Code¶
In this exercise, you will create an installation of a Bayesian inference package (OpenBUGS) and then create \ a wrapper script to unpack that installation to run jobs. It should take 30-35 minutes.
Some software cannot be compiled into a single executable, whether you compile it yourself (as in Exercise 1.1) or download it already compiled (as in Exercise 1.2). In this case, it is necessary to download or create a portable copy of the software and then use a wrapper script (as in the previous exercise) to "install" the software on a per job basis. This script can either install the software from the source code, or (as in this exercise), unpack a portable software package that you've pre-built yourself.
Our Software Example¶
For this exercise, we will be using the Bayseian inference package Open BUGS. Open BUGS is a good example of software that is not compiled to a single executable; it has multiple executables as well as a helper library.
- Do an internet search to find the Open BUGS software downloads page.
- Create a directory for this exercise on the CHTC submit server
osg-learn.chtc.wisc.edu), download the source code (
OpenBUGS-3.2.3.tar.gz), and place it in this directory.
Note: The OpenBUGS source tarball cannot be downloaded via
wget=; you will need to download the source to your own computer and then transfer it to =learn.chtc.wisc.edu.
Where to Prepare¶
Our goal is to pre-build an Open BUGS installation, and then write a script that will unpack that installation and run a simulation.
- Where can we create this pre-built installation? Based on the end of the lecture, what are our options and which would be most appropriate? Make a guess before moving on.
- Because we're on the CHTC-based submit node (
learn.chtc.wisc.edu), we have the option of using an interactive job to build the Open BUGS installation. This is a good option because the submit server is already busy with lots of users and we don't know how long the Open BUGS install will take. To submit an interactive job do the following:
- Copy the following lines into a file named
- Copy the following lines into a file named
universe = vanilla output = build.out error = build.err log = build.log should_transfer_files = YES when_to_transfer_output = ON_EXIT transfer_input_files = request_disk = 2GB request_memory = 2GB queue
- Note the lack of executable. Condor doesn't need an executable for this job because it will be interactive, meaning you are running the commands instead of Condor.
- In order to create the installation, we will need the source code to come with us. The
transfer_input_filesline is blank - fill it in with the name of our Open BUGS source tarball.
- To request an interactive job, we will add a
-iflag to the
condor_submitcommand. The whole command you enter should look like this: \
%UCL_PROMPT_SHORT% <strong>condor_submit -i build.submit</strong>
Read Through Installation Documentation¶
While you're waiting for the interactive job to start, you can start reading the Open BUGS installation documentation online.
- Find the installation instructions for Open BUGS.
- On the downloads page, there are short instructions for how to install Open BUGS. There are two options shown for installation -- which should we use?
- The first installation option given uses
sudo-- which is an administrative permission that you won't have as a normal user. Luckily, as described in the \
instructions, you can use the
--prefix option to set where Open BUGS will be installed, which will allow us to install it without administrative permissions.
Your interactive job should have started by now and we've learned about installing our program. Let's test it out.
- Before we follow the installation instructions, we should create a directory to hold our installation. You can create this in the current directory. \
%UCL_PROMPT_SHORT% <strong>mkdir openbugs</strong>
- Now run the commands to unpack the source code: \
%UCL_PROMPT_SHORT% <strong>tar zxvf OpenBUGS-3.2.3.tar.gz</strong> %UCL_PROMPT_SHORT% <strong>cd OpenBUGS-3.2.3</strong>
- Now we can follow the second set of installation instructions. For the prefix, we'll use the command
$(pwd)to capture the name of our \
current working directory and then a relative path to the
openbugs directory we created in step 1: \
%UCL_PROMPT_SHORT% <strong>./configure --prefix=$(pwd)/../openbugs</strong> %UCL_PROMPT_SHORT% <strong>make</strong> %UCL_PROMPT_SHORT% <strong>make install</strong>
- Go back to the job's main working directory: \
%UCL_PROMPT_SHORT% <strong>cd ..</strong>
\ and confirm that our installation procedure created
share directories. \
%UCL_PROMPT_SHORT% <strong>ls openbugs</strong> bin lib share
- Now we want to package up our installation, so we can use it in other jobs. We can do this simply by compressing any necessary directories \
into a single gzipped tarball. \
%UCL_PROMPT_SHORT% <strong>tar -czf openbugs.tar.gz openbugs/</strong>
- Once everything is complete, type
exitto leave the interactive job. Make sure that your tarball is in the main working directory - it will be transferred back to the submit server automatically. \
Note that we now have two tarballs in our directory -- the source tarball (
OpenBUGS-3.2.3.tar.gz), which we will no longer need and our newly built installation (
openbugs.tar.gz) which is what we will actually be using to run jobs.
Now that we've created our portable installation, we need to write a script that opens and uses the installation, similar to the process we used in the \ previous exercise. These steps should be performed back on the submit server (
- Create a script called
run_openbugs.sh. The script will first need to untar our installation, so the script should start out like this: \
tar -xzf openbugs.tar.gz \
- We're going to use the same
$(pwd)trick from the installation in order to tell the computer how to find Open BUGS. We will \
do this by setting the
PATH environment variable, to include the directory where Open BUGS is installed: \
- Finally, the wrapper script needs to not only setup Open BUGS, but actually run the program. Add the following lines to your
run_openbugs.shwrapper script. \
OpenBUGS < input.txt > results.txt
- Make sure the wrapper script has executable permissions: \
%UCL_PROMPT_SHORT% <strong>chmod u+x run_openbugs.sh</strong>
Run a Open BUGS job¶
We're almost ready! We need two more pieces to run a OpenBUGS job.
- Download the necessary input files to your directory on the submit server and then untar them. \
%UCL_PROMPT_SHORT% <strong>wget http://proxy.chtc.wisc.edu/SQUID/osgschool17/openbugs_files.tar.gz</strong> %UCL_PROMPT_SHORT% <strong>tar -xzf openbugs_files.tar.gz</strong>
- Our last step is to create a submit file for our Open BUGS job. Think about which lines this submit file will need. Make a copy of a previous submit file (you could use the blast submit file from the previous exercise as a base) and modify it as you think necessary.
- The two most important lines to modify for this job are listed below; check them against your own submit file: \
executable = run_openbugs.sh transfer_input_files = openbugs.tar.gz, openbugs_files/
\ A wrapper script will always be a job's
executable. When using a wrapper script, you must also always remember to transfer the software/source code using
transfer_input_files. \ Note: the
/ in the
transfer_input_files line indicates that we are transferring the contents of that directory (which in this case, is what we want), rather than the directory itself.
- We also need to add a requirement to ensure that the job will run on a Linux system, running major version 6. This can be done with the line: \
requirements = (OpSys == "LINUX") && (OpSysMajorVer == 6)
- Submit the job with
- Once the job completes, it should produce a