Wednesday Exercise 1.7: Using Python, Pre-Built¶
In this exercise, you will install Python, package your installation, and then use it to run jobs. It should take about 20 minutes.
We chose Python as the language for this example because: a) it is a common language used for scientific computing and b) it has a straightforward installation process and is fairly portable.
Running any Python script requires an installation of the Python interpreter. The Python interpreter is what we're using when we type
python at the command line. In order to run Python jobs on a distributed system, you will need to install the Python interpreter (what we often refer to as just "installing Python"), within the job, then run your Python script.
There are two installation approaches. The approach we will cover in this exercise is that of "pre-building" the installation (much like we did with Open BUGS thir morning). We will install Python to a specific directory, and then create a tarball of that installation directory. We can then use our tarball within jobs to run Python scripts.
Interactive Job for Pre-Building¶
The first step in our job process is building a Python installation that we can package up.
- Create a directory for this exercise on
- Download the Python source code from https://www.python.org/. \
%UCL_PROMPT_SHORT% <strong>wget https://www.python.org/ftp/python/3.6.1/Python-3.6.1.tgz</strong>
- Of our options - submit server, interactive job, personal computer - which should we use for this installation/packaging process? Once you have a guess, move to the next step.
- Due to the number of people on our submit server, we shouldn't use the submit server. Your own computer probably doesn't have the right operating system. \
The best place to install will be an interactive job. For this job, we can use the same interactive submit file as Exercise 1.4, with one change. What is it?
- Make a copy of the interactive submit file from Exercise 1.4 and change the
transfer_input_filesline to the Python tarball you just downloaded. Then submit it using the
%UCL_PROMPT_SHORT% <strong>condor_submit -i build.submit</strong>
- Once the interactive job begins, we can start our installation process. First, we have to determine how to install Python to a specific location in our working directory.
- Untar the Python source tarball and look at the
Python-3.6.1directory. What will the main installation steps be? Where will Python be installed by default? Once you've tried to answer these questions, move to the next step.
- There are some basic installation \
- Untar the Python source tarball and look at the
instructions near the top of the
README. Based on that short introduction, we can see the main steps of installation will be: \
./configure make make install
\ \ This looks a lot like the Open BUGS installation from earlier today! It turns out that this three-stage process (configure, make, make install) is a common \ way to install many software packages. \ \ Also like the Open BUGS installation, the default installation \ location for Python requires
sudo (administrative privileges) to install. However, we'd like to install to a specific location in the working directory \ so that we can compress that installation directory into a tarball. How did we do this with Open BUGS? \
- With Open BUGS we used the
-prefixoption with the
configurescript. Let's see if the Python \
configure script has this option by using the "help" option. \ \
%UCL_PROMPT_SHORT% <strong>./configure --help</strong>
\ \ Sure enough, there's a list of all the different options that can be passed to the
configure script, which includes
-prefix. Therefore, we can use the \
$(pwd) command in order to set the path correctly, just as we did earlier today.
- Now let's actually install Python!
- From the job's main working directory, create a directory to hold the installation. \
%UCL_PROMPT_SHORT% <strong>cd $_CONDOR_SCRATCH_DIR</strong> %UCL_PROMPT_SHORT% <strong>mkdir python</strong>
- Move into the
Python 3.6.1directory and run the installation commands. These may take a few minutes each. \
%UCL_PROMPT_SHORT% <strong>cd Python-3.6.1</strong> %UCL_PROMPT_SHORT% <strong>./configure --prefix=$(pwd)/../python</strong> %UCL_PROMPT_SHORT% <strong>make</strong> %UCL_PROMPT_SHORT% <strong>make install</strong>
- If I move back to the main job working directory, and look in the
pythonsubdirectory, I should see a Python installation. \
%UCL_PROMPT_SHORT% <strong>cd ..</strong> %UCL_PROMPT_SHORT% <strong>ls python/</strong> bin include lib share
- I have successfully created a self-contained Python installation. Now it just needs to be tarred up! \
%UCL_PROMPT_SHORT% <strong>tar -czf prebuilt_python.tar.gz python/</strong>
- Before exiting, we might want to know how we installed Python for later reference. \
Enter the following commands to save our history to a file: \
%UCL_PROMPT_SHORT% <strong>history > python_install.txt</strong>
- Exit the interactive job. \
- Create a script with the following lines called
import sys import os if len(sys.argv) != 2: print('Usage: %s MAXIMUM' % (os.path.basename(sys.argv))) sys.exit(1) maximum = int(sys.argv) n1 = n2 = 1 while n2 <= maximum: n1, n2 = n2, n1 + n2 print('The greatest Fibonacci number up to %d is %d' % (maximum, n1))
- What command line arguments does this script take? Try running it on the submit server.
We now have our Python installation and our Python script - we just need to write a wrapper script to run them.
- What steps do you think the wrapper script needs to perform? Create a file called
run_fib.shand write them out in plain English before moving to the next step.
- Our script will need to
- untar our
- access the
pythoncommand from our installation to run our
- untar our
- Try turning your plain English steps into commands that the computer can run.
- Your final
run_fib.shscript should look something like this: \
tar xzf prebuilt_python.tar.gz python/bin/python3 fib.py 5 \ or\
tar xzf prebuilt_python.tar.gz export PATH=$(pwd)/python/bin:$PATH python3 fib.py 5
- Make sure your
run_fib.shscript is executable.
- Make a copy of a previous submit file in your local directory (the Open BUGS submit file could be a good starting point). What changes need to be made to run this Python job? 2. Modify your submit file, then make sure you've included the key lines below: \
executable = run_fib.sh transfer_input_files = fib.py, prebuilt_python.tar.gz
- Because we pre-built our Python installation on a machine running Scientific Linux, version 6.something, we should request machines with similar characteristics. Add the following line to your submit file as well: \
requirements = (OpSys == "LINUX" && OpSysMajorVer == 6 )
- Submit the job using
condor_submit. 4. Check the
.outfile to see if the job completed.