Open Science Pool Containers¶
In order to share compute resources with the Open Science pool, sites can launch pilot jobs directly by starting an OSG-provided Docker container. The container includes a simple worker node environment and an embedded pilot; when combined with an OSG-provided authentication token (not included in the container), the pilot can connect to the Open Science pool and start executing jobs.
This technique is useful to implement backfill at a site: contributing computing resources when they would otherwise be idle. It does not allow the site to share resources between multiple pools and, if there are no matching idle jobs in the Open Science pool, the pilots may remain idle.
In order to configure the container, you will need:
- A registered resource in OSG Topology; resource registration allows OSG to do proper usage accounting and maintain contacts in case of security incidents.
- An authentication token from the OSG. Please contact OSG support to request a token for your site.
- An HTTP caching proxy ("squid server") at or near your site.
Running the Container with Docker¶
The Docker image is kept in DockerHub. In order to successfully start payload jobs:
- Configure authentication:
Open Science Pool (OSP) administrators can provide the token,
which you can then pass to the container by volume mounting it as a file under
/etc/condor/tokens-orig.d/. If you are using Docker to launch the container, this is done with the command line flag
-v /path/to/token:/etc/condor/tokens-orig.d/flock.opensciencegrid.org. Replace
/path/to/tokenwith the full path to the token you obtained from the OSP administrators.
GLIDEIN_ResourceNameto match the site name and resource name that you registered in Topology, respectively.
- Set the
OSG_SQUID_LOCATIONenvironment variable to the HTTP address of your preferred Squid instance.
- Recommended: Enable CVMFS via one of the mechanisms described below.
Strongly recommended: If you want job I/O to be done in a separate directory outside of the container, volume mount the desired directory on the host to
/pilotinside the container.
Without this, user jobs may compete for disk space with other containers on your system.
If you are using Docker to launch the container, this is done with the command line flag
-v /worker-temp-dir:/pilot. Replace
/worker-temp-dirwith a directory you created for jobs to write into. Make sure the user you run your container as has write access to this directory.
Optional: add an expression with the
GLIDEIN_Start_Extraenvironment variable to append to the HTCondor
STARTexpression; this limits the pilot to only run certain jobs.
Here is an example invocation using
docker run by hand:
docker run -it --rm --user osg \ --privileged \ -v /path/to/token:/etc/condor/tokens-orig.d/flock.opensciencegrid.org \ -v /worker-temp-dir:/pilot \ -e GLIDEIN_Site="..." \ -e GLIDEIN_ResourceName="..." \ -e GLIDEIN_Start_Extra="True" \ -e OSG_SQUID_LOCATION="..." \ -e CVMFSEXEC_REPOS=" \ oasis.opensciencegrid.org \ singularity.opensciencegrid.org" \ opensciencegrid/osgvo-docker-pilot:release
/path/to/token with the location you saved the token obtained from the OSP administrators.
Privileged mode (
--privileged) requested in the above
docker run allows the container
to mount CVMFS using cvmfsexec and invoke
singularity for user jobs.
Singularity allows OSP users to use their own container for their job (e.g., a common use case for GPU jobs).
CernVM-FS (CVMFS) is a read-only remote filesystem that many OSG jobs depend on for software and data. Supporting CVMFS inside your container will greatly increase the types of OSG jobs you can run.
There are two methods for making CVMFS available in your container: enabling cvmfsexec, or bind-mounting CVMFS from the host. Bind-mounting CVMFS will require CVMFS to be installed on the host first, but the container will need fewer privileges.
Adding CVMFS using cvmfsexec¶
cvmfsexec is a tool that can be used to mount CVMFS inside the container without requiring CVMFS to be installed on the host.
To enable cvmfsexec, specify a space-separated list of repos in the
CVMFSEXEC_REPOS environment variable.
We recommend the following repos:
cvmfsexec has the following system requirements:
On EL7, you must have kernel version >= 3.10.0-1127 (run
uname -vrto check), and user namespaces enabled. See step 1 in the Singularity Install document for details.
On EL8, you must have kernel version >= 4.18 (run
uname -vrto check).
See the cvmfsexec README details.
Note that cvmfsexec will not be run if CVMFS repos are already available in
/cvmfs via bind-mount,
regardless of the value of
Using cvmfsexec takes place in the entrypoint, which means it will still happen
even if you specify a different command to run, such as
You can bypass the entrypoint by passing
--entrypoint <cmd> where
<cmd> is some different command to run,
Setting the entrypoint this way clears the command.
Additional cvmfsexec configuration¶
There are several environment variables you can set for cvmfsexec:
CVMFSEXEC_REPOS- this is a space-separated list of CVMFS repos to mount. Leave this blank to disable cvmfsexec. OSG jobs frequently use the OASIS repo (
oasis.opensciencegrid.org) and the singularity repo (
CVMFS_HTTP_PROXY- this sets the proxy to use for CVMFS; if left blank it will find the best one via WLCG Web Proxy Auto Discovery.
CVMFS_QUOTA_LIMIT- the quota limit in MB for CVMFS; leave this blank to use the system default (4 GB)
You can add other CVMFS options by bind-mounting a config file over
note that options in environment variables take precedence over options in
You can store the cache outside of the container by volume mounting a directory to
You can store the logs outside of the container by volume mounting a directory to
Adding CVMFS via bind-mount¶
As an alternative to using cvmfsexec, you may install CVMFS on the host, and volume mount it into the container.
This will let you avoid running the container in privileged mode.
However, supporting Singularity jobs inside the container will require extra privileges,
namely the capabilities
Follow the installing CVMFS document to install CVMFS on the host.
Once you have CVMFS installed and mounted on your host, add
-v /cvmfs:/cvmfs:shared to your
docker run invocation.
This is the example at the top of the page, modified to volume mount CVMFS instead of using cvmfsexec, and using reduced privileges:
docker run -it --rm --user osg \ --security-opt seccomp=unconfined \ --security-opt systempaths=unconfined \ --security-opt no-new-privileges \ -v /cvmfs:/cvmfs:shared \ -v /path/to/token:/etc/condor/tokens-orig.d/flock.opensciencegrid.org \ -v /worker-temp-dir:/pilot \ -e GLIDEIN_Site="..." \ -e GLIDEIN_ResourceName="..." \ -e GLIDEIN_Start_Extra="True" \ -e OSG_SQUID_LOCATION="..." \ opensciencegrid/osgvo-docker-pilot:release
Fill in the values for
OSG_SQUID_LOCATION as above.