Skip to content

Install an OASIS Repository

OASIS (the OSG Application Software Installation Service) is an infrastructure, based on CVMFS, for distributing software throughout the OSG. Once software is installed into an OASIS repository, the goal is to make it available across about 90% of the OSG within an hour.

OASIS consists of keysigning infrastructure, a content distribution network (CDN), and a shared CVMFS repository that is hosted by the OSG. Many use cases will be covered by utilizing the shared repository; this document covers how to install, configure, and host your own CVMFS repository server. This server will distribute software via OASIS, but will be hosted and operated externally from the OSG project.

OASIS-based distribution and key signing is available to OSG VOs or repositories affiliated with an OSG VO. See the policy page for more information on what repositories OSG is willing to distribute.

Before Starting

CVMFS repositories work at the kernel filesystem layer, which adds more stringent host requirements than a typical OSG install. The host OS must meet ONE of the following:

  • RHEL 7.3 (or equivalent) or later. This option is recommended.


  • User IDs: If it does not exist already, the installation will create the cvmfs Linux user
  • Group IDs: If they do not exist already, the installation will create the Linux groups cvmfs and fuse
  • Network ports: This page will configure the repository to distribute using Apache HTTPD on port 8000. At the minimum, the repository needs in-bound access from the OASIS CDN.
  • Disk space: This host will need enough free disk space to host two copies of the software: one compressed and one uncompressed. /srv/cvmfs will hold all the published data (compressed and de-deuplicated). The /var/spool/cvmfs directory will contain all the data in all current transactions (uncompressed).
  • Root access will be needed to install. Software install will be done as an unprivileged user.
  • Yum will need to be configured to use the OSG repositories.

Overlay-FS limitations

CVMFS on RHEL7 only supports Overlay-FS if the underlying filesystem is ext3 or ext4; make sure /var/spool/cvmfs is one of these filesystem types.

If this is not possible, add CVMFS_DONT_CHECK_OVERLAYFS_VERSION=yes to your CVMFS configuration. Using xfs will work if it was created with ftype=1


Installation is a straightforward install via yum:

[email protected] # yum install cvmfs-server osg-oasis 

Apache and Repository Mounts

For all installs, we recommend mounting all the local repositories on startup:

[email protected] # echo "cvmfs_server mount -a" >>/etc/rc.local
[email protected] # chmod +x /etc/rc.local

The Apache HTTPD service should be configured to listen on port 8000, have the KeepAlive option enabled, and be started:

[email protected] # echo Listen 8000 >>/etc/httpd/conf.d/cvmfs.conf 
[email protected] # echo KeepAlive on >>/etc/httpd/conf.d/cvmfs.conf 
[email protected] # chkconfig httpd on 
[email protected] # service httpd start

Check Firewalls

Make sure that port 8000 is available to the Internet. Check the setting of the host- and site-level firewalls. The next steps will fail if the web server is not accessible.

Creating a Repository

Prior to creation, the repository administrator will need to make two decisions:

  • Select a repository name; typically, this is derived from the VO or project's name and ends in For example, the NoVA VO runs the repository For this section, we will use <EXAMPLE.OPENSCIENCEGRID.ORG>.
  • Select a repository owner: Software publication will need to run by a non-root Unix user account; for this document, we will use <LIBRARIAN> as the account name of the repository owner.

The initial repository creation must be run as root:

[email protected] # echo -e "\*\\t\\t-\\tnofile\\t\\t16384" >>/etc/security/limits.conf
[email protected] # ulimit -n 16384
[email protected] # cvmfs_server mkfs -o <LIBRARIAN> <EXAMPLE.OPENSCIENCEGRID.ORG>
[email protected] # cat >/srv/cvmfs/<EXAMPLE.OPENSCIENCEGRID.ORG>/.htaccess <<xEOFx
Order deny,allow
Deny from all
Allow from
Allow from ::1
Allow from
Allow from 2001:18e8:2:6::/56 2600:900:6::/48 

Here, we increase the number of open files allowed, create the repository using the mkfs command, and then limit the hosts that are allowed to access the repo to the OSG CDN.

Next, adjust the configuration in the repository as follows.

[email protected] # cat >>/etc/cvmfs/repositories.d/<EXAMPLE.OPENSCIENCEGRID.ORG>/server.conf <<xEOFx

Additionally, especially if files will be frequently deleted, enabling garbage collection is recommended in this way:

[email protected] # cat >>/etc/cvmfs/repositories.d/<EXAMPLE.OPENSCIENCEGRID.ORG>/server.conf <<xEOFx

The above assumes that you have your own mechanism to run cvmfs_server gc regularly (typically daily) at a time when it won't interfere with publications, since garbage collection and publication can't be done at the same time. CVMFS_AUTO_GC=true will automatically run garbage collection periodically after publications, but those times are not always convenient.

Also, check the cvmfs documentation for additional recommendations for special purpose repositories.

Now verify that the repository is readable over HTTP:

[email protected] # wget -qO- http://localhost:8000/cvmfs/<EXAMPLE.OPENSCIENCEGRID.ORG>/.cvmfswhitelist | cat -v

That should print several lines including some gibberish at the end.

Hosting a Repository on OASIS

In order to host a repository on OASIS, perform the following steps:

  1. Verify your VO's registration is up-to-date. All repositories need to be associated with a VO; the VO needs to assign an OASIS manager in Topology who would be responsible for the contents of any of the VO's repositories and will be contacted in case of issues. To designate an OASIS manager, have the VO manager update the Topology registration.

  2. Create a support ticket using the following template:

    Please add a new CVMFS repository to OASIS for VO <VO NAME> using the URL
        http://<FQDN>:8000/cvmfs/<OASIS REPOSITORY>
    The VO responsible manager will be <OASIS MANAGER>.

    Replace the <ANGLE BRACKET TEXT> items with the appropriate values.

  3. If the repository name matches * or *, wait for the go-ahead from the OSG representative before continuing with the remaining instructions; for all other repositories (such as *, you are done.

  4. When you are told in the ticket to proceed to the next step, first if the repository might be in a transaction abort it:

    [email protected] # su <LIBRARIAN> -c "cvmfs_server abort <EXAMPLE.OPENSCIENCEGRID.ORG>"

    Then execute the following commands:

    [email protected] # wget -O /srv/cvmfs/<EXAMPLE.OPENSCIENCEGRID.ORG>/.cvmfswhitelist \
    [email protected] # cp /etc/cvmfs/keys/ \

    Replace <EXAMPLE.OPENSCIENCEGRID.ORG> as appropriate. If the cp command prompts about overwriting an existing file, type 'y'.

  5. Verify that publishing operation succeeds:

    [email protected] # su <LIBRARIAN> -c "cvmfs_server transaction <EXAMPLE.OPENSCIENCEGRID.ORG>"
    [email protected] # su <LIBRARIAN> -c "cvmfs_server publish <EXAMPLE.OPENSCIENCEGRID.ORG>"

    Within an hour, the repository updates should appear at the OSG Operations and FNAL Stratum-1 servers.

    On success, make sure the whitelist update happens daily by creating /etc/cron.d/fetch-cvmfs-whitelist with the following contents:

    5 4 * * * <LIBRARIAN> cd /srv/cvmfs/<EXAMPLE.OPENSCIENCEGRID.ORG> && wget -qO<EXAMPLE.OPENSCIENCEGRID.ORG>/.cvmfswhitelist && mv .cvmfswhitelist


    This cronjob eliminates the need for the repository service administrator to periodically use cvmfs_server resign to update .cvmfswhitelist as described in the upstream CVMFS documentation.

  6. Update the open support ticket to indicate that the previous steps have been completed

Once the repository is fully replicated on the OSG, the VO may proceed in publishing into CVMFS using the <LIBRARIAN> account on the repository server.


We strongly recommend the repository maintainer read through the upstream documentation on maintaining repositories and content limitations.

If the repository ends in or, the VO may ask for it to be replicated outside the US. The VO should open a GGUS ticket following EGI's PROC20.

Replacing an Existing OASIS Repository Server

If a need arises to replace a server for an existing * or * repository, there are two ways to do it: one without changing the DNS name and one with changing it. The latter can take longer because it requires OSG Operations intervention.

Revision numbers must increase

CVMFS does not allow repository revision numbers to decrease, so the instructions below make sure the revision numbers only go up.

Without changing the server DNS name

If you are recreating the repository on the same machine, use the following command to remove the repository configuration while preserving the data and keys:

[email protected] # cvmfs_server rmfs -p <EXAMPLE.OPENSCIENCEGRID.ORG>

Otherwise if it is a new machine, copy the keys from /etc/cvmfs/keys/<EXAMPLE.OPENSCIENCEGRID.ORG>.* and the data from /srv/cvmfs/<EXAMPLE.OPENSCIENCEGRID.ORG> from the old server to the new, making sure that no publish operations happen on the old server while you copy the data.

Then in either case use cvmfs_server import instead of cvmfs_server mkfs in the above instructions for Creating the Repository, in order to reuse old data and keys.

If you run an old and a new machine in parallel for a while, make sure that when you put the new machine into production (by moving the DNS name) that the new machine has had at least as many publishes as the old machine, so the revision number does not decrease.

With changing the server DNS name


If you create a repository from scratch, as opposed to copying the data and keys from an old server, it is in fact better to change the DNS name of the server because that causes the OSG Operations server to reinitialize the .cvmfswhitelist.

If you create a replacement repository on a new machine from scratch, follow the normal instructions on this page above, but with the following differences in the Hosting a Repository on OASIS section:

  • In step 2, instead of asking in the support ticket to create a new repository, give the new URL and ask them to change the repository registration to that URL.
  • When you do the publish in step 5, add a -n NNNN option where NNNN is a revision number greater than the number on the existing repository. That number can be found by this command on a client machine:

    [email protected] $ attr -qg revision /cvmfs/<EXAMPLE.OPENSCIENCEGRID.ORG>
  • Skip step 6; there is no need to tell OSG Operations when you are finished.

  • After enough time has elapsed for the publish to propagate to clients, typically around 15 minutes, verify that the new chosen revision has reached a client.

Removing a Repository from OASIS

In order to remove a repository that is being hosted on OASIS, perform the following steps:

  1. If the repository has been replicated outside of the U.S., open a GGUS ticket asking that the replication be removed from EGI Stratum-1s. Wait until this ticket is resolved before proceeding.
  2. Open a support ticket asking to shut down the repository, giving the repository name (e.g., <EXAMPLE.OPENSCIENCEGRID.ORG>), and the corresponding VO.