Installing the StashCache Cache¶
This document describes how to install a StashCache cache service. This service allows a site or regional network to cache data frequently used on the OSG, reducing data transfer over the wide-area network and decreasing access latency.
The cache service must be registered with the OSG if it is to be used by clients. You may start registration procedure prior to finishing the installation by using this link along with the basic information like:
- Resource name and hostname.
- Administrative and security contact.
Installation prerequisites for the Cache¶
Before starting the installation process, consider the following mandatory points:
- Operating system: Only RHEL 7 and compatible operating systems are supported.
- User IDs: If they do not exist already, the installation will create the Linux user IDs
- Host certificate: The StashCache server uses a host certificate to advertise to a central collector. The host certificate documentation provides more information on setting up host certificates.
- Network ports: The cache service requires inbound ports, one for the xrootd protocol, one for
HTTP and, if authenticated StashCache is used, one for HTTPS. The defaults are:
- TCP port 1094 for xrootd.
- TCP port 8000 for HTTP.
- TCP port 8443 for HTTPS (optional).
- Hardware requirements: We recommend that a StashCache server has at least 10Gbps connectivity, 1TB of disk space for the cache directory, and 8GB of RAM.
If installing the (optional) authenticated StashCache, you also need to do the following:
- Service certificate: copy the host certificate to
- Set the owner of the directory and contents
[email protected] # chown -R xrootd:xrootd /etc/grid-security/xrd/
- Set the owner of the directory and contents
As with all OSG software installations, there are some one-time steps to prepare in advance:
- Ensure the host has a supported operating system. At this time, we only support RHEL7-based cache servers.
- Obtain root access to the host
- Prepare the required Yum repositories
- Install CA certificates
Installing the Cache¶
The StashCache daemon consists of an XRootD server and a service for collecting and reporting statistics about the cache. To simplify installation, OSG provides convenience RPMs that install all required software with a single command:
[email protected] # yum install --enablerepo=osg-testing stashcache-cache-server
If installing authenticated StashCache Cache server, you need an additional package:
[email protected] # yum install --enablerepo=osg-testing stashcache-cache-server-auth
Configuring the Cache¶
First, you must create a "cache directory", which will be used to store downloaded files.
By default this is
We recommend using a separate file system for the cache directory,
with at least 1 TB of storage available.
The cache directory must be writable by the
xrootd:xrootd user and group.
stashcache-cache-server provides a default configuration file,
/etc/xrootd/xrootd-stashcache-cache-server.cfg, which must be customized for your cache.
The most common lines to customize are:
all.sitename YOUR_SITE_NAME: The registered OSG resource name. This must be changed.
set cachedir = /stash: The location of the cache directory for this service.
pss.setopt: These control the logging verbosity of the cache. The defaults are relatively high in order to aid in debugging. These lines can be commented out to reduce logging; however, if issues occur, OSG support may ask you to re-enable them.
The Authfile specifies which files and directories can be read,
relative to the cache directory (
cachedir in the main config).
[email protected] # cat /etc/xrootd/Authfile-noauth u * /user/ligo -rl / rl
This permits all users (
u *) to read all directories (
/ rl) except those under
/user/ligo directory should only be readable in authenticated setups.
For more details, see the XRootD security documentation.
Configuring the Authenticated Cache (Optional)¶
The authenticated cache service is a separate instance of the StashCache cache service, and runs alongside the unauthenticated instance. You should make sure that the unauthenticated instance is functioning before setting up the authenticated instance. Before proceeding, make sure you have followed the prerequisite steps.
Add Authfile for authenticated cache¶
The Authfile for the authenticated cache is located in
This is a separate file from the non-authenticated cache Authfile.
Since the authenticated cache runs alongside the unauthenticated cache, care must be taken to avoid conflicts between the two. In particular, paths that are accessible via the authenticated cache should not be accessible via the unauthenticated cache, and vice versa.
As an example:
[email protected] # cat /etc/xrootd/Authfile-auth g /osg/ligo /user/ligo rl u ligo /user/ligo rl
This permits users in the VOMS group
/osg/ligo and users mapped to
ligo to read and list anything under
When ready with configuration, you may start your Cache server.
Configuring Optional Features¶
Adjust disk utilization¶
To adjust the disk utilization of your StashCache cache, modify the values of
pfc.diskusage 0.98 0.99
The first value and second values correspond to the low and high usage watermarks, respectively, in fractions. When the high watermark is reached, the XRootD service will automatically purge cache objects down to the low watermark.
Enable remote debugging¶
This feature enables remote debugging via the
digFS read-only file system, it's optional line in the config file that was created when configuring the cache:
xrootd.diglib * /etc/xrootd/digauth.cf
/etc/xrootd/digauth.cf may have following content:
all allow host h=abc.org all allow host h=*.xyz.edu
Managing StashCache and associated services¶
StashCache daemons are managed by systemd units. First ensure that your cache directory (default
mounted, then ensure you enable (set to start at boot) and start the StashCache-related services.
As a reminder, here are common service commands (all run as
root) for EL7:
|To...||On EL7, run the command...|
|Start a service||
|Stop a service||
|Enable a service to start on boot||
|Disable a service from starting on boot||
Public Cache Services¶
||The xrootd daemon, which performs the data transfers|
||Report cache statistics to central OSG collector|
||Required to authenticate monitoring services. See CA documentation for more info|
Authenticated Cache Services¶
In addition to the public cache services, there are three systemd units specific to the authenticated cache.
||The xrootd daemon which performs the authenticated data transfers|
||Renew a proxy for authenticated downloads to the cache|
||Trigger daily proxy renewal|
The cache server functions as a normal HTTP server and can interact with typical HTTP clients, such as
[email protected] $ curl -O http://cache_host:8000/user/dweitzel/public/blast/queries/query1
Test Cache server reports to HTCondor collector¶
To verify the cache is reporting to the central collector, run the following command:
hostname is the string returned by the hostname command. The output of the above command should provide an HTCondor ClassAd that details the status of your cache.
Registering the Cache¶
To be part of the OSG StashCache Federation, your cache must be
registered with the OSG. The service type is
XRootD cache server.
The resource may also specify which VOs it will serve data from.
To do this, add an
AllowedVOs list, with each line specifying a VO whose StashCache data the resource is willing to cache.
There are special values you can use in
ANY_PUBLICindicates that the cache is willing to serve public data from any VO.
ANYindicates that the cache is willing to serve data from any VO, both public and non-public.
There are extra requirements for serving non-public data:
- In addition to the cache allowing a VO in the
AllowedVOslist, that VO must also allow the cache in its
AllowedCacheslist. See the page on getting your VO's data into StashCache.
- There must be an authenticated XRootD instance on the cache server.
- There must be a
DNattribute in the resource with the DN of the cert used by the authenticated XRootD instance.
This is an example of a cache server that serves all public data:
MY_STASHCACHE_CACHE: FQDN: my-cache.example.net Service: XRootD cache server Description: StashCache cache server AllowedVOs: - ANY_PUBLIC
This is an example of a cache server that only serves authenticated data from the OSG VO:
MY_AUTH_STASHCACHE_CACHE: FQDN: my-auth-cache.example.net Service: XRootD cache server Description: StashCache cache server AllowedVOs: - OSG DN: /DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=my-auth-cache.example.net
Once the cache has been registered, open a help ticket with your cache name. Mention in your ticket that you would like to "Finalize the cache registration."