Skip to content

Using rsv-control

Overview

This document is for System Administrators. It details the usage of the rsv-control command for enabling, disabling, testing and running RSV probes.

rsv-control provides an interface to many RSV tasks. rsv-control can view RSV jobs, run metrics, enable or disable metrics and consumers, and allow advanced configuration.

Warning

rsv-control can be used to configure RSV as described here and in the advanced configuration document. Most site admins will be able to configure RSV by editing /etc/osg/config.d/30-rsv.ini and running osg-configure as described in the installation doc.

Using rsv-control to configure is for advanced RSV use including enabling non-default metrics. Admins who don't use rsv-control for configuration can still use it to view their RSV jobs, run RSV tests, and help debug RSV problems. Anyone can view the jobs, but you must be root or the RSV user (rsv by default) to execute other commands, e.g. run, enable and disable probes, or to turn RSV on and off.

Viewing RSV jobs

rsv-control provides two different views: viewing the desired state and viewing the current actual state.

  • Desired = what metrics and consumers will start the next time RSV is started
  • Actual = what metrics and consumers are currently running

Desired state

To view the desired state, use the --list (-l for short) flag. This will create one table for each host showing the metrics that are enabled to run against that host.

[email protected]# rsv-control --list

Metrics enabled for host: osgitb1.nhn.ou.edu              | Service
----------------------------------------------------------+--------------------
org.osg.batch.jobmanager-default-status                   | OSG-CE
org.osg.batch.jobmanagers-available                       | OSG-CE
org.osg.certificates.cacert-expiry                        | OSG-CE
org.osg.certificates.crl-expiry                           | OSG-CE
org.osg.general.osg-directories-CE-permissions            | OSG-CE
org.osg.general.osg-version                               | OSG-CE
org.osg.general.ping-host                                 | OSG-CE
org.osg.general.vdt-version                               | OSG-CE
org.osg.general.vo-supported                              | OSG-CE
org.osg.globus.gram-authentication                        | OSG-CE
org.osg.globus.gridftp-simple                             | OSG-GridFTP
org.osg.gratia.condor                                     | OSG-CE
org.osg.gratia.metric                                     | OSG-CE


Metrics enabled for host: osg-edu.cs.wisc.edu:10443       | Service
----------------------------------------------------------+--------------------
org.osg.srm.srmcp-readwrite                               | OSG-SRM
org.osg.srm.srmping                                       | OSG-SRM

Other options:

  • To view all installed metrics use the --all (-a) flag along with --list. This will print an extra table showing metrics that are disabled on all hosts.
  • If you are having problems with the output being truncated, try the --wide (-w) flag.

Actual state

To view the current, running state of RSV jobs, use the --job-list flag (-j for short). This will show all metrics and consumers running in RSV. (It queries the underlying Condor Cron system that we use to run the metrics).

[email protected]# rsv-control --job-list

Hostname: osg-edu.cs.wisc.edu
     ID OWNER      ST NEXT RUN TIME   METRIC
  154.0 rsvuser    I  11-19 12:15     org.osg.certificates.cacert-expiry
  155.0 rsvuser    R  11-19 11:23     org.osg.gratia.metric
  156.0 rsvuser    I  11-19 18:47     org.osg.general.vdt-version
  157.0 rsvuser    I  11-19 12:30     org.osg.certificates.crl-expiry
  158.0 rsvuser    I  11-19 11:31     org.osg.globus.gram-authentication
  159.0 rsvuser    I  11-19 11:41     org.osg.general.osg-version
  160.0 rsvuser    R  11-19 11:25     org.osg.batch.jobmanager-default-status
  161.0 rsvuser    I  11-20 04:59     org.osg.batch.jobmanagers-available
  162.0 rsvuser    I  11-19 11:37     org.osg.general.osg-directories-CE-permissions
  163.0 rsvuser    I  11-19 12:08     org.osg.globus.gridftp-simple
  164.0 rsvuser    I  11-19 12:09     org.osg.gratia.condor
  165.0 rsvuser    R  11-19 11:27     org.osg.general.ping-host
  166.0 rsvuser    I  11-19 18:47     org.osg.general.vo-supported

Hostname: osg-edu.cs.wisc.edu:10443
     ID OWNER      ST NEXT RUN TIME   METRIC
  113.0 rsvuser    I  11-19 11:33     org.osg.srm.srmping
  114.0 rsvuser    R  11-19 11:28     org.osg.srm.srmcp-readwrite

     ID OWNER      ST CONSUMER
  198.0 rsvuser    R  html-consumer
  199.0 rsvuser    R  gratia-consumer

The ST field indicates the current job status:

  • R = the metric is currently running
  • I = the metric is idle and will be run at the next scheduled interval
  • Any other letter may indicate a problem
  • Consumers will always appear to be running even though they will only run once every five minutes.

Running a metric

rsv-control can be used to run metrics one time against a host. This can be useful for:

  • updating the status of a metric that had a problem instead of waiting until the next scheduled run time
  • testing a metric against a host before deciding whether to enable it

Note that the record for each run will be published to all active consumers. That is, it will be published to Gratia or will show up on your local web page, if you have those enabled.

Simplest test

Use the --run (-r) flag. You must also provide the --host flag. The syntax is:

rsv-control --run --host <HOST> <METRIC> [ <METRIC2> ...]

where <METRIC> is the full metric name (e.g. org.osg.general.osg-version). You can get the metric names from the --list output.

[email protected]# rsv-control --run \
    --host osg-edu.cs.wisc.edu org.osg.general.osg-version

Running metric org.osg.general.osg-version:

metricName: org.osg.general.osg-version
metricType: status
timestamp: 2010-11-19 11:40:19 CST
metricStatus: OK
serviceType: OSG-CE
serviceURI: osg-edu.cs.wisc.edu
gatheredAt: vdt-itb.cs.wisc.edu
summaryData: OK
detailsData: OSG 1.2.15
EOT

Note the metricStatus in the example above: that's where you can see if it was successful or not. In this case, it was successful, because it printed OK.

You may run multiple metrics against a single host by specifying multiple metrics to rsv-control.

In order to run metrics against multiple hosts you must run rsv-control multiple times, once for each host.

Running all enabled metrics

When RSV is first installed it can take up to a day for each enabled metric to run once. A new option is provided to force each metric to run immediately, for all hosts. Use the --all-enabled flag along with --run. With this option it is not necessary to specify a host - all enabled metrics for all configured hosts will be run (in fact, if you do specify a host it will be ignored).

[email protected]# rsv-control -r --all-enabled

Running metric org.osg.certificates.cacert-expiry (1 of 15)

metricName: org.osg.certificates.cacert-expiry
metricType: status
timestamp: 2010-11-19 13:44:08 CST
metricStatus: OK
serviceType: OSG-CE
serviceURI: osg-edu.cs.wisc.edu
gatheredAt: vdt-itb.cs.wisc.edu
summaryData: OK
detailsData: Security Probe Version: 1.1
OK: CAs are in sync with OSG distribution
EOT


...

Passing extra configuration

If you want to pass extra configuration when running a metric without editing its configuration file you can make an INI-formatted file and pass it on the command line. For example, you can make a file like this for the org.osg.srm.srmclient-ping metric (tmp-srm.ini):

[org.osg.srm.srmclient-ping args]
srm-destination-dir=/srmcache/~
srm-webservice-path=srm/v2/server

Then use the --extra-config-file parameter and pass the path to the INI file:

[email protected]# rsv-control -r --extra-config-file tmp-srm.ini \
    --host osg-edu.cs.wisc.edu:10443 org.osg.srm.srmclient-ping

Running metric org.osg.srm.srmclient-ping:

metricName: org.osg.srm.srmclient-ping
metricType: status
timestamp: 2010-11-19 14:12:35 CST
metricStatus: OK
serviceType: OSG-SRM
serviceURI: osg-edu.cs.wisc.edu:10443
gatheredAt: vdt-itb.cs.wisc.edu
summaryData: OK
detailsData: SRM server running on osg-edu.cs.wisc.edu is alive and responding to the srmping command.
.  Details: Storage Resource Manager (SRM) Client version 2.1.5-16
Copyright (c) 2002-2009 Fermi National Accelerator Laboratory

...

Enabling and disabling metrics and consumers

Metrics and consumers can be enabled or disabled by rsv-control using the --enable and --disable flags. Note that "enable" and "disable" are desired states (this is similar to osg-control). After enabling a metric you should turn it on if you want it to be running immediately. After disabling a metric that is running, you should still turn it off (a message will print after each of these actions to remind you of this behavior).

Enabling

The syntax for enabling metrics looks similar to the syntax for running metrics:

rsv-control --enable --host <HOST> <METRIC> [ <METRIC2> ...]

You must provide a host to enable the metric against (in order to enable a metric on multiple hosts you must run rsv-control once per host).

[email protected]# rsv-control --enable \
    --host osg-edu.cs.wisc.edu org.osg.gip.consistency
Enabling metric 'org.osg.gip.consistency' for host 'osg-edu.cs.wisc.edu'

One or more metrics have been enabled and will be started the next time RSV is started.  To turn them on immediately run 'rsv-control --on'.

Consumers do not run against a specific host, they process records for all hosts. When enabling consumers a host is not required (if a host is passed it will be ignored).

[email protected]# rsv-control --enable nagios-consumer
Enabling consumer nagios-consumer

Disabling

The syntax for disabling metrics looks similar to the syntax for running metrics:

rsv-control --disable --host <HOST> <METRIC> [ <METRIC2> ...]

You must provide a host to disable the metric against (in order to disable a metric on multiple hosts you must run rsv-control once per host).

[email protected]# rsv-control --disable \
    --host vdt-itb.cs.wisc.edu org.osg.local.containercert-expiry
Disabling metric 'org.osg.local.containercert-expiry' for host 'vdt-itb.cs.wisc.edu'

One or more metrics have been disabled and will not start the next time RSV is started.  You may still need to turn them off if they are currently running.

Consumers do not run against a specific host, they process records for all hosts. When disabling consumers a host is not required (if a host is passed it will be ignored).

[email protected]# rsv-control --disable html-consumer gratia-consumer
Disabling consumer html-consumer
Disabling consumer gratia-consumer
   Consumer already disabled

Metrics and consumers can both be listed in the same disable command.

Troubleshooting

Getting more information from rsv-control

The first step to getting more information is to run rsv-control with more verbosity. Use the --verbose (-v) flag. This flag can be used with any of rsv-control's abilities (run, enable, list, etc). The verbosity levels are:

  • 0 = print nothing
  • 1 = print warnings and errors along with usual output of command being run (1 is the default level)
  • 2 = adds informational messages
  • 3 = full debugging output

Using the RSV verify tool

The --verify flag will run some basic checks for your RSV installation:

[email protected]# rsv-control --verify
Testing if Condor-Cron is running...
OK

Testing if metrics are running...
OK (98 running metrics)

Testing if consumers are running...
OK (1 running consumers)

Checking which consumers are configured...
The following consumers are enabled: html-consumer
WARNING: The gratia-consumer is not enabled.  This indicates that your
         resource is not reporting to OSG.

This tool is still under development and it does only basic checks, but it is a good first step when debugging issues.

Running the RSV profiler

RSV has a tool to collect information useful for troubleshooting into a tarball that can be shared with the developers and support staff. To use it:

[email protected]# rsv-control --profile
Running the rsv-profiler...
OSG-RSV Profiler
Analyzing...
Making tarball (rsv-profiler.tar.gz)

Note

If you are getting assistance via the trouble ticket system, you must add a .txt extension to the tarball so it can be uploaded.

[email protected]# mv rsv-profiler.tar.gz rsv-profiler.tar.gz.txt
This material is based upon work supported by the National Science Foundation under Grant Nos. 1148698 and 1836650 . Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.