Using rsv-control¶
Overview¶
This document is for System Administrators. It details the usage of the rsv-control command for enabling, disabling, testing and running RSV probes.
rsv-control provides an interface to many RSV tasks. rsv-control can view RSV jobs, run metrics, enable or disable metrics and consumers, and allow advanced configuration.
Warning
rsv-control can be used to configure RSV as described here and in the advanced configuration document. Most site admins will be able to configure RSV by editing /etc/osg/config.d/30-rsv.ini and running osg-configure as described in the installation doc.
Using rsv-control to configure is for advanced RSV use including enabling non-default metrics. Admins who don't use rsv-control for configuration can still use it to view their RSV jobs, run RSV tests, and help debug RSV problems. Anyone can view the jobs, but you must be root or the RSV user (rsv by default) to execute other commands, e.g. run, enable and disable probes, or to turn RSV on and off.
Viewing RSV jobs¶
rsv-control provides two different views: viewing the desired state and viewing the current actual state.
- Desired = what metrics and consumers will start the next time RSV is started
- Actual = what metrics and consumers are currently running
Desired state¶
To view the desired state, use the --list (-l for short) flag. This will create one table for each host showing the metrics that are enabled to run against that host.
[email protected]# rsv-control --list
Metrics enabled for host: osgitb1.nhn.ou.edu | Service
----------------------------------------------------------+--------------------
org.osg.batch.jobmanager-default-status | OSG-CE
org.osg.batch.jobmanagers-available | OSG-CE
org.osg.certificates.cacert-expiry | OSG-CE
org.osg.certificates.crl-expiry | OSG-CE
org.osg.general.osg-directories-CE-permissions | OSG-CE
org.osg.general.osg-version | OSG-CE
org.osg.general.ping-host | OSG-CE
org.osg.general.vdt-version | OSG-CE
org.osg.general.vo-supported | OSG-CE
org.osg.globus.gram-authentication | OSG-CE
org.osg.globus.gridftp-simple | OSG-GridFTP
org.osg.gratia.condor | OSG-CE
org.osg.gratia.metric | OSG-CE
Metrics enabled for host: osg-edu.cs.wisc.edu:10443 | Service
----------------------------------------------------------+--------------------
org.osg.srm.srmcp-readwrite | OSG-SRM
org.osg.srm.srmping | OSG-SRM
Other options:
- To view all installed metrics use the
--all(-a) flag along with--list. This will print an extra table showing metrics that are disabled on all hosts. - If you are having problems with the output being truncated, try the
--wide(-w) flag.
Actual state¶
To view the current, running state of RSV jobs, use the --job-list flag (-j for short). This will show all metrics and consumers running in RSV. (It queries the underlying Condor Cron system that we use to run the metrics).
[email protected]# rsv-control --job-list
Hostname: osg-edu.cs.wisc.edu
ID OWNER ST NEXT RUN TIME METRIC
154.0 rsvuser I 11-19 12:15 org.osg.certificates.cacert-expiry
155.0 rsvuser R 11-19 11:23 org.osg.gratia.metric
156.0 rsvuser I 11-19 18:47 org.osg.general.vdt-version
157.0 rsvuser I 11-19 12:30 org.osg.certificates.crl-expiry
158.0 rsvuser I 11-19 11:31 org.osg.globus.gram-authentication
159.0 rsvuser I 11-19 11:41 org.osg.general.osg-version
160.0 rsvuser R 11-19 11:25 org.osg.batch.jobmanager-default-status
161.0 rsvuser I 11-20 04:59 org.osg.batch.jobmanagers-available
162.0 rsvuser I 11-19 11:37 org.osg.general.osg-directories-CE-permissions
163.0 rsvuser I 11-19 12:08 org.osg.globus.gridftp-simple
164.0 rsvuser I 11-19 12:09 org.osg.gratia.condor
165.0 rsvuser R 11-19 11:27 org.osg.general.ping-host
166.0 rsvuser I 11-19 18:47 org.osg.general.vo-supported
Hostname: osg-edu.cs.wisc.edu:10443
ID OWNER ST NEXT RUN TIME METRIC
113.0 rsvuser I 11-19 11:33 org.osg.srm.srmping
114.0 rsvuser R 11-19 11:28 org.osg.srm.srmcp-readwrite
ID OWNER ST CONSUMER
198.0 rsvuser R html-consumer
199.0 rsvuser R gratia-consumer
The ST field indicates the current job status:
- R = the metric is currently running
- I = the metric is idle and will be run at the next scheduled interval
- Any other letter may indicate a problem
- Consumers will always appear to be running even though they will only run once every five minutes.
Running a metric¶
rsv-control can be used to run metrics one time against a host. This can be useful for:
- updating the status of a metric that had a problem instead of waiting until the next scheduled run time
- testing a metric against a host before deciding whether to enable it
Note that the record for each run will be published to all active consumers. That is, it will be published to Gratia or will show up on your local web page, if you have those enabled.
Simplest test¶
Use the --run (-r) flag. You must also provide the --host flag. The syntax is:
rsv-control --run --host <HOST> <METRIC> [ <METRIC2> ...]
where <METRIC> is the full metric name (e.g. org.osg.general.osg-version). You can get the metric names from the --list output.
[email protected]# rsv-control --run \
--host osg-edu.cs.wisc.edu org.osg.general.osg-version
Running metric org.osg.general.osg-version:
metricName: org.osg.general.osg-version
metricType: status
timestamp: 2010-11-19 11:40:19 CST
metricStatus: OK
serviceType: OSG-CE
serviceURI: osg-edu.cs.wisc.edu
gatheredAt: vdt-itb.cs.wisc.edu
summaryData: OK
detailsData: OSG 1.2.15
EOT
Note the metricStatus in the example above: that's where you can see if it was successful or not. In this case, it was successful, because it printed OK.
You may run multiple metrics against a single host by specifying multiple metrics to rsv-control.
In order to run metrics against multiple hosts you must run rsv-control multiple times, once for each host.
Running all enabled metrics¶
When RSV is first installed it can take up to a day for each enabled metric to run once. A new option is provided to force each metric to run immediately, for all hosts. Use the --all-enabled flag along with --run. With this option it is not necessary to specify a host - all enabled metrics for all configured hosts will be run (in fact, if you do specify a host it will be ignored).
[email protected]# rsv-control -r --all-enabled
Running metric org.osg.certificates.cacert-expiry (1 of 15)
metricName: org.osg.certificates.cacert-expiry
metricType: status
timestamp: 2010-11-19 13:44:08 CST
metricStatus: OK
serviceType: OSG-CE
serviceURI: osg-edu.cs.wisc.edu
gatheredAt: vdt-itb.cs.wisc.edu
summaryData: OK
detailsData: Security Probe Version: 1.1
OK: CAs are in sync with OSG distribution
EOT
...
Passing extra configuration¶
If you want to pass extra configuration when running a metric without editing its configuration file you can make an INI-formatted file and pass it on the command line. For example, you can make a file like this for the org.osg.srm.srmclient-ping metric (tmp-srm.ini):
[org.osg.srm.srmclient-ping args]
srm-destination-dir=/srmcache/~
srm-webservice-path=srm/v2/server
Then use the --extra-config-file parameter and pass the path to the INI file:
[email protected]# rsv-control -r --extra-config-file tmp-srm.ini \
--host osg-edu.cs.wisc.edu:10443 org.osg.srm.srmclient-ping
Running metric org.osg.srm.srmclient-ping:
metricName: org.osg.srm.srmclient-ping
metricType: status
timestamp: 2010-11-19 14:12:35 CST
metricStatus: OK
serviceType: OSG-SRM
serviceURI: osg-edu.cs.wisc.edu:10443
gatheredAt: vdt-itb.cs.wisc.edu
summaryData: OK
detailsData: SRM server running on osg-edu.cs.wisc.edu is alive and responding to the srmping command.
. Details: Storage Resource Manager (SRM) Client version 2.1.5-16
Copyright (c) 2002-2009 Fermi National Accelerator Laboratory
...
Enabling and disabling metrics and consumers¶
Metrics and consumers can be enabled or disabled by rsv-control using the --enable and --disable flags. Note that "enable" and "disable" are desired states (this is similar to osg-control). After enabling a metric you should turn it on if you want it to be running immediately. After disabling a metric that is running, you should still turn it off (a message will print after each of these actions to remind you of this behavior).
Enabling¶
The syntax for enabling metrics looks similar to the syntax for running metrics:
rsv-control --enable --host <HOST> <METRIC> [ <METRIC2> ...]
You must provide a host to enable the metric against (in order to enable a metric on multiple hosts you must run rsv-control once per host).
[email protected]# rsv-control --enable \
--host osg-edu.cs.wisc.edu org.osg.gip.consistency
Enabling metric 'org.osg.gip.consistency' for host 'osg-edu.cs.wisc.edu'
One or more metrics have been enabled and will be started the next time RSV is started. To turn them on immediately run 'rsv-control --on'.
Consumers do not run against a specific host, they process records for all hosts. When enabling consumers a host is not required (if a host is passed it will be ignored).
[email protected]# rsv-control --enable nagios-consumer
Enabling consumer nagios-consumer
Disabling¶
The syntax for disabling metrics looks similar to the syntax for running metrics:
rsv-control --disable --host <HOST> <METRIC> [ <METRIC2> ...]
You must provide a host to disable the metric against (in order to disable a metric on multiple hosts you must run rsv-control once per host).
[email protected]# rsv-control --disable \
--host vdt-itb.cs.wisc.edu org.osg.local.containercert-expiry
Disabling metric 'org.osg.local.containercert-expiry' for host 'vdt-itb.cs.wisc.edu'
One or more metrics have been disabled and will not start the next time RSV is started. You may still need to turn them off if they are currently running.
Consumers do not run against a specific host, they process records for all hosts. When disabling consumers a host is not required (if a host is passed it will be ignored).
[email protected]# rsv-control --disable html-consumer gratia-consumer
Disabling consumer html-consumer
Disabling consumer gratia-consumer
Consumer already disabled
Metrics and consumers can both be listed in the same disable command.
Troubleshooting¶
Getting more information from rsv-control¶
The first step to getting more information is to run rsv-control with more verbosity. Use the --verbose (-v) flag. This flag can be used with any of rsv-control's abilities (run, enable, list, etc). The verbosity levels are:
- 0 = print nothing
- 1 = print warnings and errors along with usual output of command being run (1 is the default level)
- 2 = adds informational messages
- 3 = full debugging output
Using the RSV verify tool¶
The --verify flag will run some basic checks for your RSV installation:
[email protected]# rsv-control --verify
Testing if Condor-Cron is running...
OK
Testing if metrics are running...
OK (98 running metrics)
Testing if consumers are running...
OK (1 running consumers)
Checking which consumers are configured...
The following consumers are enabled: html-consumer
WARNING: The gratia-consumer is not enabled. This indicates that your
resource is not reporting to OSG.
This tool is still under development and it does only basic checks, but it is a good first step when debugging issues.
Running the RSV profiler¶
RSV has a tool to collect information useful for troubleshooting into a tarball that can be shared with the developers and support staff. To use it:
[email protected]# rsv-control --profile
Running the rsv-profiler...
OSG-RSV Profiler
Analyzing...
Making tarball (rsv-profiler.tar.gz)
Note
If you are getting assistance via the trouble ticket system, you must add a .txt extension to the tarball so it can be uploaded.
[email protected]# mv rsv-profiler.tar.gz rsv-profiler.tar.gz.txt