OSG Release Series¶

An OSG release series is a sequence of OSG software releases that are intended to provide a painless upgrade path. For example, the 3.2 release series contains OSG software 3.2.0, 3.2.1, 3.2.2, and so forth. A release series corresponds to a set of Yum software repositories, including ones for development, testing, and production use. The Yum repositories for one release series are completely distinct from the repositories for a different release series, even though they share many common packages. A particular release within a series is a snapshot of packages and their exact versions at one point in time. When you install software from a release series, say 3.2, you get the most current versions of software packages within that series, regardless of the current release version.

When a new series is released, it is an opportunity for the OSG Technology area to add major new software packages, make substantial updates to existing packages, and remove obsolete packages. When a new series is initially released, most packages are identical to the previous release, but two adjacent series will diverge over time.

Our goal is, within a series, that one may upgrade their OSG services via yum update cleanly and without any necessary config file changes or excessive downtime.

OSG Release Series¶

Since the start of the RPM-based OSG software stack, we have offered the following release series:

OSG 3.5 started August 2019. The main differences between it and 3.4 were the introduction of the HTCondor 8.8 and 8.9 series; also the RSV monitoring probes, EL6 support, and CREAM support were all dropped.
OSG 3.4 started June 2017 and will reach its end-of-life in November 2020. The main differences between it and 3.3 are the removal of edg-mkgridmap, GUMS, BeStMan, and VOMS Admin Server packages.
OSG 3.3 started in August 2015 and was end-of-lifed in May 2018. While the files have not been removed, it is strongly recommended that it not be installed anymore. The main differences between 3.3 and 3.2 are the dropping of EL5 support, the addition of EL7 support, and the dropping of Globus GRAM support.
OSG 3.2 started in November 2013, and was end-of-lifed in August 2016. The main differences between it and 3.1 were the introduction of glideinWMS 3.2, HTCondor 8.0, and Hadoop/HDFS 2.0; also the gLite CE Monitor system was dropped in favor of osg-info-services.
OSG 3.1 started in April 2012, and was end-of-lifed in April 2015. Historically, there were 3.0.x releases as well, but there was no separate release series for 3.0 and 3.1; we simply went from 3.0.10 to 3.1.0 in the same repositories.

OSG Upcoming¶

There is one more OSG Series called "upcoming" which contains major updates planned for a future release series. The yum repositories for upcoming (osg-upcoming and osg-upcoming-testing) are available from all OSG 3.x series, and individual packages can be installed from Upcoming without needing to update entirely to a new series. Note, however, that packages in the "upcoming" repositories are tested against the most recent OSG series. As of the time of writing, osg-upcoming is meant to work with OSG 3.5.

Installing an OSG Release Series¶

See the yum repositories document for instructions on installing the OSG repositories.

Updating to OSG 3.5¶

OS Version Support

OSG 3.5 only supports EL7

If you have an existing installation based on OSG release version <= 3.4 (which will be referred to as the old series), and want to upgrade to 3.5 (the new series), we recommend the following procedure:

First, remove the old series yum repositories:
```
[email protected] # rpm -e osg-release
```
This step ensures that any local modifications to *.repo files will not prevent installing the new series repos. Any modified *.repo files should appear under /etc/yum.repos.d/ with the *.rpmsave extension. After installing the new OSG repositories (the next step) you may want to apply any changes made in the *.rpmsave files to the new *.repo files.

Install the OSG repositories:

[email protected] # yum install https://repo.opensciencegrid.org/osg/3.5/osg-3.5-el7-release-latest.rpm

Clean yum cache:

[email protected] # yum clean all --enablerepo=*

Update software:
```
[email protected] # yum update
```
Info
- Please be aware that running yum update may also update other RPMs. You can exclude packages from being updated using the --exclude=[package-name or glob] option for the yum command.
- Watch the yum update carefully for any messages about a .rpmnew file being created. That means that a configuration file had been edited, and a new default version was to be installed. In that case, RPM does not overwrite the edited configuration file but instead installs the new version with a .rpmnew extension. You will need to merge any edits that have made into the .rpmnew file and then move the merged version into place (that is, without the .rpmnew extension). Watch especially for /etc/lcmaps.db, which every site is expected to edit.

Remove any deprecated packages that were previously installed:

[email protected] # yum remove osg-version \
                       osg-control \
                       'rsv*' \
                       glite-ce-cream-client-api-c \
                       glite-lbjp-common-gsoap-plugin \
                       xacml

If you did not have any of the above packages installed, Yum will not remove any packages:

No Match for argument: osg-version
No Match for argument: osg-control
No Match for argument: rsv*
No Match for argument: glite-ce-cream-client-api-c
No Match for argument: glite-lbjp-common-gsoap-plugin
No Match for argument: xacml
No Packages marked for removal

If you are updating an HTCondor-CE host, please consult the manual HTCondor and OSG Configure instructions below.

Running into issues?

If you are not having the expected result or having problems with Yum please see the Yum troubleshooting guide

Updating to HTCondor-CE 4.x¶

The OSG 3.5 release series contains HTCondor-CE 4, a major version upgrade from the previously released versions in the OSG. See the HTCondor-CE 4.0.0 release notes for an overview of the changes. In particular, this version includes a major reorganization of the default configuration so updates will require manual intervention. To update your HTCondor-CE host(s), perform the following steps:

Update all CE packages:

[email protected] # yum update htcondor-ce 'osg-ce*'

The new default condor_mapfile is sufficient since HTCondor-CE no longer relies on GSI authentication between its daemons. If /etc/condor-ce/condor_mapfile.rpmnew exists, replace your old condor_mapfile with the .rpmnew version:
```
[email protected] # mv /etc/condor-ce/condor_mapfile.rpmnew /etc/condor-ce/condor_mapfile
```
Merge any *.rpmnew files in /etc/condor-ce/config.d/
Additionally, you may wish to make one or more of the following optional changes:
- HTCondor-CE now disables batch system job retries by default. To re-enable job retries, set the following configuration in /etc/condor-ce/config.d/99-local.conf:
```
ENABLE_JOB_RETRIES = True
```
- For non-HTCondor sites that use remote CE requirements, the new version of HTCondor-CE accepts a simplified format. For example, a snippet from an example job route in the old format:
```
set_default_remote_cerequirements = strcat("Walltime == 3600 && AccountingGroup =="", x509UserProxyFirstFQAN, "\"");
```
  May be rewritten as the following:
```
set_WallTime = 3600;
set_AccountingGroup = x509UserProxyFirstFQAN;
set_default_CERequirements = "Walltime,AccountingGroup";
```

Reload and restart the HTCondor-CE daemons:

[email protected] # systemctl daemon-reload
[email protected] # systemctl restart condor-ce

Updating to HTCondor 8.8.x¶

The OSG 3.5 release series contains HTCondor 8.8, a major version upgrade from the previously released versions in the OSG. See the detailed update instructions below to update to HTCondor 8.8.

Updating to OSG Configure 3¶

The OSG 3.5 release series contains OSG-Configure 3, a major version upgrade from the previously released versions in the OSG. See the OSG Configure release notes for an overview of the changes. To update OSG Configure on your HTCondor-CE, perform the following steps:

If you haven't already, update to OSG 3.5.
If you have site_name set in /etc/osg/config.d/40-siteinfo.ini, delete it and specify resource instead. resource should match the resource name that's registered in OSG Topology.
Set resource_group in /etc/osg/config.d/40-siteinfo.ini to the resource group registered in OSG Topology, i.e. the name of the .yaml file in OSG Topology that contains the registered resouce above.
Set host_name to the host name that is registered in OSG Topology. This may be different from the FQDN of the host if you're using a DNS alias, for example.
OSG Configure will warn about config options that it does not recognize; delete these options from the config to get rid of the warnings.

Updating to HTCondor 8.9.7+¶

Where to find HTCondor 8.9

The HTCondor development series is available through the OSG upcoming repository.

For HTCondor hosts < 8.9.7 using the SciTokens CredMon, updates to HTCondor 8.9.7+ require manual intervention and a corresponding update to python2-scitokens-credmon, available in the OSG 3.5 release repository. If you do not have the python2-scitokens-credmon package installed, you may skip these instructions. Otherwise, follow these steps for a seamless update to HTCondor 8.9.7+:

Determine if your HTCondor installation is configured to use the SciTokens CredMon:
```
# condor_config_val -v DAEMON_LIST
```
If CREDD and SEC_CREDENTIAL_MONITOR are in the output of the above command, continue onto the next step. Otherwise, your installation is not configured to use SciTokens CredMon and you should skip the rest of these instructions.

Add the following to a file in /etc/condor/config.d/:

SEC_CREDENTIAL_DIRECTORY_OAUTH = /var/lib/condor/oauth_credentials
CREDMON_OAUTH = /usr/bin/condor_credmon_oauth
SEC_CREDENTIAL_MONITOR_OAUTH_LOG = $(LOG)/CredMonOAuthLog

if version < 8.9.7
  CREDMON_OAUTH = /usr/bin/scitokens_credmon
endif

Update your DAEMON_LIST configuration from:

DAEMON_LIST = $(DAEMON_LIST), CREDD, SEC_CREDENTIAL_MONITOR

to

DAEMON_LIST = $(DAEMON_LIST), CREDD, CREDMON_OAUTH

Turn off the SchedD and CredMon daemons:

# condor_off -daemon SCHEDD
# condor_off -daemon SEC_CREDENTIAL_MONITOR
# condor_off -daemon CREDMON_OAUTH

Move the existing credential directory and set up a temporary symlink:

# mv $(condor_config_val SEC_CREDENTIAL_DIRECTORY) $(condor_config_val SEC_CREDENTIAL_DIRECTORY_OAUTH)
# ln -s $(condor_config_val SEC_CREDENTIAL_DIRECTORY_OAUTH) $(condor_config_val SEC_CREDENTIAL_DIRECTORY)

Update HTCondor and SciTokens CredMon packages:

# yum -y upgrade python2-scitokens-credmon condor

If you are running Apache on this host, reload the Apache configuration:
```
# systemctl reload httpd.service
```
Reconfigure HTCondor:
```
# condor_reconfig
```

Turn the SchedD and CredMon daemons back on:

# condor_on -daemon CREDMON_OAUTH
# condor_on -daemon SCHEDD

Clean up old CredMon configuration. Remove the following entries from your HTCondor configuration:

SEC_CREDENTIAL_DIRECTORY = /var/lib/condor/credentials
SEC_CREDENTIAL_MONITOR = /usr/bin/scitokens_credmon
SEC_CREDENTIAL_MONITOR_LOG = $(LOG)/CredMonLog

To allow for seamless HTCondor downgrades, update the if version < 8.9.7 block that you added in step 2.

if version < 8.9.7
  CREDMON_OAUTH = /usr/bin/scitokens_credmon
  SEC_CREDENTIAL_DIRECTORY = $(SEC_CREDENTIAL_DIRECTORY_OAUTH)
endif

Remove the symlink to the old credential directory that you created in step 5. This is whatever you had set SEC_CREDENTIAL_DIRECTORY to before. For example:
```
# rm /var/lib/condor/credentials
```

Updating to HTCondor 8.8.x¶

The OSG 3.5 and 3.4 release series contain HTCondor 8.8, a major version upgrade from the previously released versions in the OSG. See the HTCondor 8.8 manual for an overview of the changes. To update HTCondor on your HTCondor-CE and/or HTCondor pool hosts, perform the following steps:

Update all HTCondor packages:

[email protected] # yum update 'condor*'

HTCondor pools only:
- The DAEMON_LIST, and CONDOR_HOST configuration changed in HTCondor 8.8. Additionally in OSG 3.5, the default security was changed to use FS and pool password. If you are experiencing issues with communication between hosts in your pool after the upgrade, the default OSG configuration is listed in /etc/condor/config.d/00-osg_default_*.config: ensure that any default configuration is overriden with your own DAEMON_LIST, CONDOR_HOST, and/or security configuration in subsequent files.
- As of HTCondor 8.8, MOUNT_UNDER_SCRATCH has default values of /tmp and /var/tmp, which may cause issues if your OSG_WN_TMP is a subdirectory of either of these directories. If the partition containing your execute directories is large enough, we recommend setting your OSG_WN_TMP to /tmp or /var/tmp. If that partition is not large enough, we recommend setting your OSG_WN_TMP variable to a directory outside of /tmp or /var/tmp.
HTCondor-CE hosts only: The HTCondor 8.8 series changed the default job route matching order from round-robin to first matching route. To use the old round-robin matching order, add the following configuration to /etc/condor-ce/config.d/99-local.conf:
```
JOB_ROUTER_ROUND_ROBIN_SELECTION = True
```

Clean-up deprecated packages:

[email protected] # yum remove 'rsv*' glite-ce-cream-client-api-c

OSG Release Series¶