Skip to content

Topology Service

This document contains information about the service that runs https://topology.opensciencegrid.org and https://topology-itb.opensciencegrid.org. (These sites are also accessible as https://my.opensciencegrid.org and https://my-itb.opensciencegrid.org.)

The source code for the service is in https://github.com/opensciencegrid/topology, in the src/ subdirectory. This repository also contains the public part of the data that gets served.

Deployment

Topology is a webapp run with Apache on the host topology.opensciencegrid.org. The ITB instance runs on the host topology-itb.opensciencegrid.org. The hosts are VMs at Nebraska; for SSH access, contact Derek Weitzel or Brian Bockelman.

Installation

These instructions assume an EL 7 host with the EPEL repositories available. The software will be installed into /opt/topology. (The ITB instance should be installed into /opt/topology-itb instead.) The following steps should be done as root.

  1. Install prerequisites:

    # yum install python36 gridsite httpd mod_ssl

  2. Clone the repository:

    # cd /opt
# git clone https://github.com/opensciencegrid/topology

    (move the resulting directory to topology-itb for the ITB instance)

  3. Set up the virtualenv in the clone -- from /opt/topology or /opt/topology-itb:

    # python36 -m venv venv
# . ./venv/bin/activate
# pip install -r requirements.txt

File system locations

The following files/directories must exist and have the proper permissions:

Location Purpose Ownership Mode
/opt/topology Production software install root:root 0644
/opt/topology-itb ITB software install root:root 0644
/etc/opt/topology/config-production.py Production config root:root 0644
/etc/opt/topology/config-itb.py ITB config root:root 0644
/etc/opt/topology/bitbucket Private key for contact info repo apache:root 0600
/etc/opt/topology/bitbucket.pub Public key for contact info repo apache:root 0644
~apache/.ssh SSH dir for Apache apache:root 0700
~apache/.ssh/known_hosts Known hosts file for Apache apache:root 0644
/var/cache/topology Checkouts of topology and contacts data for production instance apache:apache 0755
/var/cache/topology-itb Checkouts of topology and contacts data for ITB instance apache:apache 0755

~apache/.ssh/known_hosts must contain an entry for bitbucket.org; use ssh-keyscan bitbucket.org to get the appropriate entry.

Software configuration

Configuration is in /etc/opt/topology/config-production.py and config-itb.py. The files are in Python format and override default settings in src/webapp/default_config.py in the topology repo.

HTTPD configuration is in /etc/httpd; we use the modules mod_ssl, mod_gridsite, and mod_wsgi. The first two are installed via yum; the .so file for mod_wsgi is located in /opt/topology/venv/lib/python3.6/site-packages/mod_wsgi/server/ or /opt/topology-itb/venv/lib/python3.6/site-packages/mod_wsgi/server/ for the ITB instance.

TODO Derek can you write something here?

Data configuration

Configuration is in /etc/opt/topology/config-production.py and config-itb.py.

Variable Purpose
TOPOLOGY_DATA_DIR The directory containing a clone of the topology repository for data use
TOPOLOGY_DATA_REPO The remote tracking repository of TOPOLOGY_DATA_DIR
TOPOLOGY_DATA_BRANCH The remote tracking branch of TOPOLOGY_DATA_DIR
CONTACT_DATA_DIR The directory containing a clone of the contact repository for data use
CONTACT_DATA_REPO The remote tracking repository of CONTACT_DATA_DIR
(default: "[email protected]:opensciencegrid/contact.git")
CONTACT_DATA_BRANCH The remote tracking branch of CONTACT_DATA_BRANCH
(default: "master")
CACHE_LIFETIME Frequency of automatic data updates in seconds
(default: 900)

Puppet ensures that the production contact and topology clones are up to date with their configured remote tracking repo and branch. Puppet does not manage the ITB data directories so they need to be updated by hand during testing.

Testing changes on the ITB instance

All changes should be tested on the ITB instance before deploying to production. If you can, test them on your local machine first. These instructions assume that the code has not been merged to master.

  1. Update the ITB software installation at /opt/topology-itb and note the current branch:

    # cd /opt/topology-itb
# git fetch --all
# git status

  2. Check out the branch you are testing. If the target remote is not configured, add it:

    # git checkout -b <BRANCH> <REMOTE>/<BRANCH NAME>

  3. Verify that you are using the intended data associated with the code you are testing:

    1. If the data format has changed in an incompatible way, modify /etc/opt/topology/config-itb.py:

      1. Backup the ITB configuration file:

        # cd /etc/opt/topology
# cp config-itb.py{,.bak}

      2. Change the TOPOLOGY_DATA_DIR and/or CONTACT_DATA_DIR lines to point to a new directories so the previous data does not get overwritten with incompatible data.

    2. If you need to use a different branch for the data, switch to it:

      1. Check the branch of TOPOLOGY_DATA_DIR from /etc/opt/topology/config-itb.py

        # cd <TOPOLOGY_DATA_DIR>
# git fetch --all
# git status

      2. Note the previous branch, you will need this later

      3. If the target remote is not configured, add it
      4. Check out the target branch:
        # git checkout -b <BRANCH NAME> <REMOTE>/<BRANCH NAME>

    3. Pull any upstream changes to ensure that your branch is up to date:

      # git pull

  4. Restart httpd:

    # systemctl restart httpd

  5. Test the web interface at https://topology-itb.opensciencegrid.org.

    Errors and output are in /var/log/httpd/error_log.

Reverting changes

  1. Switch /opt/topology-itb to the previous branch:

    # cd /opt/topology-itb
# git checkout <BRANCH>

  2. If you made config changes to /etc/opt/topology/config-itb.py, restore the backup.

  3. If you checked out a different branch for data, revert it back to the old branch.

  4. Restart httpd:

    # systemctl restart httpd

  5. Test the web interface at https://topology-itb.opensciencegrid.org.

Updating the production instance

Updating the production instance is similar to updating ITB instance.

  1. Update master on the Git clone at /opt/topology:

    # cd /opt/topology
# git pull origin master

  2. Make config changes to /etc/opt/topology/config-production.py if necessary.

  3. Restart httpd:

    # systemctl restart httpd

  4. Test the web interface at https://topology.opensciencegrid.org.

    Errors and output are in /var/log/httpd/error_log.

Reverting changes

  1. Switch /opt/topology to the previous master:

    # cd /opt/topology
### (use `git reflog` to find the previous commit that was used)
# git reset --hard <COMMIT>

  2. If you made config changes to /etc/opt/topology/config-production.py, revert them.

  3. Restart httpd:

    # systemctl restart httpd

  4. Test the web interface at https://topology.opensciencegrid.org.