Warning

This is a technology preview document and will probably change content and location withouth notice.

Installation of FileBeats for Submit nodes¶

This document is for frontend administrators. It describes the installation of Filebeats to continuously upload the HTCondor submit host transfer log to Elastic Search.

Introduction¶

A submit host (HTCondor schedd) is a login node where users submit jobs to the Grid. One interesting log that it produces is the TransferLog. The TransferLogs report all the transfers of files between compute node and submit nodes. In this guide we describe the installation of Filebeats to upload this log to Elastic Search.

Installation¶

FileBeat Installation¶

For the installation of filebeats follow the official instruction to set up the repositories and install filebeats as described here.

Configuration¶

Configuration of Filebeats¶

The configuration of filebeats revolves around this file /etc/filebeat/filebeat.yml. Bellow are the steps to modify the different sections of this file

The Filebeat Inputs section, the input should look like this:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/condor/XferStatsLog

The output logstash section should look like:

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["gracc.opensciencegrid.org:6938"]
  # Optional SSL. By default is off. 
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/etc/grid-security/certificates/InCommon-IGTF-Server-CA.pem"]
  #  Certificate for SSL client authentication
  ssl.certificate: "/etc/grid-security/hostcert.pem"
  # Client Certificate Key
  ssl.key: "/etc/grid-security/hostkey.pem"

Comment out all of the Elasticsearch output since we are using LogStash

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]

# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"

The general section should look like this, where <HOSTNAME> should be replaced by the hostname of the machine you are installing filebeats on.
```
#================================ General =====================================
name: <HOSTNAME>
tags: ["xfer-log"]
```
Test that the configuration is correct by running:
```
[email protected] # filebeat test config
```

Start the filebeats services:

[email protected] # service filebeat start

Configuration of HTCondor¶

For the configuration of the HTCondor submit host to use the TransferLog follow the next instructions:

Note

The transfer metrics was introduced in HTCondor 8.6 series. You need to be running a version equal or greater than 8.6.1 to enable it.

Create a file named /etc/condor/config.d/50-transferLog.config with the following contents:

SHADOW_DEBUG = D_STATS
SHADOW_STATS_LOG = $(LOG)/XferStatsLog
STARTER_STATS_LOG = $(LOG)/XferStatsLog

Reconfigure condor:
```
[email protected] # condor_reconfig
```
Make sure that after a couple of minutes the new log /var/log/condor/XferStatsLog is present.