OSG Build Tools¶
This page documents the tools used for RPM development for the OSG Software Stack. See the RPM development guide for the principles on which these tools are based.
The tools are distributed in the
osg-build RPM in our repositories, but can also be used from a Git clone of opensciencegrid/osg-build on GitHub.
This page is up-to-date as of
osg-build version 1.10.1.
This is the primary tool used in building source and binary RPMs.
osg-build <TASK> [options] <PACKAGE DIRECTORY> [...]
package_directory is a directory containing an
osg/ and/or an
upstream/ subdirectory. See the RPM development guide for how these directories are organized.
Prebuilds the final source package, then builds it remotely using the Koji instance hosted at UW-Madison. https://koji.chtc.wisc.edu By default, the resulting RPMs will end up in the osg-minefield repositories based on the most recent OSG major version (e.g. 3.4). You may specify a different set of repos with
--repo, described later. RPMs from the osg-minefield repositories are regularly pulled to the osg-development repositories hosted by the GOC at http://repo.opensciencegrid.org Unless you specify otherwise (by passing
--el7 or specifying a different koji tag/target), the package will be built for both el6 and el7. This is the method used to build final versions of packages you expect to ship.
Prebuilds the final source package, then runs
rpmlint on it to check for various problems. You will need to have
rpmlint installed. People on UW CSL machines should add
/p/vdt/workspace/rpmlint to their $PATH.
Prebuilds the final source package, then builds it locally using
mock, and stores the resulting source and binary RPMs in the package-specific
Prebuilds the final source package from upstream sources (if any) and local files (if any). May create or overwrite the
Prebuilds the final source package, then calls
rpmbuild -bp on the result, extracting and patching the source files (and performing any other steps defined in the
%prep section of the spec file. The resulting sources will be under
Prebuilds the final source package, then builds it locally using
rpmbuild, and stores the resulting source and binary RPMs in the package-specific
Collects the upstream local sources and spec file, then calls
quilt setup on the spec file, extracting the source files and adding the patches to a quilt series file. See Quilt documentation (PDF link) for more information on quilt; also look at the example in the Usage Patterns section below. Similar to
prepare (in fact,
rpmbuild -bp behind the scenes), but the source tree is in pre-patch state, and various quilt commands can be used to apply and modify patches. Unpacks into
_quilt as of
_final_srpm_contents in previous versions. Requires
quilt. People on UW CSL machines should add
/p/vdt/workspace/quilt/bin to their
/p/vdt/workspace/quilt/share/man to their
This section lists the command-line options.
Prints the built-in usage information and exits without doing anything else.
Prints the version of
osg-build and exits without doing anything else.
-a, --autoclean, --no-autoclean¶
Before each build, clean out the contents of the underscore directories (_build_results, _final_srpm_contents, _upstream_srpm_contents, _upstream_tarball_contents). If the directories are not cleaned up, earlier builds of a package may interfere with later ones.
--no-autoclean will disable this.
Has no effect with the
-c, --cache-prefix prefix¶
Sets the prefix for upstream source cache references. The prefix must be a valid URI starting with either
file, or one of the following special values:
- AFS (corresponds to
file:///p/vdt/public/html/upstream, which is the location of the VDT cache using AFS from a UW CS machine).
- VDT (corresponds to
http://vdt.cs.wisc.edu/upstream, which is the location of the VDT cache from off-site).
- AUTO (AFS if available, VDT if not)
The upstream source cache must be organized as described above. All files referenced by
.source files in the affected packages must exist in the cache, or a runtime error will occur.
Has no effect with the
--el6, --el7, --redhat-release version (Config: redhat_release)¶
Sets the distro version to build for. This affects the %dist tag, the mock config, and the default koji tag and target (unless otherwise specified).
--el6 is equivalent to
--el7 is equivalent to
Sets the verbosity of the script. Valid values are:
Do not display as much information. Equivalent to
Display more information. Equivalent to
-w, --working-directory path¶
Use path as the root directory of the files created by the script. For example, if path is
$HOME/working, and the package being built is
ndt, the following tree will be created:
If path is
TEMP, a randomly named directory under
/tmp is used as the working directory.
The default setting is to use the package directory as the working directory.
Has no effect with the
Options specific to prebuild task¶
If set, all upstream tarballs will be extracted into
_upstream_tarball_contents/ during the prebuild step. This flag is now mostly redundant with the
Options specific to rpmbuild and mock tasks¶
Sets the distribution tag added on to the end of the release in the RPM (
mock tasks only ).
-t, --target-arch arch¶
Specify an architecture to build packages for (
mock tasks only ).
Default is unspecified, which builds for the current machine architecture.
Options specific to mock task¶
Enable/disable deletion of the mock buildroot after a successful build.
-m, --mock-config path¶
mock configuration file to use. This file details how to set up the build environment used by mock for the build, including Yum repositories from which to install dependencies and certain predefined variables (e.g., the distribution tag
--mock-config-from-koji build tag¶
Creates a mock config from a Koji build tag. This is the most accurate way to replicate the build environment that Koji will provide (outside of Koji). The build tag is based on the distro version (el6, el7) and the OSG major version (3.3, 3.4). For 3.4 on el6, it is:
osg-3.4-el6-build Also requires the Koji command-line tools (package
koji), obtainable from the osg repositories. Since this uses koji, some of the koji-specific options may apply, namely:
Options specific to koji task¶
Do not actually run koji, merely show the command(s) that will be run. For debugging purposes.
For scratch builds without
--vcs only. Download the resulting RPMs and logs from the build into the
-k, --kojilogin, --koji-login login¶
Sets the login to use for the koji task. This should most likely be your CN. If not specified, will extract it from your client cert (
The koji target to use for building.
osg-el6 for el6 and
osg-el7 for el7.
The koji tag to add packages to. See the Koji Workflow guide for more information on the terminology. The special value
TARGET uses the destination tag defined in the koji target.
--ktt, --koji-tag-and-target arg¶
Shorthand for setting both --koji-tag and --koji-target to arg.
Enable/disable use of the
osg-koji wrapper script around koji. See below for a description of
Specifies the method osg-build will use to interface with Koji. This can be
--wait, --no-wait, --nowait¶
Wait for koji tasks to finish. Bad for running multiple builds in a single command, since you will have to type in your passphrase for the first one, wait for it to complete, then type in your passphrase for the second one, wait for it to complete, etc. If you want to wait for multiple tasks to finish, use the
koji watch-task command or look at the website https://koji.chtc.wisc.edu.
--wait used to be the default until
regen-repo koji task on the build tag after each koji build, to update the build repository used for the next build. Not useful unless you are launching multiple builds. This enables you to launch builds that depend on each other. Doesn't work too well with
--no-wait, since the next build may be started before the regen-repo task is complete. Waiting will keep the next build from being queued until the regen-repo is complete.
Perform scratch builds. A scratch build does not go into a repository, but the name-version-release (NVR) of the created RPMs are not considered used, so the build may be modified and repeated without needing a release bump. This has the same use case as the mock task: creating packages that you want to test before releasing. If you do not have a machine with mock set up, or want to test exactly the environment that Koji provides, scratch builds might be more convenient.
--vcs, --no-vcs, --svn, --no-svn¶
Have Koji check the package out from a version control system instead of creating an SRPM on the local machine and submitting that to Koji. Currently, SVN and Git are supported. If this flag is specified, you may use SVN URLs or [email protected] pairs to specify the packages to build. You may continue specify package directories from an SVN checkout, in which case osg-build will use
svn info to find the right [email protected] pair to use and warn you about uncommitted changes. osg-build will also warn you about an outdated working directory.
--vcs defaults to
true for non-scratch builds, and
false for scratch builds.
--repo=destination repository, --upcoming¶
Selects the repositories (osg-3.3, upcoming, etc.) to build packages for. Currently valid repositories are:
|osg||OSG Software development repos for trunk (this is the default)|
|osg-3.3 (or just 3.3)||OSG Software development repos for 3.2 branch|
|upcoming||OSG Software development repos for upcoming branch|
|internal||OSG Software internal branch|
|hcc||Holland Computing Center (Nebraska) testing repos|
--upcoming is an alias for
Note that the repo selection affects which VCS paths you are allowed to build from. For example, you are not allowed to build from branches/osg-3.3 (from the OSG SVN) into the 'osg' repo, or from HCC's git repositories into the 'upcoming' repo.
This script displays the differences between the latest packages in two koji tags.
koji-tag-diff osg-3.4-el6-development osg-3.4-el7-testing
This prints the packages that are in osg-3.4-el6-development but not in osg-3.4-el7-testing, or vice versa.
This script runs automated tests for
osg-build. Only a few tests have been implemented so far.
This is a script to fetch an SRPM from a remote site, copy it into the upstream cache on AFS, and create an SVN package dir (if needed) with an
upstream/*.source file. By default it will put downloaded files into the VDT upstream cache (/p/vdt/public/html/upstream), but you can pass
--upstream-root=<UPSTREAM DIR> to put them somewhere else. If called with the
-e argument, it will extract the spec file from the SRPM and place it into the
osg subdir in SVN. If called with the
-d argument, it will extract the spec file and compare it to the existing spec file in the
osg subdir. The script hasn't been touched in a while and needs a good deal of cleanup. A planned feature is to allow doing a three-way diff between the existing RPM before OSG modifications, the new RPM before OSG modifications and the existing RPM after OSG modifications.
This is a wrapper script around the
koji command line tool. It automatically specifies parameters to access the OSG's koji instance, and forces SSL authentication. It takes the same parameters as
koji and passes them on.
An additional command,
osg-koji setup exists, which performs the following tasks:
- Create a koji configuration in
- Create a CA bundle for verifying the server.
Use either files in
/etc/grid-security/certificates, or (if those are not found), from files downloaded from the DOEGrids and DigiCert sites.
- Create a client cert file. This can be a symlink to your grid proxy, or it can be a file created from your grid public and private key files.
The location of those files can be specified by the
--userkeyarguments. If unspecified,
Run this script to push packages from one set of repos to another (e.g. from development to testing), according to the OSG software promotion guidelines.
Once the packages are promoted, the script will generate code to cut and paste into a JIRA comment.
osg-promote [-r|--route <ROUTE>]... [options] <PACKAGE OR BUILD> [...]
Promote the latest build of
osg-versionto testing for the current release series
osg-promote -r testing osg-version
Promote the latest builds of
osg-ceto testing for the 3.3 and 3.4 release series
osg-promote -r 3.3-testing -r 3.4-testing osg-ce
osg-build-1.5.0-1to testing for the current release series
osg-promote -r testing osg-build-1.5.0-1
Display help and a list of valid routes.
package or build¶
A package (e.g.
osg-version) or build (e.g.
osg-version-3.3.0-1.osg33.el6) to promote. You may omit the dist tag (the
If a package is specified, the most recent version of that package will be promoted.
If a build is specified, that build and the build that has the same version-release for the other distro version(s) will be promoted. That is, if you specify the route
3.3-testing and the build
foo-1-1.osg33.el7 will be promoted.
This may be specified multiple times, to promote multiple packages. The NVRs of each set of builds for a package must match.
-r ROUTE, --route ROUTE¶
The promotion route to use. Use
osg-promote -h to get a list of valid routes. This may be specified multiple times. For example, to promote for both 3.4 and 3.3, pass
-r 3.4-testing -r 3.3-testing.
If not specified, the
testing route is used, which corresponds to the testing route for the latest release series.
Do not promote, just show what would be done.
--el6-only / --el7-only¶
Only promote packages for el6 / el7.
--no-el6 / --no-el7¶
Do not promote packages for el6 / el7.
Ignore rejections due to version mismatch between dvers or missing package for one dver.
Regenerate the destination repos after promoting.
Do not prompt before promotion.
Common Usage Patterns¶
Verify that all files necessary to build the package are in the right place¶
osg-build prebuild <PACKAGEDIR>.
Fetch and extract all source files for examination¶
osg-build prebuild --full-extract <PACKAGEDIR>. Look inside the
Get a post-patch version of the upstream sources for examination¶
osg-build prepare <PACKAGEDIR>. Look inside the
See which patches work with a new version of a package, update or remove them¶
- Place the new source tarball into the upstream cache, edit the version in the spec file and *.sources files as necessary
osg-build quilt <PACKAGEDIR>.
- Enter the extracted sources inside the
_final_srpm_contentsdirectory. You should see a file called
seriesand a symlink called
quilt seriesto get a list of patches in order of application.
quilt pushto apply the next patch.
- If the patch applies cleanly, continue.
- If the patch applies with some fuzz, type
quilt refreshto update the offsets in the patch.
- If the patch does not apply and you wish to remove it, type
quilt delete <PATCH NAME>(delete only removes it from the series file, not the disk)
- If the patch does not apply and you wish to fix it, either type
quilt push -fto interactively apply the patch, or
quilt delete <PATCH NAME>the patch and use
quilt refreshto edit files and make a new patch from your changes. Consult the
quilt(1)manpage for more info.
- If you have a new patch, run
quilt import <PATCHFILE>to add the patch to the series file, and run
quilt pushto apply it.
- If you have changes to make to the source code that you want to save as a patch, type
quilt new <PATCHNAME>, edit the files, type
quilt add <FILE>on each file you edited, then type
quilt refreshto recreate the patch.
- Once you're all done, copy the patches in the
patches/directory to the
osg/dir in SVN, run
quilt seriesto get the application order and update the spec file accordingly.
See if a package builds successfully for OSG 3.4¶
- If you have all the build dependencies of the package installed, run
osg-build rpmbuild <PACKAGEDIR>. The resulting RPMs will be in the
- If you do not have all the build dependencies installed, or want to make sure you specified all of the necessary ones and the package builds from a clean environment, run
osg-build mock --mock-config-from-koji osg-3.4-el6-build <PACKAGEDIR>. The resulting RPMs will be in the
- If you do not have mock installed, or want to exactly replicate the build environment in Koji, run
osg-build koji --scratch <PACKAGEDIR>. You may download the resulting RPMs from kojiweb https://koji.chtc.wisc.edu/koji or pass
osg-build kojiand they will get downloaded to the
Check for potential errors in a package¶
osg-build lint <PACKAGEDIR>.
Create and test a final build of a package for all platforms for upcoming¶
svn commityour changes in
osg-build koji --repo=upcoming <PACKAGEDIR>
- Wait for the
osg-upcoming-minefieldrepos to be regenerated containing the new version of your package. You can run
osg-koji wait-repo osg-upcoming-el<X>-development --build=<PACKAGENAME-VERSION-RELEASE>and wait for that process to finish (substitute
7for X). Or, you can just check kojiweb https://koji.chtc.wisc.edu/koji/tasks.
- On your test machine, make sure the
osg-upcoming-minefieldrepo is enabled (edit
/etc/yum.repos.d/osg-el6-upcoming-minefield.repo). Clean your cache (
yum clean all; yum clean expire-cache).
- Install your software, see if it works.
Promote the latest build of a package to testing for the current OSG release series¶
osg-promote -r testing <PACKAGE>
Promote the latest build of a package to testing for the 3.3 and 3.4 release series¶
osg-promote -r 3.3-testing -r 3.4-testing <PACKAGE>