Introduction to the BeStMan-Gateway for US ATLAS

Wei Yang
Stanford Linear Accelerator Center

What is BeStMan-gateway
Installation and Configuration
Special Notes
Running BeStMan-Gateway on XrootdFS

What is BeStMan-gateway

The Berkeley Storage Resource Manager, developed by the Scientific Data Management Group of Lawrence Berkeley National Laboratory, is a SRM v2.2 implementation for generic posix file systems. Since BeStMan version, the full BeStMan and BeStMan-xrootd are distributed under one package, and are called BeStMan full mode and gateway mode respectively. The BeStMan Gateway, or BeStMan-gateway is the successor of the BeStMan-xrootd, a generic SRM v2.2 load balancing frontend for GridFTP servers. It implements a subset of SRM v2.2 specifications, and is specifically designed to work with gLite File Transfer Service (FTS), glite-url-copy and LCG-utils, componments of gLite software used by the ATLAS experiment on the Large Hadron Collider. It also works with other SRM client implementations from Fermi National Accelerator Laboratory and LBNL.

The following SRM v2.2 functions have been implemented or partially implemented:

BeStMan-gateway doesn't do any request queuing or space management, nor does it transfer data. It relies on its srmPrepareToPut() and srmPrepareToGet() functions, along with the GridFTP servers to do the actual data transfer. On the other hand, with only a subset of SRM v2.2 functions, BeStMan-gateway is relatively light weight compare to full suite BeStMan or other SRM v2.2 implementations.

BeStMan-gateway requires one or more GridFTP servers. It can access the backend storage system via these GridFTP servers. It can also access the backend storage if the storage system provides a Unix-like posix file system interface. To work with the ATLAS Distributed Data Management system (DDM) , a file system interface for the storage system is required.

Though BeStMan-gateway is the successor of BeStMan-xrootd, BeStMan-gateway is designed to work with any storage system that has a GridFTP server and a posix file system interface. A few configuration keywords may still be named after Xrootd, they do not mean that BeStMan-gateway is for the Xrootd storage only.

Older releases of BeStMan (BeStMan 1) is replaced by a BeStMan 2 which use Jetty as its java container. Most of the feathers described in this document are common to both BeStMan 1 and BeStMan 2. We will mark those features that only exist in BeStMan 2.

Installation and Configuration

BeStMan software is available at the BeStMan web site. The version we tested is The installation can be done without root privilege. To install, expand the tar ball in, for example /opt and then do the following steps:

cd /opt/bestman/setup
./configure --enable-gateway-mode
--with-java-home=/path/to/jdk1.6 (Java 1.6 or higher is required)
--enable-serveronly (do not install BeStMan client)
--enable-eventlog (or --disable-eventlog)
--enable-gums (If GUMS server is available)
--with-gums-dn="DN of a host certificate GUMS server accepts"
(optional. The default is to use the DN of the BeStMan-gateway server)
--enable-checkfile-fs (srmLs/Rm/Mkdir/Rmdir/Mv via filesystem, required)
--with-srm-owner=daemon (See Running as non-root)
--with-tokens-list="Token1[desc:Token1_Description][space in GB][owner:atlas][retention:CUSTODIAL][latency:ONLINE][usedBytesCommand:/bin/echo 1234567890];Token2..."
--disable-pathfortoken (Associate a space token to a filesystem path)

This will create a configuration file /opt/bestman/conf/bestman.rc. Some of the configuration lines in that file are explained below.

GridFTP servers

A list of GridFTP servers used by BeStMan-gatweay:


Later versions of BeStMan 1 (not sure exactly which one) and BeStMan 2 and later allows one to put a list of GridFTP endpoints in a flat file and dynamically change it without restarting. Using this feature requires the LBNL.RFF package which can be found at:

It is also available in recent BeStMan 2 package. In the BeStMan configuration file, one needs to replace the above GridFTP configuation line by:


An example of the flat file looks like this


The file allows "#" prefixed comment lines but do not allow empty lines.

Using a GUMS server

BeStMan-gateway can work with a GUMS server by adding the following lines to the configuration file:

GUMSCurrHostDN=DN of a host certificate your GUMS server accepts

Accessing Storage via Filesystem

BeStMan-gateway can access the underline storage system (srmLs, srmRm, srmMkdir, srmRmdir, srmMv) via file system or via GridFTP servers. However, gLite tools such as glite-url-copy (used by FTS) and LCG-utils require that BeStMan-gateway to access the storage via a file system.


Running as non-root

To run BeStMan-gateway as an ordinary user and to do ls, rm, mkdir, rmdir operations via file system, that user will need sudo provileges in order to act on behave of the GSI authenticated users. To do so, the following entry need to be in BeStMan-gateway configuration file:


"noSudoOnLs=true" is the default (not using sudo for a 'ls' operation) because this operation is used frequently, and the overhead of 'sudo /bin/ls' is high.

To use sudo, the following entries need to be added to /etc/sudoers:

Cmnd_Alias SRM_CMD = /bin/ls, /bin/rm, /bin/mkdir, /bin/rmdir, /bin/mv
Runas_Alias USER_LST = ALL, !root
daemon hostname=(USER_LST) NOPASSWD: SRM_CMD

Note that some sudo program supports "requiretty" (please check the sudoers man page). If so, "requiretty" needs to be set to off in /etc/sudoers by adding the following line:

Defaults !requiretty

Note that using sudo is not needed for Xrootd storage unless ACL is enabled in XrootdFS.

Support of Static Space Tokens

BeStMan-gateway supports predefined, static space tokens with the following line in the configuration file:

staticTokenList=Token1[desc:Token1_Description][space in GB][owner:atlas][retention:CUSTODIAL][latency:ONLINE][usedBytesCommand:/bin/echo 1234567890];Token2[desc:Token2_Description]...

In BeStMan 2 or latter, the [space in GB] which list the total space (in GBytes) under the specified space token can also be an external command that returns the space in bytes. For example: [totalBytesCommand:echo 10000000000000].

The retention policy can be CUSTODIAL, OUTPUT or REPLICA. The command follows "usedBytesCommand" will be run by BeStMan-gateway to obtain used space info of the coresponding space token. The above "/bin/echo 1234567890" is just an example. The actual full path command should be provided by site adminstrators. The output of the external command should be similar to the above example. The unit is byte. Here is an example of the external program for xrootd:

   To compile: gcc this_program.c
   Assuming xrootd cluster with usage info cofigured, run the following to 
   get the usage info:
   "this_program /xrootdfs_mount_point/?oss.cgroup=Token1"
#include <stdio.h>
#include <sys/types.h>
#include <attr/xattr.h>

main(int argc, char* argv[]) {
    ssize_t rc;
    char value[1024];

    rc = getxattr(argv[1],argv[2],value,1023);
    if (rc == -1) exit(-1);
BeStMan-gateway doesn't keep trace or enforce static space tokens. For Xrootd storage, BeStMan-gateway can add the space token info to the TURL it emits if the following line is in the configuration file, and it is up to the Xrootd storage system to decide what to do with that info. For other storage, the token info is ignored.


Associating a Space Token to a Filesystem Path

Somce storage systems associated a space token to a filesystem path. For those storage systems, the usage of a space token can be obtained by calling statvfs() on the path. This info can be used to enable space availability checking when BeStMan-gateway receives a srmPrepareToPut() request. Xrootd storage does not have such an association. Therefore, one can set the following line to disable it:


Adler32 checksum

BeStMan-gateway supports Adler32 checksum calculation on the fly. It also support using an external command to calculate checksum. The external command must take a file name at command line, and provide Adler32 checksum in hexadecimal as the first string of the first output line. The rest of the output is ignored.

hexChecksumCommand="your external checksum command"

Many storage systems BeStMan-gateway supported don't store checksum. They calculate checksum on demand. Some SRM clients set checksum option by default when they send srmLs() request to SRM servers. This behavior may trigger unnecessary checksum calculation. To avoid overload a storage system by this type of checksum request, an option is provided to tune off directory browsing in order to avoid unintended checksum calcultion:


When this option is set to true, srmLs() of a directory will only return the information of the directory itself. No information is provided about files in that directory.

Other configurations for BeStMan-gateway

The following must be presented in the configuration file for BeStMan-gateway:


Special Notes

Running BeStMan-gateway on XrootdFS

As we explained previously, BeStMan-gateway needs to run on top of XrootdFS. The XrootdFS software can be found at here. The order to start BeStMan-gateway and XrootdFS is not important.

BeStMan-gateway needs to work together with the GridFTP servers. A GridFTP server Data Storage Interface (DSI) module for Xrootd storage is available from the Xrootd software. The source code of the DSI module has instructions on how to compile and use it.


Thanks the LBNL SRM team, especially Alex Sim, Junmin Gu and Vijaya Natarajan to get all these work out. Also thanks Paolo Tedesco of CERN and Hironori Ito of BNL for very useful information regarding the interaction between SRM and FTS.

First draft created Sep. 8, 2008, Last updated Jun 2, 2011