Introduction to the BeStMan-xrootd for US ATLAS

Wei Yang
Stanford Linear Accelerator Center

What is BeStMan-xrootd
Installation and Configuration

Special Notes
Running BeStMan-xrootd on XrootdFS
Acknowledgement

What is BeStMan-xrootd

BeStMan-xrootd is a generic SRM v2.2 load balancing frontend for GridFTP servers. It is developed by the Scientific Data Management Group of Lawrence Berkeley National Laboratory. It implements a subset of SRM v2.2 specifications, and is specifically designed to work with gLite File Transfer Service (FTS) and glite-url-copy, a componment of gLite software used by the ATLAS experiment on the Large Hadron Collider.

The following SRM v2.2 functions have been implemented or partially implemented:

srmLs()
srmRm()
srmMkdir()
srmRmdir()
srmPrepareToPut()
srmPrepareToGet()
srmPutDone()
srmReleaseFiles()
srmGetSpaceTokens()

BeStMan-xrootd doesn't do any request queuing or space management, nor does it transfer data. It relies on its srmPrepareToPut() and srmPrepareToGet() functions, along with the GridFTP servers to do the actual data transfer. On the other hand, with only a subset of SRM v2.2 functions, BeStMan-xrootd is relatively light weight compare to full suite BeStMan or other SRM v2.2 implementations.

BeStMan-xrootd requires one or more GridFTP servers. It can access the backend storage system via these GridFTP servers. It can also access the backend storage if a storage system provides a Unix-like Posix file system interface. To work with the current version of FTS and glite-url-copy , a file system interface for the storage system is required.

BeStMan-xrootd was originally designed for the Xrootd storage system. However, it can be used with any storage system that has a GridFTP server and a Posix file system interface. Though a few directory structures and configuration keywords are still named after Xrootd, they do not mean that BeStMan-xrootd is for Xrootd storage only.

Installation and Configuration

BeStMan-xrootd software is available at the BeStMan web site. The version we tested is 2.2.0.8.c3 which requires Java 1.6 (Interested users can also download SRM clients from there). As root, expand the tar ball in, for example /opt and then do the following steps:

cd /opt/bestman-xrootd/setup
./configure --with-srm-home=/opt/bestman-xrootd --with-java-home=your_jdk1.6

This will create a configuration file /opt/bestman-xrootd/conf/bestman.rc. Add the following lines to this configuration file:

supportedProtocolList=gsiftp://host1;gsiftp://host2;gsiftp://host2
checkSizeWithXrootdFS=true
checkSizeWithGsiftp=false
checkFileSysWithGsiftp=false

The above three options tell BeStMan-xrootd whether to do srmLs, srmRm, srmMkdir, srmRmdir, srmMv via file system or GridFTP servers. If all are true, file system will be used first, and GridFTP servers as a fall back mechanism. If all are false, BeStMan-xrootd will not start.

The future support of srmLs, etc. over GridFTP is unclear at this point because LCG/gLite clients will not pass a correct proxy to BeStMan-xrootd which will be used again to authenticate with the GridFTP server. Users are advised to set exactly as the above so that BeStMan-xrootd will use file system for those operations.

Running as non-root

To run BeStMan-xrootd as an ordinary user and to do the above operations via file system, that user will need sudo provileges in order to act as the GSI authenticated users. To do so, the following entry need to be in BeStMan-xrootd configuration file:

useSudo=true

and the following entries need to be added to /etc/sudoers:

Cmnd_Alias SRM_CMD = /bin/rm, /bin/mkdir, /bin/rmdir, /bin/rm
Runas_Alias USER_LST = ALL, !root
daemon hostname=(USER_LST) NOPASSWD: SRM_CMD

The current version of BeStMan-xrootd assumes that all directories are world readable, and does not use sudo for srmLs.

Note that some sudo program supports "requiretty" (please check the sudoers man page). If so, "requiretty" needs to be set to off in /etc/sudoers file by adding the following line:

Defaults !requiretty

Xrootd storge doesn't support user ownership so far. Using sudo is not necessary for Xrootd storage.

Support of Static Space Tokens

BeStMan-xrootd supports predefined, static space tokens by defining the following line to the configuration file:

staticTokenList=Token1[space in BG][desc:Token1_Description];Token2[space in BG][desc:Token2_Description]

BeStMan-xrootd doesn't keep trace or enforce static space tokens. For Xrootd storage, BeStMan-Xrootd can add the space token info to the TURL it emits if the following line is in the configuration file, and it is up to the Xrootd storage system to decide what to do with that info. For other storage, the token info is ignored.

xrootdTokenCompName=oss.cgroup

Special Notes

To use BeStMan-xrootd with US ATLAS Distributed Data Management system, BeStMan-xrootd needs to be configured to use file system to do I/O (SRM functions for stat(), unlink(), mkdir(), rmdir()). It seems that FTS and glite-url-copy has a issue dealing with limited proxy. Due to this issue BeStMan-xrootd can not use GridFTP for I/O under this situation.
In one of the most common scenarios, FTS and glite-url-copy issues two srmLs() followed by a srmPrepareToPut(). If 'supportedProtocolList' has three GridFTP servers, BeStMan-xrootd's round-robin algorithm will always tell FTS/glite-url-copy to use the 3rd GridFTP server. This scenario should be avoided.
Avoid running BeStMan-xrootd as root. A client can access any file that the BeStMan-xrootd process' effective user can access. We suggest to run as a non-privileged user such as 'daemon'. To run in such a way, make sure this user can access files pointed by the 'CertFileName' and 'KeyFileName' keywords in the configuration file. Also note that since a client can access 'KeyFileName', using the host certificate should be avoided. BeStMan-xrootd takes service certificate as well.
We only tested BeStMan-xrootd on a system running 32bit Linux 2.6 kernel. BeStMan-xrootd is written in Java. So we expect it will work on 64bit platforms as well.
The lcg-util (lcg-ls, lcg-cp, etc) requires that BeStMan-xrootd uses a host certificate, not a service certificate.
ATLAS space tokens such as ATLASDATADISK, ATLASMCDISK, etc. are really being used by LCG/gLite utilities as space token descriptions.

Running BeStMan-xrootd on XrootdFS

As we explained previously, BeStMan-xrootd needs to run on top of XrootdFS. The XrootdFS software can be found at here. The order to start BeStMan-xrootd and XrootdFS is not important.

BeStMan-xrootd needs to work together with the GridFTP servers. A GridFTP server Data Storage Interface (DSI) module for Xrootd storage is available from the Xrootd software. The source code of the DSI module has instructions on how to compile and use it.

Acknowledgement

Thanks the LBNL SRM team, especially Alex Sim, Junmin Gu and Vijaya Natarajan to get all these work out. Also thanks Paolo Tedesco of CERN and Hironori Ito of BNL for very useful information regarding the interaction between SRM and FTS.

First draft created Jan. 4, 2008
Updated Jun. 10, 2008