Introduction to the BeStMan-Gateway for US ATLAS

What is BeStMan-gateway
Installation and Configuration
Special Notes
Running BeStMan-Gateway on XrootdFS
Acknowledgement

What is BeStMan-gateway

The following SRM v2.2 functions have been implemented or partially implemented:

• srmPing()
• srmGetTransferProtocols()
• srmLs()
• srmRm()
• srmMkdir()
• srmRmdir()
• srmPrepareToPut()
• srmPrepareToGet()
• srmPutDone()
• srmReleaseFiles()
• srmGetSpaceTokens()
• srmGetSpaceMetaData()

BeStMan-gateway requires one or more GridFTP servers. It can access the backend storage system via these GridFTP servers. It can also access the backend storage if the storage system provides a Unix-like posix file system interface. To work with the ATLAS Distributed Data Management system (DDM) , a file system interface for the storage system is required.

Though BeStMan-gateway is the successor of BeStMan-xrootd, BeStMan-gateway is designed to work with any storage system that has a GridFTP server and a posix file system interface. A few configuration keywords may still be named after Xrootd, they do not mean that BeStMan-gateway is for the Xrootd storage only.

Older releases of BeStMan (BeStMan 1) is replaced by a BeStMan 2 which use Jetty as its java container. Most of the feathers described in this document are common to both BeStMan 1 and BeStMan 2. We will mark those features that only exist in BeStMan 2.

Installation and Configuration

This will create a configuration file /opt/bestman/conf/bestman.rc. Some of the configuration lines in that file are explained below.

GridFTP servers

supportedProtocolList=gsiftp://host1;gsiftp://host2;gsiftp://host2

Later versions of BeStMan 1 (not sure exactly which one) and BeStMan 2 2.2.0.12 and later allows one to put a list of GridFTP endpoints in a flat file and dynamically change it without restarting. Using this feature requires the LBNL.RFF package which can be found at:

https://codeforge.lbl.gov/projects/bestmancontrib/

It is also available in recent BeStMan 2 package. In the BeStMan configuration file, one needs to replace the above GridFTP configuation line by:

protocolSelectionPolicy=class=plugin.ReadFromFile&jarFile=LBNL.RFF.jar&name=gsiftp¶m=flat_file_of_gridftp_list
pluginLib=directory_contains_the_LBNL.RFF.jar

An example of the flat file looks like this

gsiftp://gridftp1.abc.com
gsiftp://gridftp2.abc.com

The file allows "#" prefixed comment lines but do not allow empty lines.

Using a GUMS server

GridMapFileName=/opt/bestman/conf/grid-mapfile.empty
GUMSserviceURL=https://gums_host:port/gums/services/GUMSAuthorizationServicePort
GUMSCurrHostDN=DN of a host certificate your GUMS server accepts

Accessing Storage via Filesystem

checkSizeWithFS=true
checkSizeWithGsiftp=false

Running as non-root

accessFileSysViaSudo=true
noSudoOnLs=true

"noSudoOnLs=true" is the default (not using sudo for a 'ls' operation) because this operation is used frequently, and the overhead of 'sudo /bin/ls' is high.

To use sudo, the following entries need to be added to /etc/sudoers:

Cmnd_Alias SRM_CMD = /bin/ls, /bin/rm, /bin/mkdir, /bin/rmdir, /bin/mv
Runas_Alias USER_LST = ALL, !root
daemon hostname=(USER_LST) NOPASSWD: SRM_CMD

Note that some sudo program supports "requiretty" (please check the sudoers man page). If so, "requiretty" needs to be set to off in /etc/sudoers by adding the following line:

Defaults !requiretty

Note that using sudo is not needed for Xrootd storage unless ACL is enabled in XrootdFS.

Support of Static Space Tokens

/*
   To compile: gcc this_program.c
   Assuming xrootd cluster with usage info cofigured, run the following to 
   get the usage info:
   "this_program /xrootdfs_mount_point/?oss.cgroup=Token1 xroot.space.oss.used"
*/
#include <stdio.h>
#include <sys/types.h>
#include <attr/xattr.h>

main(int argc, char* argv[]) {
    ssize_t rc;
    char value[1024];

    rc = getxattr(argv[1],argv[2],value,1023);
    if (rc == -1) exit(-1);
    value[rc]='\0';
    printf("%s\n",value);
    exit(0);
}

Associating a Space Token to a Filesystem Path

pathForToken=false

Adler32 checksum

defaultChecksumType=adler32
showChecksumWhenListingFile=true
hexChecksumCommand="your external checksum command"

Many storage systems BeStMan-gateway supported don't store checksum. They calculate checksum on demand. Some SRM clients set checksum option by default when they send srmLs() request to SRM servers. This behavior may trigger unnecessary checksum calculation. To avoid overload a storage system by this type of checksum request, an option is provided to tune off directory browsing in order to avoid unintended checksum calcultion:

NoDirBrowsing=true

When this option is set to true, srmLs() of a directory will only return the information of the directory itself. No information is provided about files in that directory.

Other configurations for BeStMan-gateway

noCacheLog=true
CacheLogLocation=/var/log
disableSpaceMgt=true

Special Notes

• To use BeStMan-gateway with US ATLAS Distributed Data Management system, BeStMan-gateway needs to be configured to use file system to do I/O (SRM functions for stat(), unlink(), mkdir(), rmdir()). It seems that for security reason, gLite tools (glite-url-copy and LCG-utilss) have an issue dealing with limited proxies. Due to this issue BeStMan-gateway can not use GridFTP for I/O under this situation.
• In one of the most common scenarios, FTS and glite-url-copy issues two srmLs() followed by a srmPrepareToPut(). If 'supportedProtocolList' has three GridFTP servers, BeStMan-gateway's round-robin algorithm will always tell FTS/glite-url-copy to use the 3rd GridFTP server. This scenario should be avoided.
• Avoid running BeStMan-gateway as root. A client can access any file that the BeStMan-gateway process' effective user can access. We suggest to run as a non-privileged user such as 'daemon'. To run in such a way, make sure this user can access files pointed by the 'CertFileName' and 'KeyFileName' keywords in the configuration file. Also note that since a client can access 'KeyFileName', using host certificate should be avoided. BeStMan-gateway takes service certificate as well. However,
• The LCG-utils (lcg-ls, lcg-cp, etc) requires that BeStMan-gateway uses a host certificate, not a service certificate.
• We only tested BeStMan-gateway on a system running 32bit Linux 2.6 kernel. BeStMan-gateway is written in Java. So we expect it will work on 64bit platforms as well.
• ATLAS space tokens such as ATLASDATADISK, ATLASMCDISK, etc. are really beng used by LCG/gLite utilities as space token descriptions.
• One can increase MAX_JAVA_HEAP (in MB) in /opt/bestman/sbin/bestman.server to deal with heavy usage.

BeStMan-gateway needs to work together with the GridFTP servers. A GridFTP server Data Storage Interface (DSI) module for Xrootd storage is available from the Xrootd software. The source code of the DSI module has instructions on how to compile and use it.

Acknowledgement

First draft created Sep. 8, 2008, Last updated Jun 2, 2011