Detailed Description

Manage a collection of data files on a remote machine.

Data repositories are defined in the user's $HOME/.kvrootrc; some typical examples are given in $KVROOT/KVFiles/.kvrootrc:

DataRepository: ccali
ccali.DataRepository.Type: remote
ccali.DataRepository.RootDir:       cchpssindra:/hpss/in2p3.fr/group/indra
ccali.DataRepository.ReadProtocol:     root
ccali.DataRepository.XRDServer:      ccxrdsn012:1999
ccali.DataRepository.XRDRootDir:       /hpss/in2p3.fr/group/indra
ccali.DataRepository.XRDTunnel.host:       ccali.in2p3.fr
ccali.DataRepository.XRDTunnel.port:          10000
ccali.DataRepository.XRDTunnel.user:
ccali.DataRepository.RemoteAvailableRuns.protocol:  curl
ccali.DataRepository.RemoteAvailableRuns.url:   http://indra.in2p3.fr/KaliVedaDoc
ccali.DataRepository.FileTransfer.type:    bbftp
ccali.DataRepository.FileTransfer.server:    ccbbftp.in2p3.fr
ccali.DataRepository.FileTransfer.user:

<p<blockquote>

This is the definition of a "remote" data repository. A remote data repository is principally characterised by the fact that access to the data files is via some non-local protocol: in this case it is xrootd ("...ReadProtocol: root"). The host name and port of the xrootd server are given ("...XRDServer: ccxrdsn012:1999"), as well as the root directory to be used ("...XRDRootDir: /hpss/in2p3.fr/group/indra").

In order to use a remote repository, some way to access the database of available data files for each run type must be defined. Here it is the curl programme which is used to read them via a website ("...RemoteAvailableRuns.protocol: curl"; "...RemoteAvailableRuns.url: http://indra.in2p3.fr/KaliVedaDoc").

Another particularity of remote data repositories is that some additional stuff may be needed in order to be able to access the data. This is handled by KVRemoteDataRepository::IsConnected(). In the present example, an SSH tunnel is set up in order to allow secure connection to the xrootd server ("...XRDTunnel.*:").

Finally, transfer of data between data repositories is handled by KVDataTransfer and child classes. In the present example, this is configured to use bbftp ("...FileTransfer.type: bbftp"), for which the required details are given.

Definition at line 68 of file KVRemoteDataRepository.h.

#include <KVRemoteDataRepository.h>

Inheritance diagram for KVRemoteDataRepository:

[legend]

Public Member Functions
	KVRemoteDataRepository ()
	Default constructor. More...

virtual	~ KVRemoteDataRepository ()

virtual Bool_t	CheckFileStatus (const KVDataSet ds, const Char_t datatype, const Char_t *runfile)

virtual Bool_t	CheckSubdirExists (const Char_t dir, const Char_t subdir=0)

virtual void	CopyFileFromRepository (const KVDataSet ds, const Char_t datatype, const Char_t filename, const Char_t destination)

virtual int	CopyFileToRepository (const Char_t source, const KVDataSet ds, const Char_t datatype, const Char_t filename)

virtual void	DeleteFile (const KVDataSet ds, const Char_t datatype, const Char_t *filename, Bool_t confirm=kTRUE)

virtual KVUniqueNameList *	GetDirectoryListing (const KVDataSet ds, const Char_t datatype="", const Char_t *subdir="")

virtual Bool_t	GetFileInfo (const KVDataSet ds, const Char_t datatype, const Char_t *runfile, FileStat_t &fs)

virtual const Char_t *	GetFullPathToOpenFile (const KVDataSet dataset, const Char_t datatype, const Char_t *runfile)

virtual const Char_t *	GetFullPathToTransferFile (const KVDataSet dataset, const Char_t datatype, const Char_t *runfile)

virtual Bool_t	IsConnected ()

virtual Bool_t	IsRemote () const

virtual void	MakeSubdirectory (const KVDataSet ds, const Char_t datatype="")

Public Member Functions inherited from KVDataRepository
	KVDataRepository ()
	Default constructor. More...

virtual	~ KVDataRepository ()

virtual Bool_t	CanWrite () const

void	cd ()

virtual void	CommitFile (TFile file, const Char_t datatype, const KVDataSet *dataset)

virtual int	CopyFile (const char f, const char t, Bool_t overwrite=kFALSE)

void	CreateAllNeededSubdirectories (const KVDataSet DataSet, const Char_t DataType)

virtual TFile *	CreateNewFile (const KVDataSet dataset, const Char_t datatype, const Char_t *filename)

TSystem *	FindHelper (const char path, void dirptr=0)

virtual KVDataSetManager *	GetDataSetManager () const
	Return pointer to data set manager for this repository. More...

virtual const Char_t *	GetFileTransferExec () const
	returns full path to executable used for remote file transfer More...

virtual const Char_t *	GetFileTransferServer () const
	returns server url used for remote file transfer More...

virtual const Char_t *	GetFileTransferType () const
	returns protocol used for remote file transfer More...

virtual const Char_t *	GetFileTransferUser () const
	returns user name used for remote file transfer More...

virtual const Char_t *	GetRootDirectory () const
	returns root directory of data repository (fLocalrootdir) More...

Bool_t	HelperIsConsistentWith (TSystem helper, const char path, void *dirptr=0)

virtual Bool_t	Init ()

virtual KVAvailableRunsFile *	NewAvailableRunsFile (const Char_t , const KVDataSet )

virtual TObject *	OpenDataSetRunFile (const KVDataSet ds, const Char_t type, Int_t run, Option_t *opt="")

virtual void	Print (Option_t *opt="") const
	Print info on repository. More...

virtual void	PrintAvailableDatasetsUpdateWarning () const

Public Member Functions inherited from KVBase
	KVBase ()
	Default constructor. More...

	KVBase (const Char_t name, const Char_t title="")
	Ctor for object with given name and type. More...

	KVBase (const KVBase &)
	copy ctor More...

virtual	~ KVBase ()

virtual void	Clear (Option_t *opt="")
	Clear object properties : name, type/title, number, label. More...

virtual void	Copy (TObject &) const
	Make a copy of this object. More...

const Char_t *	GetLabel () const

UInt_t	GetNumber () const

UInt_t	GetNumberOfObjects () const

virtual TObject *	GetObject () const

virtual const Char_t *	GetType () const

Bool_t	HasLabel () const

virtual Bool_t	IsCalled (const Char_t *name) const

Bool_t	IsLabelled (const Char_t *l) const

virtual Bool_t	IsType (const Char_t *typ) const

virtual void	List ()

KVBase &	operator= (const KVBase &)
	copy assignment operator More...

Double_t	ProtectedGetX (const TF1 *func, Double_t val, int &status, Double_t xmin=0.0, Double_t xmax=0.0) const

void	SetLabel (const Char_t *lab)

virtual void	SetNumber (UInt_t num)

virtual void	SetType (const Char_t *str)

Private Member Functions
virtual KVDataSetManager *	NewDataSetManager ()
	Create and return pointer to new data set manager. More...

Additional Inherited Members
Public Types inherited from KVBase
enum	EKaliVedaBits { kIsKaliVedaObject = BIT(23) }

Static Public Member Functions inherited from KVDataRepository
static KVDataRepository *	NewRepository (const Char_t *type)

Static Public Member Functions inherited from KVBase
static Bool_t	AreEqual (Double_t x, Double_t y, Long64_t maxdif=1)
	Comparison between two 64-bit floating-point values. More...

static void	BackupFileWithDate (const Char_t *path)

static void	CombineFiles (const Char_t file1, const Char_t file2, const Char_t *newfilename, Bool_t keep=kTRUE)

static void	Deprecated (const char method, const char advice)

static Bool_t	FindClassSourceFiles (const Char_t class_name, KVString &imp_file, KVString &dec_file, const Char_t dir_name=".")

static Bool_t	FindExecutable (TString &exec, const Char_t *path="$(PATH)")

static const Char_t *	FindFile (const Char_t *search, TString &wfil)

static const Char_t *	GetBINDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetDATABASEFilePath ()

static const Char_t *	GetDATADIRFilePath (const Char_t *namefile="")

static Bool_t	GetDataSetEnv (const Char_t dataset, const Char_t type, Bool_t defval)

static const Char_t *	GetDataSetEnv (const Char_t dataset, const Char_t type, const Char_t *defval)

static Double_t	GetDataSetEnv (const Char_t dataset, const Char_t type, Double_t defval)

static const Char_t *	GetETCDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetExampleFilePath (const Char_t library, const Char_t namefile)
	Return full path to example file for given library (="KVMultiDet", "BackTrack", etc.) More...

static const Char_t *	GetINCDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetKVBuildDate ()
	Returns KaliVeda build date. More...

static const Char_t *	GetKVBuildDir ()
	Returns top-level directory used for build. More...

static const Char_t *	GetKVBuildTime ()
	Returns KaliVeda build time. More...

static const Char_t *	GetKVBuildType ()
	Returns KaliVeda build type (cmake build: Release, Debug, RelWithDebInfo, ...) More...

static const Char_t *	GetKVBuildUser ()
	Returns username of person who performed build. More...

static const Char_t *	GetKVSourceDir ()
	Returns top-level directory of source tree used for build. More...

static const Char_t *	GetKVVersion ()
	Returns KaliVeda version string. More...

static const Char_t *	GetLIBDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetListOfPlugins (const Char_t *base)

static const Char_t *	GetListOfPluginURIs (const Char_t *base)

static const Char_t *	GetPluginURI (const Char_t base, const Char_t plugin)

static void	GetTempFileName (TString &base)

static const Char_t *	GetTEMPLATEDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetWORKDIRFilePath (const Char_t *namefile="")

static const Char_t *	gitBranch ()
	Returns git branch of sources. More...

static const Char_t *	gitCommit ()
	Returns last git commit of sources. More...

static void	InitEnvironment ()

static bool	is_gnuinstall ()

static Bool_t	IsThisAPlugin (const TString &uri, TString &base)

static TPluginHandler *	LoadPlugin (const Char_t base, const Char_t uri="0")

static Bool_t	OpenContextMenu (const char method, TObject obj, const char *alt_method_name="")

static void	OpenTempFile (TString &base, std::ofstream &fp)

static void	PrintSplashScreen ()
	Prints welcome message and infos on version etc. More...

static Bool_t	SearchAndOpenKVFile (const Char_t name, KVSQLite::database &dbfile, const Char_t kvsubdir="")

static Bool_t	SearchAndOpenKVFile (const Char_t name, std::ifstream &file, const Char_t kvsubdir="", KVLockfile *locks=0)

static Bool_t	SearchAndOpenKVFile (const Char_t name, std::ofstream &file, const Char_t kvsubdir="", KVLockfile *locks=0)

static Bool_t	SearchKVFile (const Char_t name, TString &fullpath, const Char_t kvsubdir="")

static const Char_t *	WorkingDirectory ()

Constructor & Destructor Documentation

◆ KVRemoteDataRepository()

KVRemoteDataRepository::KVRemoteDataRepository ( )

Default constructor.

Definition at line 22 of file KVRemoteDataRepository.cpp.

◆ ~ KVRemoteDataRepository()

virtual KVRemoteDataRepository::~ KVRemoteDataRepository ( )

virtual

Member Function Documentation

◆ CheckFileStatus()

Bool_t KVRemoteDataRepository::CheckFileStatus	(	const KVDataSet *	ds,
		const Char_t *	datatype,
		const Char_t *	runfile
	)

virtual

Checks if the run file of given type is physically present in dataset subdirectory, i.e. (schematically), if

/root_of_data_repository/[datasetdir]/[datatype]/[runfile]

exists. If it does, the returned value is kTRUE (=1). For a remote data repository, this is not possible. Always returns kFALSE.

Reimplemented from KVDataRepository.

Definition at line 106 of file KVRemoteDataRepository.cpp.

◆ CheckSubdirExists()

Bool_t KVRemoteDataRepository::CheckSubdirExists	(	const Char_t *	dir,
		const Char_t *	subdir = `0`
	)

virtual

Returns kTRUE if the following path is valid /root_of_data_repository/dir/[subdir] For a remote data repository, this is not possible. Always returns kFALSE.

Reimplemented from KVDataRepository.

Definition at line 48 of file KVRemoteDataRepository.cpp.

◆ CopyFileFromRepository()

void KVRemoteDataRepository::CopyFileFromRepository	(	const KVDataSet *	ds,
		const Char_t *	datatype,
		const Char_t *	filename,
		const Char_t *	destination
	)

virtual

Copy file [datasetdir]/[datatype]/[filename] from the repository to [destination] We check if the file to copy exists. For remote repositories, this uses the bbftp protocol (if available). NEEDS IMPLEMENTATION.

Reimplemented from KVDataRepository.

Definition at line 132 of file KVRemoteDataRepository.cpp.

◆ CopyFileToRepository()

int KVRemoteDataRepository::CopyFileToRepository	(	const Char_t *	source,
		const KVDataSet *	ds,
		const Char_t *	datatype,
		const Char_t *	filename
	)

virtual

Copy file [source] to [datasetdir]/[datatype]/[filename] in the repository For remote repositories, this uses the bbftp protocol (if available). NEEDS IMPLEMENTATION.

Reimplemented from KVDataRepository.

Definition at line 152 of file KVRemoteDataRepository.cpp.

◆ DeleteFile()

void KVRemoteDataRepository::DeleteFile	(	const KVDataSet *	ds,
		const Char_t *	datatype,
		const Char_t *	filename,
		Bool_t	confirm = `kTRUE`
	)

virtual

Delete repository file [datasetdir]/[datatype]/[filename] Impossible on distant data repositories.

Reimplemented from KVDataRepository.

Definition at line 202 of file KVRemoteDataRepository.cpp.

◆ GetDirectoryListing()

KVUniqueNameList * KVRemoteDataRepository::GetDirectoryListing	(	const KVDataSet *	ds,
		const Char_t *	datatype = `""`,
		const Char_t *	subdir = `""`
	)

virtual

Impossible on distant data repositories. Will return 0 (null pointer).

Reimplemented from KVDataRepository.

Definition at line 186 of file KVRemoteDataRepository.cpp.

◆ GetFileInfo()

Bool_t KVRemoteDataRepository::GetFileInfo	(	const KVDataSet *	ds,
		const Char_t *	datatype,
		const Char_t *	runfile,
		FileStat_t &	fs
	)

virtual

Checks if the run file of given type is physically present in dataset subdirectory, i.e. (schematically), if

/root_of_data_repository/[datasetdir]/[datatype]/[runfile]

exists. If it does, the returned value is kTRUE (=1), in which case the FileStat_t object contains information about the file. For a remote data repository, this is not possible. Always returns kFALSE.

Reimplemented from KVDataRepository.

Definition at line 74 of file KVRemoteDataRepository.cpp.

◆ GetFullPathToOpenFile()

const Char_t * KVRemoteDataRepository::GetFullPathToOpenFile	(	const KVDataSet *	dataset,
		const Char_t *	datatype,
		const Char_t *	runfile
	)

virtual

When accessing remote data repositories, the operating system corresponding to the remote file system may not be the same as the local operating system (corresponding to gSystem). Therefore use of gSystem->ConcatFileName may lead to paths which mix '/' and '\'. We check for this here by converting the result of KVDataRepository::GetFullPath depending on whether the root directory of this repository contains '/' or '\'

Reimplemented from KVDataRepository.

Definition at line 287 of file KVRemoteDataRepository.cpp.

◆ GetFullPathToTransferFile()

const Char_t * KVRemoteDataRepository::GetFullPathToTransferFile	(	const KVDataSet *	dataset,
		const Char_t *	datatype,
		const Char_t *	runfile
	)

virtual

Used by KVDataTransfer. Returns the full path needed to transfer a runfile belonging to the given dataset either from or to the repository, using sftp or bbftp etc. This is a concatenation of the repository root directory with the dataset subdirectories and filename. However, for a remote data repository, the type of operating system may be different to gSystem, i.e. the system on the local machine. In this case, gSystem->ConcatFileName may not necessarily give the right paths for the remote machine: e.g. local machine is Windows, remote repository is Linux, paths for Linux machine will have a mix of "/" and "\" in them. We try to remedy this by looking at fLocalrootdir: if it contains "/" we take the result of the concatenation performed by KVDataRepository::GetFullPathToTransferFile and replace any "\" by "/", and vice versa.

Reimplemented from KVDataRepository.

Definition at line 242 of file KVRemoteDataRepository.cpp.

◆ IsConnected()

Bool_t KVRemoteDataRepository::IsConnected ( )

virtual

Before reading data in a remote data repository, it may be necessary to 'connect' to the repository in some way: e.g. we may need to open an SSH tunnel to an xrootd server. Call this method to make sure any required connection has been made, and if successful the method returns kTRUE; otherwise return kFALSE.

XROOTD via SSH tunnel

Configure using the following environment variables:

ccali.DataRepository.XRDTunnel.host: ccali.in2p3.fr ccali.DataRepository.XRDTunnel.port: 10000

The host and port to use are obligatory. Optionally you may also change the following:

ccali.DataRepository.XRDTunnel.retry: #max number of tries before giving up on connection, default = 30

ccali.DataRepository.XRDTunnel.user: #username for login to tunnel host. default is local username.

Reimplemented from KVDataRepository.

Definition at line 332 of file KVRemoteDataRepository.cpp.

◆ IsRemote()

virtual Bool_t KVRemoteDataRepository::IsRemote ( ) const

inlinevirtual

Reimplemented from KVDataRepository.

Definition at line 77 of file KVRemoteDataRepository.h.

◆ MakeSubdirectory()

void KVRemoteDataRepository::MakeSubdirectory	(	const KVDataSet *	ds,
		const Char_t *	datatype = `""`
	)

virtual

Create a new subdirectory in the repository Impossible on distant data repositories.

Reimplemented from KVDataRepository.

Definition at line 171 of file KVRemoteDataRepository.cpp.

◆ NewDataSetManager()

KVDataSetManager * KVRemoteDataRepository::NewDataSetManager ( )

privatevirtual

Create and return pointer to new data set manager.

Reimplemented from KVDataRepository.

Definition at line 218 of file KVRemoteDataRepository.cpp.

Detailed Description

Public Member Functions

Private Member Functions

Additional Inherited Members

Constructor & Destructor Documentation

◆ KVRemoteDataRepository()

◆ ~ KVRemoteDataRepository()

Member Function Documentation

◆ CheckFileStatus()

◆ CheckSubdirExists()

◆ CopyFileFromRepository()

◆ CopyFileToRepository()

◆ DeleteFile()

◆ GetDirectoryListing()

◆ GetFileInfo()

◆ GetFullPathToOpenFile()

◆ GetFullPathToTransferFile()

◆ IsConnected()

XROOTD via SSH tunnel

◆ IsRemote()

◆ MakeSubdirectory()

◆ NewDataSetManager()