Detailed Description

Combine ROOT file containing objects with SQLite database with info on the objects.

KVSQLROOTFile is intended to solve a longstanding problem with ROOT files containing many (and possibly "large") objects: in order to make any selection of the objects based on their associated properties, or indeed just to list the objects according to their properties, requires to read into memory each of the objects in question, which can be very slow.

When objects are written to a KVSQLROOTFile, the object itself is written in a ROOT file with an automatically generated unique identifier, while a list of associated metadata (chosen by the user) is written in a separate SQLite database file (see KVSQLite::database).

It is then possible to list the metadata of all objects in the file without reading them into memory, just by consulting the SQLite database. Selected objects can also be retrieved from the ROOT file based on the stored metadata.

Example of use

To create a new file:

KVSQLROOTFile f("my_file.sqlroot","recreate")

f

#define f(i)

KVSQLROOTFile

Combine ROOT file containing objects with SQLite database with info on the objects.

Definition: KVSQLROOTFile.h:76

This will create the following directory struture in the working directory:

my_file.sqlroot
      |
      |
      +--- objInfos.sqlite
      |
      +--- objStore.root

Store some objects with associated metadata:

auto n = new TNamed("A","A named object")
f.WriteObject(n, {{"name","X"},{"date","yesterday"},{"size [GB]",0.6}})
n = new TNamed("B","Another named object")
f.WriteObject(n, {{"date","today"},{"size",1.3}})

Note that the file is automatically closed when the KVSQLROOTFile goes out of scope.

Open a file and browse its contents:

KVSQLROOTFile f("my_file.sqlroot");
f.ls();
 
name     class    date     size
===========================================================
A     TNamed      yesterday   0.600000
B     TNamed      today    1.300000

Retrieve object(s) based on metadata selection:

KVUnownedList L
f.FillListOfObjectsWithSelection(&L, "size < 1.0")
L.ls()
OBJ: KVUnownedList   KVSeqCollection_0  : 0
 OBJ: TNamed   A  A named object : 0 at: 0x56513a9b58f0

Author: John Frankland

Date: Tue Sep 6 16:49:41 2022

Definition at line 76 of file KVSQLROOTFile.h.

#include <KVSQLROOTFile.h>

Inheritance diagram for KVSQLROOTFile:

[legend]

Public Member Functions
	KVSQLROOTFile (const KVString &filepath, Option_t *option="READ")

	~KVSQLROOTFile ()

void	FillListOfObjectsWithSelection (KVSeqCollection *list, const KVString &where)

void	FillListOfObjectsWithSelection (KVSeqCollection *list, const KVString &where, const KVString &numberlist_column, int value)

TObject *	Get (const KVString &name) const
	Return pointer to object with given name. More...

void	ls (Option_t *="") const
	List the contents of the file with associated infos. More...

void	WriteObject (const TObject *, const KVNameValueList &)

Public Member Functions inherited from KVBase
	KVBase ()
	Default constructor. More...

	KVBase (const Char_t name, const Char_t title="")
	Ctor for object with given name and type. More...

	KVBase (const KVBase &)
	copy ctor More...

virtual	~ KVBase ()

virtual void	Clear (Option_t *opt="")
	Clear object properties : name, type/title, number, label. More...

virtual void	Copy (TObject &) const
	Make a copy of this object. More...

const Char_t *	GetLabel () const

UInt_t	GetNumber () const

UInt_t	GetNumberOfObjects () const

virtual TObject *	GetObject () const

virtual const Char_t *	GetType () const

Bool_t	HasLabel () const

virtual Bool_t	IsCalled (const Char_t *name) const

Bool_t	IsLabelled (const Char_t *l) const

virtual Bool_t	IsType (const Char_t *typ) const

virtual void	List ()

KVBase &	operator= (const KVBase &)
	copy assignment operator More...

virtual void	Print (Option_t *option="") const

Double_t	ProtectedGetX (const TF1 *func, Double_t val, int &status, Double_t xmin=0.0, Double_t xmax=0.0) const

void	SetLabel (const Char_t *lab)

virtual void	SetNumber (UInt_t num)

virtual void	SetType (const Char_t *str)

Public Member Functions inherited from TNamed
	TNamed ()

	TNamed (const char name, const char title)

	TNamed (const TNamed &named)

	TNamed (const TString &name, const TString &title)

virtual	~TNamed ()

TObject *	Clone (const char *newname="") const override

Int_t	Compare (const TObject *obj) const override

virtual void	FillBuffer (char *&buffer)

const char *	GetName () const override

const char *	GetTitle () const override

ULong_t	Hash () const override

TClass *	IsA () const override

Bool_t	IsSortable () const override

TNamed &	operator= (const TNamed &rhs)

virtual void	SetName (const char *name)

virtual void	SetNameTitle (const char name, const char title)

virtual void	SetTitle (const char *title="")

virtual Int_t	Sizeof () const

void	Streamer (TBuffer &) override

void	StreamerNVirtual (TBuffer &ClassDef_StreamerNVirtual_b)

Public Member Functions inherited from TObject
	TObject ()

	TObject (const TObject &object)

virtual	~TObject ()

void	AbstractMethod (const char *method) const

virtual void	AppendPad (Option_t *option="")

virtual void	Browse (TBrowser *b)

ULong_t	CheckedHash ()

virtual const char *	ClassName () const

virtual void	Delete (Option_t *option="")

virtual Int_t	DistancetoPrimitive (Int_t px, Int_t py)

virtual void	Draw (Option_t *option="")

virtual void	DrawClass () const

virtual TObject *	DrawClone (Option_t *option="") const

virtual void	Dump () const

virtual void	Error (const char method, const char msgfmt,...) const

virtual void	Execute (const char method, const char params, Int_t *error=nullptr)

virtual void	Execute (TMethod method, TObjArray params, Int_t *error=nullptr)

virtual void	ExecuteEvent (Int_t event, Int_t px, Int_t py)

virtual void	Fatal (const char method, const char msgfmt,...) const

virtual TObject *	FindObject (const char *name) const

virtual TObject *	FindObject (const TObject *obj) const

virtual Option_t *	GetDrawOption () const

virtual const char *	GetIconName () const

virtual char *	GetObjectInfo (Int_t px, Int_t py) const

virtual Option_t *	GetOption () const

virtual UInt_t	GetUniqueID () const

virtual Bool_t	HandleTimer (TTimer *timer)

Bool_t	HasInconsistentHash () const

virtual void	Info (const char method, const char msgfmt,...) const

virtual Bool_t	InheritsFrom (const char *classname) const

virtual Bool_t	InheritsFrom (const TClass *cl) const

virtual void	Inspect () const

void	InvertBit (UInt_t f)

Bool_t	IsDestructed () const

virtual Bool_t	IsEqual (const TObject *obj) const

virtual Bool_t	IsFolder () const

R__ALWAYS_INLINE Bool_t	IsOnHeap () const

R__ALWAYS_INLINE Bool_t	IsZombie () const

void	MayNotUse (const char *method) const

virtual Bool_t	Notify ()

void	Obsolete (const char method, const char asOfVers, const char *removedFromVers) const

void	operator delete (void *ptr)

void	operator delete (void ptr, void vp)

void	operator delete[] (void *ptr)

void	operator delete[] (void ptr, void vp)

void *	operator new (size_t sz)

void *	operator new (size_t sz, void *vp)

void *	operator new[] (size_t sz)

void *	operator new[] (size_t sz, void *vp)

TObject &	operator= (const TObject &rhs)

virtual void	Paint (Option_t *option="")

virtual void	Pop ()

virtual Int_t	Read (const char *name)

virtual void	RecursiveRemove (TObject *obj)

void	ResetBit (UInt_t f)

virtual void	SaveAs (const char filename="", Option_t option="") const

virtual void	SavePrimitive (std::ostream &out, Option_t *option="")

void	SetBit (UInt_t f)

void	SetBit (UInt_t f, Bool_t set)

virtual void	SetDrawOption (Option_t *option="")

virtual void	SetUniqueID (UInt_t uid)

void	StreamerNVirtual (TBuffer &ClassDef_StreamerNVirtual_b)

virtual void	SysError (const char method, const char msgfmt,...) const

R__ALWAYS_INLINE Bool_t	TestBit (UInt_t f) const

Int_t	TestBits (UInt_t f) const

virtual void	UseCurrentStyle ()

virtual void	Warning (const char method, const char msgfmt,...) const

virtual Int_t	Write (const char *name=nullptr, Int_t option=0, Int_t bufsize=0)

virtual Int_t	Write (const char *name=nullptr, Int_t option=0, Int_t bufsize=0) const

Private Member Functions
TObject *	get_object_with_UUID (const KVString &name) const
	Retrieve object from file using its name. More...

KVSQLite::table &	get_objInfos () const

KVSQLite::table &	get_objTable () const

void	restore_working_directory ()

void	save_working_directory ()

KVString	UUID_for_object (const KVString &) const
	Return unique identifier used to store object with given name in ROOT file. More...

Private Attributes
TString	fCurrentROOTFilePath
	full path to current ROOT file More...

std::unique_ptr< KVSQLite::database >	fObjDB

std::unordered_map< std::string, TObject * >	fObjList
	for quick look-up of objects using unique id More...

std::unique_ptr< TFile >	fObjStore

TDirectory *	savDir

Additional Inherited Members
Public Types inherited from KVBase
enum	EKaliVedaBits { kIsKaliVedaObject = BIT(23) }

Public Types inherited from TObject
enum	EDeprecatedStatusBits

enum	EStatusBits

Static Public Member Functions inherited from KVBase
static Bool_t	AreEqual (Double_t x, Double_t y, Long64_t maxdif=1)
	Comparison between two 64-bit floating-point values. More...

static void	BackupFileWithDate (const Char_t *path)

static void	CombineFiles (const Char_t file1, const Char_t file2, const Char_t *newfilename, Bool_t keep=kTRUE)

static void	Deprecated (const char method, const char advice)

static Bool_t	FindClassSourceFiles (const Char_t class_name, KVString &imp_file, KVString &dec_file, const Char_t dir_name=".")

static Bool_t	FindExecutable (TString &exec, const Char_t *path="$(PATH)")

static const Char_t *	FindFile (const Char_t *search, TString &wfil)

static const Char_t *	GetBINDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetDATABASEFilePath ()

static const Char_t *	GetDATADIRFilePath (const Char_t *namefile="")

static Bool_t	GetDataSetEnv (const Char_t dataset, const Char_t type, Bool_t defval)

static const Char_t *	GetDataSetEnv (const Char_t dataset, const Char_t type, const Char_t *defval)

static Double_t	GetDataSetEnv (const Char_t dataset, const Char_t type, Double_t defval)

static const Char_t *	GetETCDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetExampleFilePath (const Char_t library, const Char_t namefile)
	Return full path to example file for given library (="KVMultiDet", "BackTrack", etc.) More...

static const Char_t *	GetINCDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetKVBuildDate ()
	Returns KaliVeda build date. More...

static const Char_t *	GetKVBuildDir ()
	Returns top-level directory used for build. More...

static const Char_t *	GetKVBuildTime ()
	Returns KaliVeda build time. More...

static const Char_t *	GetKVBuildType ()
	Returns KaliVeda build type (cmake build: Release, Debug, RelWithDebInfo, ...) More...

static const Char_t *	GetKVBuildUser ()
	Returns username of person who performed build. More...

static const Char_t *	GetKVSourceDir ()
	Returns top-level directory of source tree used for build. More...

static const Char_t *	GetKVVersion ()
	Returns KaliVeda version string. More...

static const Char_t *	GetLIBDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetListOfPlugins (const Char_t *base)

static const Char_t *	GetListOfPluginURIs (const Char_t *base)

static const Char_t *	GetPluginURI (const Char_t base, const Char_t plugin)

static void	GetTempFileName (TString &base)

static const Char_t *	GetTEMPLATEDIRFilePath (const Char_t *namefile="")

static const Char_t *	GetWORKDIRFilePath (const Char_t *namefile="")

static const Char_t *	gitBranch ()
	Returns git branch of sources. More...

static const Char_t *	gitCommit ()
	Returns last git commit of sources. More...

static void	InitEnvironment ()

static bool	is_gnuinstall ()

static Bool_t	IsThisAPlugin (const TString &uri, TString &base)

static TPluginHandler *	LoadPlugin (const Char_t base, const Char_t uri="0")

static Bool_t	OpenContextMenu (const char method, TObject obj, const char *alt_method_name="")

static void	OpenTempFile (TString &base, std::ofstream &fp)

static void	PrintSplashScreen ()
	Prints welcome message and infos on version etc. More...

static Bool_t	SearchAndOpenKVFile (const Char_t name, KVSQLite::database &dbfile, const Char_t kvsubdir="")

static Bool_t	SearchAndOpenKVFile (const Char_t name, std::ifstream &file, const Char_t kvsubdir="", KVLockfile *locks=0)

static Bool_t	SearchAndOpenKVFile (const Char_t name, std::ofstream &file, const Char_t kvsubdir="", KVLockfile *locks=0)

static Bool_t	SearchKVFile (const Char_t name, TString &fullpath, const Char_t kvsubdir="")

static const Char_t *	WorkingDirectory ()

Static Public Member Functions inherited from TNamed
static TClass *	Class ()

static const char *	Class_Name ()

static constexpr Version_t	Class_Version ()

static const char *	DeclFileName ()

Static Public Member Functions inherited from TObject
static TClass *	Class ()

static const char *	Class_Name ()

static constexpr Version_t	Class_Version ()

static const char *	DeclFileName ()

static Longptr_t	GetDtorOnly ()

static Bool_t	GetObjectStat ()

static void	SetDtorOnly (void *obj)

static void	SetObjectStat (Bool_t stat)

Public Attributes inherited from TObject
	kBitMask

	kCanDelete

	kCannotPick

	kHasUUID

	kInconsistent

	kInvalidObject

	kIsOnHeap

	kIsReferenced

	kMustCleanup

	kNoContextMenu

	kNotDeleted

	kObjInCanvas

	kOverwrite

	kSingleKey

	kWriteDelete

	kZombie

Protected Member Functions inherited from TObject
virtual void	DoError (int level, const char location, const char fmt, va_list va) const

void	MakeZombie ()

Protected Attributes inherited from TNamed
TString	fName

TString	fTitle

Protected Attributes inherited from TObject
	kOnlyPrepStep

Constructor & Destructor Documentation

◆ KVSQLROOTFile()

KVSQLROOTFile::KVSQLROOTFile	(	const KVString &	filepath,
		Option_t *	option = `"READ"`
	)

Open or (re)create a combined ROOT object store & SQL database with info on the stored objects.

Parameters

[in]	filepath	path to file. any shell variables will be expanded.
[in]	option	see TFile constructors for possible values. default="READ".

This will actually open or create two files in the directory given by filepath:

objStore.root - ROOT file containing objects
objInfos.sqlite - SQLite database containing metadata for objects

When first created, the SQLite database contains a table objTable with the following columns:

obj_idx	name	class	unique_id
*	*	*	*

The unique_id for each object is generated using TUUID.

Definition at line 58 of file KVSQLROOTFile.cpp.

◆ ~KVSQLROOTFile()

KVSQLROOTFile::~KVSQLROOTFile ( )

Manually 'delete' ROOT file in order to force writing to disk, then change access permissions to make it writable for group

Definition at line 117 of file KVSQLROOTFile.cpp.

Member Function Documentation

◆ FillListOfObjectsWithSelection() [1/2]

void KVSQLROOTFile::FillListOfObjectsWithSelection	(	KVSeqCollection *	list,
		const KVString &	where
	)

Fill the list given as argument with pointers to all objects which obey the given selection.

The 'where' string will be used as the WHERE clause of an SQLite selection.

Examples:

WHERE column_1 = 100;
 
WHERE column_2 IN (1,2,3);
 
WHERE column_3 LIKE 'An%';
 
WHERE column_4 BETWEEN 10 AND 20;

Note that string arguments should be enclosed in single quotes.

The columns of both the objTable (name, class) and objInfos tables can be used in the selection.

Definition at line 278 of file KVSQLROOTFile.cpp.

◆ FillListOfObjectsWithSelection() [2/2]

void KVSQLROOTFile::FillListOfObjectsWithSelection	(	KVSeqCollection *	list,
		const KVString &	where,
		const KVString &	numberlist_column,
		int	value
	)

Fill the list given as argument with pointers to all objects which obey the given selection. 'numberlist_column' is the name of a column in the objInfos table containing strings which may be interpreted as KVNumberList objects. Only the entries with a number list containing 'value' will be selected.

The 'where' string will be used as the WHERE clause of an SQLite selection.

Examples:

WHERE column_1 = 100;
 
WHERE column_2 IN (1,2,3);
 
WHERE column_3 LIKE 'An%';
 
WHERE column_4 BETWEEN 10 AND 20;

Note that string arguments should be enclosed in single quotes.

The columns of both the objTable (name, class) and objInfos tables can be used in the selection.

Definition at line 332 of file KVSQLROOTFile.cpp.

◆ Get()

TObject * KVSQLROOTFile::Get ( const KVString & name ) const

Return pointer to object with given name.

Definition at line 214 of file KVSQLROOTFile.cpp.

◆ get_object_with_UUID()

TObject * KVSQLROOTFile::get_object_with_UUID ( const KVString & name ) const

private

Retrieve object from file using its name.

Definition at line 26 of file KVSQLROOTFile.cpp.

◆ get_objInfos()

KVSQLite::table& KVSQLROOTFile::get_objInfos ( ) const

inlineprivate

Definition at line 94 of file KVSQLROOTFile.h.

◆ get_objTable()

KVSQLite::table& KVSQLROOTFile::get_objTable ( ) const

inlineprivate

Definition at line 90 of file KVSQLROOTFile.h.

◆ ls()

void KVSQLROOTFile::ls ( Option_t * = "" ) const

virtual

List the contents of the file with associated infos.

Reimplemented from TNamed.

Definition at line 226 of file KVSQLROOTFile.cpp.

◆ restore_working_directory()

void KVSQLROOTFile::restore_working_directory ( )

inlineprivate

Definition at line 85 of file KVSQLROOTFile.h.

◆ save_working_directory()

void KVSQLROOTFile::save_working_directory ( )

inlineprivate

Definition at line 81 of file KVSQLROOTFile.h.

◆ UUID_for_object()

KVString KVSQLROOTFile::UUID_for_object ( const KVString & name ) const

private

Return unique identifier used to store object with given name in ROOT file.

Definition at line 10 of file KVSQLROOTFile.cpp.

◆ WriteObject()

void KVSQLROOTFile::WriteObject	(	const TObject *	obj,
		const KVNameValueList &	infos
	)

Write object in ROOT file, store infos given in list

Parameters

obj	pointer to object to store
infos	list of metadata associated with object

If it does not already exist, this will add to the SQLite database a table objInfos with the following columns:

info_idx	obj_idx
*	*

This table will be augmented with columns corresponding to the names of the metadata contained in infos. e.g. if infos = {{"name","john"},{"age",50}} the table will become

info_idx	obj_idx	name	age
*	*	*	*

Definition at line 148 of file KVSQLROOTFile.cpp.

Member Data Documentation

◆ fCurrentROOTFilePath

TString KVSQLROOTFile::fCurrentROOTFilePath

private

full path to current ROOT file

Definition at line 104 of file KVSQLROOTFile.h.

◆ fObjDB

std::unique_ptr<KVSQLite::database> KVSQLROOTFile::fObjDB

private

Definition at line 78 of file KVSQLROOTFile.h.

◆ fObjList

std::unordered_map<std::string, TObject*> KVSQLROOTFile::fObjList

mutableprivate

for quick look-up of objects using unique id

Definition at line 99 of file KVSQLROOTFile.h.

◆ fObjStore

std::unique_ptr<TFile> KVSQLROOTFile::fObjStore

private

Definition at line 77 of file KVSQLROOTFile.h.

◆ savDir

TDirectory* KVSQLROOTFile::savDir

private

Definition at line 80 of file KVSQLROOTFile.h.

Detailed Description

Example of use

Public Member Functions

Private Member Functions

Private Attributes

Additional Inherited Members

Constructor & Destructor Documentation

◆ KVSQLROOTFile()

◆ ~KVSQLROOTFile()

Member Function Documentation

◆ FillListOfObjectsWithSelection() [1/2]

◆ FillListOfObjectsWithSelection() [2/2]

◆ Get()

◆ get_object_with_UUID()

◆ get_objInfos()

◆ get_objTable()

◆ ls()

◆ restore_working_directory()

◆ save_working_directory()

◆ UUID_for_object()

◆ WriteObject()

Member Data Documentation

◆ fCurrentROOTFilePath

◆ fObjDB

◆ fObjList

◆ fObjStore

◆ savDir