dtoolcore.storagebroker

Disk storage broker.

class dtoolcore.storagebroker.BaseStorageBroker

Base storage broker class defining the required interface.

add_item_metadata(handle, key, value)

Store the given key:value pair for the item associated with handle.

Parameters:

• handle – handle for accessing an item before the dataset is frozen
• key – metadata key
• value – metadata value

create_structure()

Create necessary structure to hold a dataset.

delete_key(key)

Delete the file/object associated with the key.

delete_tag(tag)

Delete a tag from a dataset.

Parameters:tag – tag

generate_base_uri(uri)

Return dataset base URI given a uri.

classmethod generate_uri(name, uuid, base_uri)

Return dataset URI.

get_admin_metadata()

Return the admin metadata as a dictionary.

get_admin_metadata_key()

Return the admin metadata key.

get_annotation(annotation_name)

Return value of the annotation associated with the key.

Returns:annotation (string, int, float, bool) Raises:DtoolCoreAnnotationKeyError if the annotation does not exist

get_annotation_key(annotation_name)

Return the annotation key.

get_hash(handle)

Return the hash.

get_item_abspath(identifier)

Return absolute path at which item content can be accessed.

Parameters:identifier – item identifier Returns:absolute path from which the item content can be accessed

get_item_metadata(handle)

Return dictionary containing all metadata associated with handle.

In other words all the metadata added using the add_item_metadata method.

Parameters:handle – handle for accessing an item before the dataset is frozen Returns:dictionary containing item metadata

get_manifest()

Return the manifest as a dictionary.

get_manifest_key()

Return the manifest key.

get_overlay(overlay_name)

Return overlay as a dictionary.

get_overlay_key(overlay_name)

Return the overlay key.

get_readme_content()

Return the README descriptive metadata as a string.

get_readme_key()

Return the admin metadata key.

get_relpath(handle)

Return the relative path.

get_size_in_bytes(handle)

Return the size in bytes.

get_tag_key(tag)

Return the tag key.

get_text(key)

Return the text associated with the key.

get_utc_timestamp(handle)

Return the UTC timestamp.

has_admin_metadata()

Return True if the administrative metadata exists.

This is the definition of being a “dataset”.

item_properties(handle)

Return properties of the item with the given handle.

iter_item_handles()

Return iterator over item handles.

list_annotation_names()

Return list of annotation names.

classmethod list_dataset_uris(base_uri, config_path)

Return list containing URIs in location given by base_uri.

list_overlay_names()

Return list of overlay names.

list_tags()

Return list of tags.

post_freeze_hook()

Post dtoolcore.ProtoDataSet.freeze() cleanup actions.

This method is called at the end of the dtoolcore.ProtoDataSet.freeze() method.

In the dtoolcore.storage_broker.DiskStorageBroker it removes the temporary directory for storing item metadata fragment files.

pre_freeze_hook()

Pre dtoolcore.ProtoDataSet.freeze() actions.

This method is called at the beginning of the dtoolcore.ProtoDataSet.freeze() method.

It may be useful for remote storage backends to generate caches to remove repetitive time consuming calls

put_admin_metadata(admin_metadata)

Store the admin metadata.

put_annotation(annotation_name, annotation)

Set/update value of the annotation associated with the key.

Raises:DtoolCoreAnnotationTypeError if the type of the value is not str, int, float or bool.

put_item(fpath, relpath)

Put item with content from fpath at relpath in dataset.

Missing directories in relpath are created on the fly.

Parameters:

• fpath – path to the item on disk
• relpath – relative path name given to the item in the dataset as a handle

Returns:

the handle given to the item

put_manifest(manifest)

Store the manifest.

put_overlay(overlay_name, overlay)

Store the overlay.

put_readme(content)

Store the readme descriptive metadata.

put_tag(tag)

Annotate the dataset with a tag.

put_text(key, text)

Put the text into the storage associated with the key.

update_readme(content)

Update the readme descriptive metadata.

class dtoolcore.storagebroker.DiskStorageBroker(uri, config_path=None)

Storage broker to interact with datasets on local disk storage.

The dtoolcore.ProtoDataSet class uses the dtoolcore.storage_broker.DiskStorageBroker to construct datasets by writing to disk and the dtoolcore.DataSet class uses it to read datasets from disk.

add_item_metadata(handle, key, value)

Store the given key:value pair for the item associated with handle.

Parameters:

• handle – handle for accessing an item before the dataset is frozen
• key – metadata key
• value – metadata value

delete_key(key)

Delete the file/object associated with the key.

classmethod generate_uri(name, uuid, base_uri)

Return dataset URI.

get_admin_metadata_key()

Return the path to the admin metadata file.

get_annotation_key(annotation_name)

Return the path to the annotation file.

get_dtool_readme_key()

Return the path to the dtool readme file.

get_hash(handle)

Return the hash.

get_item_abspath(identifier)

Return absolute path at which item content can be accessed.

Parameters:identifier – item identifier Returns:absolute path from which the item content can be accessed

get_item_metadata(handle)

Return dictionary containing all metadata associated with handle.

In other words all the metadata added using the add_item_metadata method.

Parameters:handle – handle for accessing an item before the dataset is frozen Returns:dictionary containing item metadata

get_manifest_key()

Return the path to the readme file.

get_overlay_key(overlay_name)

Return the path to the overlay file.

get_readme_key()

Return the path to the readme file.

get_size_in_bytes(handle)

Return the size in bytes.

get_structure_key()

Return the path to the structure parameter file.

get_tag_key(tag)

Return the path to the tag file.

get_text(key)

Return the text associated with the key.

get_utc_timestamp(handle)

Return the UTC timestamp.

has_admin_metadata()

Return True if the administrative metadata exists.

This is the definition of being a “dataset”.

hasher = <dtoolcore.filehasher.FileHasher object>

Attribute used by dtoolcore.ProtoDataSet to write the hash function name to the manifest.

iter_item_handles()

Return iterator over item handles.

key = 'file'

Attribute used to define the type of storage broker.

list_annotation_names()

Return list of annotation names.

classmethod list_dataset_uris(base_uri, config_path)

Return list containing URIs in location given by base_uri.

list_overlay_names()

Return list of overlay names.

list_tags()

Return list of tags.

post_freeze_hook()

Post dtoolcore.ProtoDataSet.freeze() cleanup actions.

This method is called at the end of the dtoolcore.ProtoDataSet.freeze() method.

In the dtoolcore.storage_broker.DiskStorageBroker it removes the temporary directory for storing item metadata fragment files.

pre_freeze_hook()

Pre dtoolcore.ProtoDataSet.freeze() actions.

This method is called at the beginning of the dtoolcore.ProtoDataSet.freeze() method.

It may be useful for remote storage backends to generate caches to remove repetitive time consuming calls

put_item(fpath, relpath)

Put item with content from fpath at relpath in dataset.

Missing directories in relpath are created on the fly.

Parameters:

• fpath – path to the item on disk
• relpath – relative path name given to the item in the dataset as a handle, i.e. a Unix-like relpath

Returns:

the handle given to the item

put_text(key, text)

Put the text into the storage associated with the key.

exception dtoolcore.storagebroker.DiskStorageBrokerValidationWarning

exception dtoolcore.storagebroker.StorageBrokerOSError