Key-Value Store API

This module specifies the key-value store API for the various package management and installation systems that are in use at Enthought and our clients.

The key-value store API exposes an interface on top of whatever backend implementation is used by subclasses. This permits code which requires access to a key-value store to use it in a uniform way without having to care about where and how the data is stored. The API is also agnostic about what is being stored, and so while the key use case is for egg repositories, potentially any data values can be stored in the key-value store.

class encore.storage.abstract_store.Value

Abstract base class for file-like objects used by Key-Value stores

size

The size of the data in bytes, or None if a continuous stream or unknown.

Type:

int

created

The creation time of the key as a floating point UTC timestamp in seconds after the Unix Epoch.

Type:

timestamp

modified

The modification time of the key as a floating point UTC timestamp in seconds after the Unix Epoch.

Type:

timestamp

abstract property data

The byte stream of data contained in the value

iterdata(buffer_size=1048576, progress=None)

Return an iterator over the data stream

abstract property metadata

The metadata dictionary of the value

abstract property permissions

The permissions dictionary of the value

This is only available if the user has ownership privileges for the key. Because different stores have different permission conventions, this will not be used when setting a value.

abstract range(start=None, end=None)

Return a stream with a range of bytes from the data

class encore.storage.abstract_store.AbstractReadOnlyStore

Abstract base class for read-only Key-Value Store API

This class implements some of the API so that it can be used with super() where appropriate.

event_manager

Every store is assumed to have an event_manager attribute which implements the BaseEventManager API.

abstract connect(credentials=None)

Connect to the key-value store, optionally with authentication

This method creates or connects to any long-lived resources that the store requires.

Parameters:

credentials – An object that can supply appropriate credentials to to authenticate the use of any required resources. The exact form of the credentials is implementation-specific, but may be as simple as a (username, password) tuple.

Raises:

AuthorizationError : – If the credentials are not valid, this error will be raised.

abstract disconnect()

Disconnect from the key-value store

This method disposes or disconnects to any long-lived resources that the store requires.

exists(key)

Test whether or not a key exists in the key-value store

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns:

exists – Whether or not the key exists in the key-value store.

Return type:

bool

abstract get(key)

Retrieve a stream of data and metdata from a given key in the key-value store.

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns:

value – An instance of a Value subclass which holds references to the data, metadata and other information about the key.

Return type:

instance of Value

Raises:

KeyError : – If the key is not found in the store, a KeyError is raised.

get_data(key)

Retrieve a stream from a given key in the key-value store.

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns:

data – A readable file-like object the that provides stream of data from the key-value store.

Return type:

file-like

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

get_data_range(key, start=None, end=None)

Retrieve a partial stream from a given key in the key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• start (int or None) – The first byte to return

• end (int or None) – The last byte of to return

Returns:

data – A readable file-like object the that provides stream of data from the key-value store.

Return type:

file-like

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

get_metadata(key, select=None)

Retrieve the metadata for a given key in the key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• select (iterable of strings or None) – Which metadata keys to populate in the result. If unspecified, then return the entire metadata dictionary.

Returns:

metadata – A dictionary of metadata associated with the key. The dictionary has keys as specified by the select argument. If a key specified in select is not present in the metadata, then it will not be present in the returned value.

Return type:

dict

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

glob(pattern)

Return keys which match glob-style patterns

Parameters:

pattern (string) – Glob-style pattern to match keys with.

Returns:

result – A iterable of keys which match the glob pattern.

Return type:

iterable

abstract is_connected()

Whether or not the store is currently connected

Returns:

connected – Whether or not the store is currently connected.

Return type:

bool

multiget(keys)

Retrieve the data and metadata for a collection of keys.

Parameters:

keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

Returns:

result – An iterator of (data, metadata) pairs.

Return type:

iterator of (file-like, dict) tuples

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

multiget_data(keys)

Retrieve the data for a collection of keys.

Parameters:

keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

Returns:

result – An iterator of file-like data objects corresponding to the keys.

Return type:

iterator of file-like

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

multiget_metadata(keys, select=None)

Retrieve the metadata for a collection of keys in the key-value store.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• select (iterable of strings or None) – Which metadata keys to populate in the results. If unspecified, then return the entire metadata dictionary.

Returns:

metadatas – An iterator of dictionaries of metadata associated with the key. The dictionaries have keys as specified by the select argument. If a key specified in select is not present in the metadata, then it will not be present in the returned value.

Return type:

iterator of dicts

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

abstract query(select=None, **kwargs)

Query for keys and metadata matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of metadata for the key-value store.

Parameters:

• select (iterable of strings or None) – An optional list of metadata keys to return. If this is not None, then the metadata dictionaries will only have values for the specified keys populated.

• kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns:

result – An iterable of (key, metadata) tuples where metadata matches all the specified values for the specified metadata keywords. If a key specified in select is not present in the metadata of a particular key, then it will not be present in the returned value.

Return type:

iterable

query_keys(**kwargs)

Query for keys matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of keys for the key-value store.

This is equivalent to self.query(**kwargs).keys(), but potentially more efficiently implemented.

Parameters:

kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns:

result – An iterable of key-value store keys whose metadata matches all the specified values for the specified metadata keywords.

Return type:

iterable

to_bytes(key, buffer_size=1048576)

Efficiently store the data associated with a key into a bytes object.

This method can be optionally overriden by subclasses to proved a more efficient way of copy the data from the underlying data store to a bytes object. The default implementation uses the get() method together with chunked reads from the returned data stream and join.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

Returns:

• bytes – The contents of the file-like object as bytes.

• Events

• ——

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to extracting the data.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is extracted.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after extracting the data.

to_file(key, path, buffer_size=1048576)

Efficiently store the data associated with a key into a file.

This method can be optionally overriden by subclasses to proved a more efficient way of copy the data from the underlying data store to a path in the filesystem. The default implementation uses the get() method together with chunked reads from the returned data stream to the disk.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• path (string) – A file system path to store the data to.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to disk.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to disk.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to disk.

class encore.storage.abstract_store.AbstractStore

Abstract base class for Key-Value Store API

This class implements some of the API so that it can be used with super() where appropriate.

event_manager

Every store is assumed to have an event_manager attribute which implements the BaseEventManager API.

abstract delete(key)

Delete a key from the repsository.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• Events –

• ------ –

• StoreDeleteEvent – On successful completion of a transaction, a StoreDeleteEvent should be emitted with the key.

from_bytes(key, data, buffer_size=1048576)

Efficiently store a bytes object as the data associated with a key.

This method can be optionally overriden by subclasses to proved a more efficient way of copy the data from a bytes object to the underlying data store. The default implementation uses the set() method together with a cStringIO.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• data (bytes) – The data as a bytes object.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

from_file(key, path, buffer_size=1048576)

Efficiently read data from a file into a key in the key-value store.

This method can be optionally overriden by subclasses to proved a more efficient way of copy the data from a path in the filesystem to the underlying data store. The default implementation uses the set() method together with chunked reads from the disk which are fed into the data stream.

This makes no attempt to set metadata.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• path (string) – A file system path to read the data from.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

multiset(keys, values, buffer_size=1048576)

Set the data and metadata for a collection of keys.

Where supported by an implementation, this should perform the whole collection of sets as a single transaction.

Like zip() if keys and values have different lengths, then any excess values in the longer list should be silently ignored.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• values (iterable of (file-like, dict) tuples) – An iterator that provides the (data, metadata) pairs for the corresponding keys.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.

multiset_data(keys, datas, buffer_size=1048576)

Set the data for a collection of keys.

Where supported by an implementation, this should perform the whole collection of sets as a single transaction.

Like zip() if keys and datas have different lengths, then any excess values in the longer list should be silently ignored.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• datas (iterable of file-like objects) – An iterator that provides the data file-like objects for the corresponding keys.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.

multiset_metadata(keys, metadatas)

Set the metadata for a collection of keys.

Where supported by an implementation, this should perform the whole collection of sets as a single transaction.

Like zip() if keys and metadatas have different lengths, then any excess values in the longer list should be silently ignored.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• metadatas (iterable of dicts) – An iterator that provides the metadata dictionaries for the corresponding keys.

• Events –

• ------ –

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.

multiupdate_metadata(keys, metadatas)

Update the metadata for a collection of keys.

Where supported by an implementation, this should perform the whole collection of sets as a single transaction.

Like zip() if keys and metadatas have different lengths, then any excess values in the longer list should be silently ignored.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• metadatas (iterable of dicts) – An iterator that provides the metadata dictionaries for the corresponding keys.

• Events –

• ------ –

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.

abstract set(key, value, buffer_size=1048576)

Store a stream of data into a given key in the key-value store.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• value (instance of Value) – An instance of a Value subclass.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

abstract set_data(key, data, buffer_size=1048576)

Replace the data for a given key in the key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• data (file-like) – A readable file-like object the that provides stream of data from the key-value store.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

abstract set_metadata(key, metadata)

Set new metadata for a given key in the key-value store.

This replaces the existing metadata set for the key with a new set of metadata.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

• Events –

• ------ –

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

abstract transaction(notes)

Provide a transaction context manager

Implementations which have no native notion of transactions may choose not to implement this.

This method provides a context manager which creates a data store transaction in its __enter__() method, and commits it in its __exit__() method if no errors occur. Intended usage is:

with repo.transaction("Writing data..."):
    # everything written in this block is part of the transaction
    ...

If the block exits without error, the transaction commits, otherwise the transaction should roll back the state of the underlying data store to the start of the transaction.

Parameters:

notes (string) – Some information about the transaction, which may or may not be used by the implementation.

Returns:

• transaction (context manager) – A context manager for the transaction.

• Events

• ——

• StoreTransactionStartEvent – This event should be emitted on entry into the transaction.

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreTransactionEndEvent – This event should be emitted on successful conclusion of the transaction, before any Set or Delete events are emitted.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set during the transaction.

• StoreDeleteEvent – On successful completion of a transaction, a StoreDeleteEvent should be emitted with the key for all deleted keys.

abstract update_metadata(key, metadata)

Update the metadata for a given key in the key-value store.

This performs a dictionary update on the existing metadata with the provided metadata keys and values

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

• Events –

• ------ –

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

class encore.storage.abstract_store.AbstractAuthorizingStore

Abstract base class for Key-Value Store API with permissioning

This class implements some of the API so that it can be used with super() where appropriate.

Permission information is available only to authenticated users who are designated as owners of a particular key. Permissions are simply strings representing some right that the store allows, the only required permission being ‘owned’.

Each permission has a set of tags which are granted that permission. A tag represents a user, group or role that will be granted that permission. The meaning of tags is also store dependent: a filesystem-based store may have tags for ‘user’, ‘group’ and ‘other’; while a web-based store may derive its tags from a role-based authentication system.

event_manager

Every store is assumed to have an event_manager attribute which implements the BaseEventManager API.

abstract get_permissions(key)

Return the set of permissions the user has

Parameters:

key (str) – The key for the resource which you want to know the permissions.

Returns:

permissions – A dictionary whose keys are the permissions and values are sets of tags which have that permission.

Return type:

dict of str: set of str

Raises:

• KeyError : – This error will be raised if the key does not exist or the user is not authorized to see it.

• AuthorizationError : – This error will be raised if user is authorized to see the key, but is not an owner.

abstract set_permissions(key, permissions)

Set the permissions on a key the user owns

Parameters:

• key (str) – The key for the resource which you want to know the permissions.

• permissions (dict of str: set of str) – A dictionary whose keys are the permissions and values are sets of tags which have that permission. There must be an ‘owned’ permission with at least one tag.

Raises:

• KeyError : – This error will be raised if the key does not exist or the user is not authorized to see it.

• AuthorizationError : – This error will be raised if user is authorized to see the key, but is not an owner.

abstract update_permissions(key, permissions)

Add permissions on a key the user owns

The tags provided in the permissions dictionary will be added to the existing set of tags for each permission.

Parameters:

• key (str) – The key for the resource which you want to know the permissions.

• permissions (dict of str: set of str) – A dictionary whose keys are the permissions and values are sets of tags which have that permission.

Raises:

• KeyError : – This error will be raised if the key does not exist or the user is not authorized to see it.

• AuthorizationError : – This error will be raised if user is authorized to see the key, but is not an owner.

abstract property user_tag

A tag that represents the user