Sqlite Store

This is a simple implementation of the key-value store API that lives in a sqlite database. Each key is stored in a row which consists of the key, index columns, metadata and data. The index columns are a specified subset of the metadata that can be queried more quickly.

This class is provided in part as a sample implementation of the API.

class encore.storage.sqlite_store.SqliteStore(location=':memory:', table='store', index='dynamic', index_columns=None)

Sqlite-based Store

The file-like objects returned by data methods are cStringIO objects.

Warning

The table name and metadata names used as index columns are not sanitized. To prevent SQL injection these should never be directly derived from user-supplied values. This is particularly important for indexed queries.

__init__(location=':memory:', table='store', index='dynamic', index_columns=None)

connect(credentials=None)

Connect to the key-value store

This connects to the specified location and creates the table, if needed. Sqlite has no notion of authentication, so credentials are ignored.

delete(key)

Delete a key from the repsository.

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

disconnect()

Disconnect from the key-value store

This clears the reference to the sqlite connection object, allowing it to be garbage-collected.

exists(key)

Test whether or not a key exists in the key-value store

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns:

exists – Whether or not the key exists in the key-value store.

Return type:

bool

from_bytes(key, data, buffer_size=1048576)

Efficiently read data from a bytes object into a key in the key-value store.

This makes no attempt to set metadata.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• data (bytes) – The data as a bytes object.

• buffer_size (int) – This is ignored.

from_file(key, path, buffer_size=1048576)

Efficiently read data from a file into a key in the key-value store.

This makes no attempt to set metadata.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• path (string) – A file system path to read the data from.

• buffer_size (int) – This is ignored.

get(key)

Retrieve a stream of data and metdata from a given key in the key-value store.

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns:

• data (file-like) – A readable file-like object that provides stream of data from the key-value store

• metadata (dictionary) – A dictionary of metadata for the key.

Raises:

KeyError : – If the key is not found in the store, a KeyError is raised.

get_data(key)

Retrieve a stream from a given key in the key-value store.

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns:

data – A readable file-like object the that provides stream of data from the key-value store.

Return type:

file-like

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

get_metadata(key, select=None)

Retrieve the metadata for a given key in the key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• select (iterable of strings or None) – Which metadata keys to populate in the result. If unspecified, then return the entire metadata dictionary.

Returns:

metadata – A dictionary of metadata associated with the key. The dictionary has keys as specified by the metadata_keys argument.

Return type:

dict

Raises:

KeyError : – This will raise a key error if the key is not present in the store, and if any metadata key is requested which is not present in the metadata.

glob(pattern)

Return keys which match glob-style patterns

Parameters:

pattern (string) – Glob-style pattern to match keys with.

Returns:

result – A iterable of keys which match the glob pattern.

Return type:

iterable

is_connected()

Whether or not the store is currently connected

Returns:

connected – Whether or not the store is currently connected.

Return type:

bool

multiget(keys)

Retrieve the data and metadata for a collection of keys.

Parameters:

keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

Returns:

result – An iterator of (data, metadata) pairs.

Return type:

iterator of (file-like, dict) tuples

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

multiget_data(keys)

Retrieve the data for a collection of keys.

Parameters:

keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

Returns:

result – An iterator of file-like data objects corresponding to the keys.

Return type:

iterator of file-like

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

multiget_metadata(keys, select=None)

Retrieve the metadata for a collection of keys in the key-value store.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• select (iterable of strings or None) – Which metadata keys to populate in the results. If unspecified, then return the entire metadata dictionary.

Returns:

metadatas – An iterator of dictionaries of metadata associated with the key. The dictionaries have keys as specified by the select argument. If a key specified in select is not present in the metadata, then it will not be present in the returned value.

Return type:

iterator of dicts

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

multiset(keys, values, buffer_size=1048576)

Set the data and metadata for a collection of keys.

Where supported by an implementation, this should perform the whole collection of sets as a single transaction.

Like zip() if keys and values have different lengths, then any excess values in the longer list should be silently ignored.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• values (iterable of (file-like, dict) tuples) – An iterator that provides the (data, metadata) pairs for the corresponding keys.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.

multiset_data(keys, datas, buffer_size=1048576)

Set the data for a collection of keys.

Where supported by an implementation, this should perform the whole collection of sets as a single transaction.

Like zip() if keys and datas have different lengths, then any excess values in the longer list should be silently ignored.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• datas (iterable of file-like objects) – An iterator that provides the data file-like objects for the corresponding keys.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.

multiset_metadata(keys, metadatas)

Set the metadata for a collection of keys.

Where supported by an implementation, this should perform the whole collection of sets as a single transaction.

Like zip() if keys and metadatas have different lengths, then any excess values in the longer list should be silently ignored.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• metadatas (iterable of dicts) – An iterator that provides the metadata dictionaries for the corresponding keys.

• Events –

• ------ –

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.

multiupdate_metadata(keys, metadatas)

Update the metadata for a collection of keys.

Where supported by an implementation, this should perform the whole collection of sets as a single transaction.

Like zip() if keys and metadatas have different lengths, then any excess values in the longer list should be silently ignored.

Parameters:

• keys (iterable of strings) – The keys for the resources in the key-value store. Each key is a unique identifier for a resource within the key-value store.

• metadatas (iterable of dicts) – An iterator that provides the metadata dictionaries for the corresponding keys.

• Events –

• ------ –

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set.

query(select=None, **kwargs)

Query for keys and metadata matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of metadata for the key-value store.

Parameters:

• select (iterable of strings or None) – An optional list of metadata keys to return. If this is not None, then the metadata dictionaries will only have values for the specified keys populated.

• kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns:

result – An iterable of keys, metadata tuples where metadata matches all the specified values for the specified metadata keywords.

Return type:

iterable

query_keys(**kwargs)

Query for keys matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of keys for the key-value store.

This is equivalent to self.query(**kwargs).keys(), but potentially more efficiently implemented.

Parameters:

kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns:

result – An iterable of key-value store keys whose metadata matches all the specified values for the specified metadata keywords.

Return type:

iterable

set(key, value, buffer_size=1048576)

Store a stream of data into a given key in the key-value store.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• value (tuple of file-like, dict) – A pair of objects, the first being a readable file-like object that provides stream of data from the key-value store. The second is a dictionary of metadata for the key.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

set_data(key, data, buffer_size=1048576)

Replace the data for a given key in the key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• data (file-like) – A readable file-like object the that provides stream of data from the key-value store.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

set_metadata(key, metadata)

Set new metadata for a given key in the key-value store.

This replaces the existing metadata set for the key with a new set of metadata.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

to_bytes(key, buffer_size=1048576)

Efficiently store the data associated with a key into a bytes object.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• buffer_size (int) – This is ignored.

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

to_file(key, path, buffer_size=1048576)

Efficiently store the data associated with a key into a file.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• path (string) – A file system path to store the data to.

• buffer_size (int) – This is ignored.

Raises:

KeyError : – This will raise a key error if the key is not present in the store.

transaction(notes)

Provide a transaction context manager

update_metadata(key, metadata)

Update the metadata for a given key in the key-value store.

This performs a dictionary update on the existing metadata with the provided metadata keys and values

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

Raises:

KeyError : – This will raise a key error if the key is not present in the store.