MountedStore

A store which combines two stores by mounting one of the stores at a particular point in the other store’s key space, prefixing all references to keys with the mount point. This is similar in concept to mounting filesystems.

class encore.storage.mounted_store.MountedStore(mount_point, mount_store, backing_store)

A key-value store that mounts another store at a particular key prefix

The backing store is treated as read-only, and only modifications are allowed to the first store, and only for keys which match the mounting prefix.

The primary purpose for this is to have a local cache of a subsection of a remote store, such as a StaticURLStore or DynamicURLStore.

Parameters:

• mount_point (str) – Key prefix for the mounted store.

• mount_store (AbstractStore) – The store to be mounted

• backing_store (AbstractStore) – The store that we are mounting against

connect(credentials=None)

Connect to the key-value store, optionally with authentication

This method creates or connects to any long-lived resources that the store requires.

Parameters:

credentials – An object that can supply appropriate credentials to to authenticate the use of any required resources. The exact form of the credentials is implementation-specific, but may be as simple as a (username, password) tuple.

delete(key)

Delete a key from the repsository.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• Events –

• ------ –

• StoreDeleteEvent – On successful completion of a transaction, a StoreDeleteEvent should be emitted with the key.

disconnect()

Disconnect from the key-value store

This method disposes or disconnects to any long-lived resources that the store requires.

get(key)

Retrieve a stream of data and metdata from a given key in the key-value store.

Parameters:

key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

Returns:

• data (file-like) – A readable file-like object that provides stream of data from the key-value store

• metadata (dictionary) – A dictionary of metadata for the key.

Raises:

KeyError : – If the key is not found in the store, a KeyError is raised.

info()

Get information about the key-value store

Returns:

metadata – A dictionary of metadata giving information about the key-value store.

Return type:

dict

is_connected()

Whether or not the store is currently connected

Returns:

connected – Whether or not the store is currently connected.

Return type:

bool

push(key)

Move a key from the mount store to the backing store

query(select=None, **kwargs)

Query for keys and metadata matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of metadata for the key-value store.

Parameters:

• select (iterable of strings or None) – An optional list of metadata keys to return. If this is not None, then the metadata dictionaries will only have values for the specified keys populated.

• kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns:

result – An iterable of (key, metadata) tuples where metadata matches all the specified values for the specified metadata keywords. If a key specified in select is not present in the metadata of a particular key, then it will not be present in the returned value.

Return type:

iterable

query_keys(**kwargs)

Query for keys matching metadata provided as keyword arguments

This provides a very simple querying interface that returns precise matches with the metadata. If no arguments are supplied, the query will return the complete set of keys for the key-value store.

This is equivalent to self.query(**kwargs).keys(), but potentially more efficiently implemented.

Parameters:

kwargs – Arguments where the keywords are metadata keys, and values are possible values for that metadata item.

Returns:

result – An iterable of key-value store keys whose metadata matches all the specified values for the specified metadata keywords.

Return type:

iterable

set(key, value, buffer_size=1048576)

Store a stream of data into a given key in the key-value store.

This may be left unimplemented by subclasses that represent a read-only key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• value (tuple of file-like, dict) – A pair of objects, the first being a readable file-like object that provides stream of data from the key-value store. The second is a dictionary of metadata for the key.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

set_data(key, data, buffer_size=1048576)

Replace the data for a given key in the key-value store.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• data (file-like) – A readable file-like object the that provides stream of data from the key-value store.

• buffer_size (int) – An optional indicator of the number of bytes to read at a time. Implementations are free to ignore this hint or use a different default if they need to. The default is 1048576 bytes (1 MiB).

• Events –

• ------ –

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

set_metadata(key, metadata)

Set new metadata for a given key in the key-value store.

This replaces the existing metadata set for the key with a new set of metadata.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

• Events –

• ------ –

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata

transaction(notes)

Provide a transaction context manager

Implementations which have no native notion of transactions may choose not to implement this.

This method provides a context manager which creates a data store transaction in its __enter__() method, and commits it in its __exit__() method if no errors occur. Intended usage is:

with repo.transaction("Writing data..."):
    # everything written in this block is part of the transaction
    ...

If the block exits without error, the transaction commits, otherwise the transaction should roll back the state of the underlying data store to the start of the transaction.

Parameters:

notes (string) – Some information about the transaction, which may or may not be used by the implementation.

Returns:

• transaction (context manager) – A context manager for the transaction.

• Events

• ——

• StoreTransactionStartEvent – This event should be emitted on entry into the transaction.

• StoreProgressStartEvent – For buffering implementations, this event should be emitted prior to writing any data to the underlying store.

• StoreProgressStepEvent – For buffering implementations, this event should be emitted periodically as data is written to the underlying store.

• StoreProgressEndEvent – For buffering implementations, this event should be emitted after finishing writing to the underlying store.

• StoreTransactionEndEvent – This event should be emitted on successful conclusion of the transaction, before any Set or Delete events are emitted.

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata for each key that was set during the transaction.

• StoreDeleteEvent – On successful completion of a transaction, a StoreDeleteEvent should be emitted with the key for all deleted keys.

update_metadata(key, metadata)

Set new metadata for a given key in the key-value store.

This replaces the existing metadata set for the key with a new set of metadata.

Parameters:

• key (string) – The key for the resource in the key-value store. They key is a unique identifier for the resource within the key-value store.

• metadata (dict) – A dictionary of metadata to associate with the key. The dictionary keys should be strings which are valid Python identifiers.

• Events –

• ------ –

• StoreSetEvent – On successful completion of a transaction, a StoreSetEvent should be emitted with the key & metadata